You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2020/10/21 11:51:56 UTC

[GitHub] [flink] JingsongLi opened a new pull request #13724: [FLINK-19581][orc] Introduce Orc ColumnarRow File source bulk Format

JingsongLi opened a new pull request #13724:
URL: https://github.com/apache/flink/pull/13724


   
   ## What is the purpose of the change
   
   Implement File source ORC Vectorized Reader
   
   ## Brief change log
   
   The `AbstractOrcFileInputFormat` reference from https://github.com/apache/flink/pull/13401
      - The base for ORC readers, Implements the reader initialization, vectorized reading, and pooling of column vector objects.
         The format of `Row` and `GenericRow` can be implemented based on this class.
      - Use `OrcShim` to do orc related operations.
      - Use both `offset` and `recordsAfterOffset` in `CheckpointedPosition` for better fitting the concept of `RecordReader`. Avoid skipping some of the middle rows when using pushdown filter.
   
   The `OrcColumnarRowFileInputFormat`:
      - This class can add extra fields through `ColumnBatchFactory`, for example, add partition fields.
      - Provides `ColumnarRow` as a view of column batch for high performance.
   
   ## Verifying this change
   
   `OrcColumnarRowFileInputFormatTest`
   
   ## Does this pull request potentially affect one of the following parts:
   
     - Dependencies (does it add or upgrade a dependency): no
     - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
     - The serializers: no
     - The runtime per-record code paths (performance sensitive): no
     - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: no
     - The S3 file system connector: no
   
   ## Documentation
   
     - Does this pull request introduce a new feature? yes
     - If yes, how is the feature documented? JavaDocs
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot commented on pull request #13724: [FLINK-19581][orc] Introduce Orc ColumnarRow File source bulk Format

Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #13724:
URL: https://github.com/apache/flink/pull/13724#issuecomment-713528928


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "bf777e65a831b6c6ee8f1c22ae8f6a232ee87deb",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "bf777e65a831b6c6ee8f1c22ae8f6a232ee87deb",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * bf777e65a831b6c6ee8f1c22ae8f6a232ee87deb UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] JingsongLi commented on pull request #13724: [FLINK-19581][orc] Introduce Orc ColumnarRow File source bulk Format

Posted by GitBox <gi...@apache.org>.
JingsongLi commented on pull request #13724:
URL: https://github.com/apache/flink/pull/13724#issuecomment-723110613


   Thanks @lirui-apache and @StephanEwen for your help and review, merging...


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] JingsongLi edited a comment on pull request #13724: [FLINK-19581][orc] Introduce Orc ColumnarRow File source bulk Format

Posted by GitBox <gi...@apache.org>.
JingsongLi edited a comment on pull request #13724:
URL: https://github.com/apache/flink/pull/13724#issuecomment-722776712


   Hi @StephanEwen, `int[] selected` in `VectorizedRowBatch` is used by Hive vectorization computation engine. (Hive filter node can do zero-copy filter). ORC `RecordReader` does not use this field.
   
   About pushed down predicates, I have done some research and testing, and I think that at least Orc will guarantee the equivalence of `RecordReader.getRowNumber` and `RecordReader.seekToRow`, no matter how many predicates there are. So I think it should be safe.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13724: [FLINK-19581][orc] Introduce Orc ColumnarRow File source bulk Format

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13724:
URL: https://github.com/apache/flink/pull/13724#issuecomment-713528928


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "bf777e65a831b6c6ee8f1c22ae8f6a232ee87deb",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "bf777e65a831b6c6ee8f1c22ae8f6a232ee87deb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f619476bce3df1f3cff0b0579a0888a98d5312cb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8015",
       "triggerID" : "f619476bce3df1f3cff0b0579a0888a98d5312cb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "012beed0dd7796ec1fc67a6e717431540461a7e0",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8957",
       "triggerID" : "012beed0dd7796ec1fc67a6e717431540461a7e0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d25c91253ce75af6d2d87b0bae62bc7dbd575bb",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8979",
       "triggerID" : "3d25c91253ce75af6d2d87b0bae62bc7dbd575bb",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * bf777e65a831b6c6ee8f1c22ae8f6a232ee87deb UNKNOWN
   * 012beed0dd7796ec1fc67a6e717431540461a7e0 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8957) 
   * 3d25c91253ce75af6d2d87b0bae62bc7dbd575bb Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8979) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] JingsongLi merged pull request #13724: [FLINK-19581][orc] Introduce Orc ColumnarRow File source bulk Format

Posted by GitBox <gi...@apache.org>.
JingsongLi merged pull request #13724:
URL: https://github.com/apache/flink/pull/13724


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13724: [FLINK-19581][orc] Introduce Orc ColumnarRow File source bulk Format

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13724:
URL: https://github.com/apache/flink/pull/13724#issuecomment-713528928


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "bf777e65a831b6c6ee8f1c22ae8f6a232ee87deb",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "bf777e65a831b6c6ee8f1c22ae8f6a232ee87deb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f619476bce3df1f3cff0b0579a0888a98d5312cb",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8015",
       "triggerID" : "f619476bce3df1f3cff0b0579a0888a98d5312cb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "012beed0dd7796ec1fc67a6e717431540461a7e0",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "012beed0dd7796ec1fc67a6e717431540461a7e0",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * bf777e65a831b6c6ee8f1c22ae8f6a232ee87deb UNKNOWN
   * f619476bce3df1f3cff0b0579a0888a98d5312cb Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8015) 
   * 012beed0dd7796ec1fc67a6e717431540461a7e0 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13724: [FLINK-19581][orc] Introduce Orc ColumnarRow File source bulk Format

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13724:
URL: https://github.com/apache/flink/pull/13724#issuecomment-713528928


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "bf777e65a831b6c6ee8f1c22ae8f6a232ee87deb",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "bf777e65a831b6c6ee8f1c22ae8f6a232ee87deb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f619476bce3df1f3cff0b0579a0888a98d5312cb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8015",
       "triggerID" : "f619476bce3df1f3cff0b0579a0888a98d5312cb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "012beed0dd7796ec1fc67a6e717431540461a7e0",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8957",
       "triggerID" : "012beed0dd7796ec1fc67a6e717431540461a7e0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d25c91253ce75af6d2d87b0bae62bc7dbd575bb",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3d25c91253ce75af6d2d87b0bae62bc7dbd575bb",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * bf777e65a831b6c6ee8f1c22ae8f6a232ee87deb UNKNOWN
   * 012beed0dd7796ec1fc67a6e717431540461a7e0 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8957) 
   * 3d25c91253ce75af6d2d87b0bae62bc7dbd575bb UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13724: [FLINK-19581][orc] Introduce Orc ColumnarRow File source bulk Format

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13724:
URL: https://github.com/apache/flink/pull/13724#issuecomment-713528928


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "bf777e65a831b6c6ee8f1c22ae8f6a232ee87deb",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "bf777e65a831b6c6ee8f1c22ae8f6a232ee87deb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f619476bce3df1f3cff0b0579a0888a98d5312cb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8015",
       "triggerID" : "f619476bce3df1f3cff0b0579a0888a98d5312cb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "012beed0dd7796ec1fc67a6e717431540461a7e0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8957",
       "triggerID" : "012beed0dd7796ec1fc67a6e717431540461a7e0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d25c91253ce75af6d2d87b0bae62bc7dbd575bb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8979",
       "triggerID" : "3d25c91253ce75af6d2d87b0bae62bc7dbd575bb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "eff771f1557f8fe7dc60a92c22dd1db6a347e047",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9045",
       "triggerID" : "eff771f1557f8fe7dc60a92c22dd1db6a347e047",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * bf777e65a831b6c6ee8f1c22ae8f6a232ee87deb UNKNOWN
   * eff771f1557f8fe7dc60a92c22dd1db6a347e047 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9045) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] lirui-apache commented on a change in pull request #13724: [FLINK-19581][orc] Introduce Orc ColumnarRow File source bulk Format

Posted by GitBox <gi...@apache.org>.
lirui-apache commented on a change in pull request #13724:
URL: https://github.com/apache/flink/pull/13724#discussion_r517778450



##########
File path: flink-formats/flink-orc/src/main/java/org/apache/flink/orc/OrcColumnarRowFileInputFormat.java
##########
@@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.orc;
+
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.connector.file.src.FileSourceSplit;
+import org.apache.flink.connector.file.src.util.Pool;
+import org.apache.flink.orc.shim.OrcShim;
+import org.apache.flink.orc.vector.ColumnBatchFactory;
+import org.apache.flink.orc.vector.OrcVectorizedBatchWrapper;
+import org.apache.flink.table.data.ColumnarRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.vector.ColumnVector;
+import org.apache.flink.table.data.vector.VectorizedColumnBatch;
+import org.apache.flink.table.filesystem.ColumnarRowIterator;
+import org.apache.flink.table.filesystem.PartitionFieldExtractor;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.flink.table.types.logical.RowType;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch;
+import org.apache.orc.TypeDescription;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.orc.OrcSplitReaderUtil.convertToOrcTypeWithPart;
+import static org.apache.flink.orc.OrcSplitReaderUtil.getNonPartNames;
+import static org.apache.flink.orc.OrcSplitReaderUtil.getSelectedOrcFields;
+import static org.apache.flink.orc.vector.AbstractOrcColumnVector.createFlinkVector;
+import static org.apache.flink.orc.vector.AbstractOrcColumnVector.createFlinkVectorFromConstant;
+
+/**
+ * An ORC reader that produces a stream of {@link ColumnarRowData} records.
+ *
+ * <p>This class can add extra fields through {@link ColumnBatchFactory}, for example,
+ * add partition fields, which can be extracted from path. Therefore, the {@link #getProducedType()}
+ * may be different and types of extra fields need to be added.
+ */
+public class OrcColumnarRowFileInputFormat<BatchT, SplitT extends FileSourceSplit> extends
+		AbstractOrcFileInputFormat<RowData, BatchT, SplitT> {
+
+	private static final long serialVersionUID = 1L;
+
+	private final ColumnBatchFactory<BatchT, SplitT> batchFactory;
+	private final RowType projectedOutputType;
+
+	public OrcColumnarRowFileInputFormat(
+			final OrcShim<BatchT> shim,
+			final Configuration hadoopConfig,
+			final TypeDescription schema,
+			final int[] selectedFields,
+			final List<OrcFilters.Predicate> conjunctPredicates,
+			final int batchSize,
+			final ColumnBatchFactory<BatchT, SplitT> batchFactory,
+			final RowType projectedOutputType) {
+		super(shim, hadoopConfig, schema, selectedFields, conjunctPredicates, batchSize);
+		this.batchFactory = batchFactory;
+		this.projectedOutputType = projectedOutputType;
+	}
+
+	@Override
+	public OrcReaderBatch<RowData, BatchT> createReaderBatch(
+			final SplitT split,
+			final OrcVectorizedBatchWrapper<BatchT> orcBatch,
+			final Pool.Recycler<OrcReaderBatch<RowData, BatchT>> recycler,
+			final int batchSize) {
+
+		final VectorizedColumnBatch flinkColumnBatch = batchFactory.create(split, orcBatch.getBatch());
+		return new VectorizedColumnReaderBatch<>(orcBatch, flinkColumnBatch, recycler);
+	}
+
+	@Override
+	public TypeInformation<RowData> getProducedType() {
+		return InternalTypeInfo.of(projectedOutputType);
+	}
+
+	// ------------------------------------------------------------------------
+
+	/**
+	 * One batch of ORC columnar vectors and Flink column vectors.
+	 */
+	private static final class VectorizedColumnReaderBatch<BatchT> extends OrcReaderBatch<RowData, BatchT> {
+
+		private final VectorizedColumnBatch flinkColumnBatch;
+		private final ColumnarRowIterator result;
+
+		VectorizedColumnReaderBatch(
+				final OrcVectorizedBatchWrapper<BatchT> orcBatch,
+				final VectorizedColumnBatch flinkColumnBatch,
+				final Pool.Recycler<OrcReaderBatch<RowData, BatchT>> recycler) {
+			super(orcBatch, recycler);
+			this.flinkColumnBatch = flinkColumnBatch;
+			this.result = new ColumnarRowIterator(new ColumnarRowData(flinkColumnBatch), this::recycle);
+		}
+
+		@Override
+		public RecordIterator<RowData> convertAndGetIterator(
+				final OrcVectorizedBatchWrapper<BatchT> orcBatch,
+				final long startingOffset) {
+			// no copying from the ORC column vectors to the Flink columns vectors necessary,
+			// because they point to the same data arrays internally design
+			int batchSize = orcBatch.size();
+			flinkColumnBatch.setNumRows(batchSize);
+			result.set(batchSize, startingOffset, 0);
+			return result;
+		}
+	}
+
+	/**
+	 * Create a partitioned {@link OrcColumnarRowFileInputFormat}, the partition columns can be
+	 * generated by split.
+	 */
+	public static <SplitT extends FileSourceSplit> OrcColumnarRowFileInputFormat<VectorizedRowBatch, SplitT> createPartitionedFormat(
+			OrcShim<VectorizedRowBatch> shim,
+			Configuration hadoopConfig,
+			RowType tableType,

Review comment:
       Can we make this consistent with `ParquetColumnarRowInputFormat`? E.g. take produced RowType and no need for `selectedFields`?

##########
File path: flink-formats/flink-orc/src/main/java/org/apache/flink/orc/OrcFileSystemFormatFactory.java
##########
@@ -18,45 +18,45 @@
 
 package org.apache.flink.orc;
 
-import org.apache.flink.api.common.io.FileInputFormat;
-import org.apache.flink.api.common.io.InputFormat;
 import org.apache.flink.api.common.serialization.BulkWriter;
-import org.apache.flink.api.common.serialization.Encoder;
 import org.apache.flink.configuration.ConfigOption;
 import org.apache.flink.configuration.ReadableConfig;
-import org.apache.flink.core.fs.FileInputSplit;
-import org.apache.flink.core.fs.Path;
+import org.apache.flink.connector.file.src.FileSourceSplit;
+import org.apache.flink.connector.file.src.reader.BulkFormat;
+import org.apache.flink.orc.shim.OrcShim;
 import org.apache.flink.orc.vector.RowDataVectorizer;
 import org.apache.flink.orc.writer.OrcBulkWriterFactory;
+import org.apache.flink.table.connector.ChangelogMode;
+import org.apache.flink.table.connector.format.BulkDecodingFormat;
+import org.apache.flink.table.connector.format.EncodingFormat;
+import org.apache.flink.table.connector.sink.DynamicTableSink;
+import org.apache.flink.table.connector.source.DynamicTableSource;
 import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.vector.VectorizedColumnBatch;
 import org.apache.flink.table.expressions.Expression;
-import org.apache.flink.table.factories.FileSystemFormatFactory;
+import org.apache.flink.table.expressions.ResolvedExpression;
+import org.apache.flink.table.factories.BulkReaderFormatFactory;
+import org.apache.flink.table.factories.BulkWriterFormatFactory;
+import org.apache.flink.table.factories.DynamicTableFactory;
+import org.apache.flink.table.filesystem.FileSystemOptions;
+import org.apache.flink.table.filesystem.PartitionFieldExtractor;
 import org.apache.flink.table.types.DataType;
 import org.apache.flink.table.types.logical.LogicalType;
 import org.apache.flink.table.types.logical.RowType;
-import org.apache.flink.table.utils.PartitionPathUtils;
 
 import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch;
 import org.apache.orc.TypeDescription;
 
-import java.io.IOException;
 import java.util.ArrayList;
-import java.util.Arrays;
 import java.util.HashSet;
-import java.util.LinkedHashMap;
 import java.util.List;
-import java.util.Optional;
 import java.util.Properties;
 import java.util.Set;
 
-import static org.apache.flink.table.data.vector.VectorizedColumnBatch.DEFAULT_SIZE;
-import static org.apache.flink.table.filesystem.RowPartitionComputer.restorePartValueFromType;
-
 /**
- * Orc {@link FileSystemFormatFactory} for file system.
+ * Orc format factory for file system.
  */
-public class OrcFileSystemFormatFactory implements FileSystemFormatFactory {
+public class OrcFileSystemFormatFactory implements BulkReaderFormatFactory, BulkWriterFormatFactory {

Review comment:
       Rename this class since it's no longer a `FileSystemFormatFactory`?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] JingsongLi commented on a change in pull request #13724: [FLINK-19581][orc] Introduce Orc ColumnarRow File source bulk Format

Posted by GitBox <gi...@apache.org>.
JingsongLi commented on a change in pull request #13724:
URL: https://github.com/apache/flink/pull/13724#discussion_r517821825



##########
File path: flink-formats/flink-orc/src/main/java/org/apache/flink/orc/OrcColumnarRowFileInputFormat.java
##########
@@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.orc;
+
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.connector.file.src.FileSourceSplit;
+import org.apache.flink.connector.file.src.util.Pool;
+import org.apache.flink.orc.shim.OrcShim;
+import org.apache.flink.orc.vector.ColumnBatchFactory;
+import org.apache.flink.orc.vector.OrcVectorizedBatchWrapper;
+import org.apache.flink.table.data.ColumnarRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.vector.ColumnVector;
+import org.apache.flink.table.data.vector.VectorizedColumnBatch;
+import org.apache.flink.table.filesystem.ColumnarRowIterator;
+import org.apache.flink.table.filesystem.PartitionFieldExtractor;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.flink.table.types.logical.RowType;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch;
+import org.apache.orc.TypeDescription;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.orc.OrcSplitReaderUtil.convertToOrcTypeWithPart;
+import static org.apache.flink.orc.OrcSplitReaderUtil.getNonPartNames;
+import static org.apache.flink.orc.OrcSplitReaderUtil.getSelectedOrcFields;
+import static org.apache.flink.orc.vector.AbstractOrcColumnVector.createFlinkVector;
+import static org.apache.flink.orc.vector.AbstractOrcColumnVector.createFlinkVectorFromConstant;
+
+/**
+ * An ORC reader that produces a stream of {@link ColumnarRowData} records.
+ *
+ * <p>This class can add extra fields through {@link ColumnBatchFactory}, for example,
+ * add partition fields, which can be extracted from path. Therefore, the {@link #getProducedType()}
+ * may be different and types of extra fields need to be added.
+ */
+public class OrcColumnarRowFileInputFormat<BatchT, SplitT extends FileSourceSplit> extends
+		AbstractOrcFileInputFormat<RowData, BatchT, SplitT> {
+
+	private static final long serialVersionUID = 1L;
+
+	private final ColumnBatchFactory<BatchT, SplitT> batchFactory;
+	private final RowType projectedOutputType;
+
+	public OrcColumnarRowFileInputFormat(
+			final OrcShim<BatchT> shim,
+			final Configuration hadoopConfig,
+			final TypeDescription schema,
+			final int[] selectedFields,
+			final List<OrcFilters.Predicate> conjunctPredicates,
+			final int batchSize,
+			final ColumnBatchFactory<BatchT, SplitT> batchFactory,
+			final RowType projectedOutputType) {
+		super(shim, hadoopConfig, schema, selectedFields, conjunctPredicates, batchSize);
+		this.batchFactory = batchFactory;
+		this.projectedOutputType = projectedOutputType;
+	}
+
+	@Override
+	public OrcReaderBatch<RowData, BatchT> createReaderBatch(
+			final SplitT split,
+			final OrcVectorizedBatchWrapper<BatchT> orcBatch,
+			final Pool.Recycler<OrcReaderBatch<RowData, BatchT>> recycler,
+			final int batchSize) {
+
+		final VectorizedColumnBatch flinkColumnBatch = batchFactory.create(split, orcBatch.getBatch());
+		return new VectorizedColumnReaderBatch<>(orcBatch, flinkColumnBatch, recycler);
+	}
+
+	@Override
+	public TypeInformation<RowData> getProducedType() {
+		return InternalTypeInfo.of(projectedOutputType);
+	}
+
+	// ------------------------------------------------------------------------
+
+	/**
+	 * One batch of ORC columnar vectors and Flink column vectors.
+	 */
+	private static final class VectorizedColumnReaderBatch<BatchT> extends OrcReaderBatch<RowData, BatchT> {
+
+		private final VectorizedColumnBatch flinkColumnBatch;
+		private final ColumnarRowIterator result;
+
+		VectorizedColumnReaderBatch(
+				final OrcVectorizedBatchWrapper<BatchT> orcBatch,
+				final VectorizedColumnBatch flinkColumnBatch,
+				final Pool.Recycler<OrcReaderBatch<RowData, BatchT>> recycler) {
+			super(orcBatch, recycler);
+			this.flinkColumnBatch = flinkColumnBatch;
+			this.result = new ColumnarRowIterator(new ColumnarRowData(flinkColumnBatch), this::recycle);
+		}
+
+		@Override
+		public RecordIterator<RowData> convertAndGetIterator(
+				final OrcVectorizedBatchWrapper<BatchT> orcBatch,
+				final long startingOffset) {
+			// no copying from the ORC column vectors to the Flink columns vectors necessary,
+			// because they point to the same data arrays internally design
+			int batchSize = orcBatch.size();
+			flinkColumnBatch.setNumRows(batchSize);
+			result.set(batchSize, startingOffset, 0);
+			return result;
+		}
+	}
+
+	/**
+	 * Create a partitioned {@link OrcColumnarRowFileInputFormat}, the partition columns can be
+	 * generated by split.
+	 */
+	public static <SplitT extends FileSourceSplit> OrcColumnarRowFileInputFormat<VectorizedRowBatch, SplitT> createPartitionedFormat(
+			OrcShim<VectorizedRowBatch> shim,
+			Configuration hadoopConfig,
+			RowType tableType,

Review comment:
       We can not, because orc needs full schema to reader, but parquet does not need.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13724: [FLINK-19581][orc] Introduce Orc ColumnarRow File source bulk Format

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13724:
URL: https://github.com/apache/flink/pull/13724#issuecomment-713528928


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "bf777e65a831b6c6ee8f1c22ae8f6a232ee87deb",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "bf777e65a831b6c6ee8f1c22ae8f6a232ee87deb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f619476bce3df1f3cff0b0579a0888a98d5312cb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8015",
       "triggerID" : "f619476bce3df1f3cff0b0579a0888a98d5312cb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "012beed0dd7796ec1fc67a6e717431540461a7e0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8957",
       "triggerID" : "012beed0dd7796ec1fc67a6e717431540461a7e0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d25c91253ce75af6d2d87b0bae62bc7dbd575bb",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8979",
       "triggerID" : "3d25c91253ce75af6d2d87b0bae62bc7dbd575bb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "eff771f1557f8fe7dc60a92c22dd1db6a347e047",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9045",
       "triggerID" : "eff771f1557f8fe7dc60a92c22dd1db6a347e047",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * bf777e65a831b6c6ee8f1c22ae8f6a232ee87deb UNKNOWN
   * 3d25c91253ce75af6d2d87b0bae62bc7dbd575bb Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8979) 
   * eff771f1557f8fe7dc60a92c22dd1db6a347e047 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9045) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot commented on pull request #13724: [FLINK-19581][orc] Introduce Orc ColumnarRow File source bulk Format

Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #13724:
URL: https://github.com/apache/flink/pull/13724#issuecomment-713514997


   Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
   to review your pull request. We will use this comment to track the progress of the review.
   
   
   ## Automated Checks
   Last check on commit bf777e65a831b6c6ee8f1c22ae8f6a232ee87deb (Wed Oct 21 11:54:31 UTC 2020)
   
   **Warnings:**
    * No documentation files were touched! Remember to keep the Flink docs up to date!
   
   
   <sub>Mention the bot in a comment to re-run the automated checks.</sub>
   ## Review Progress
   
   * ❓ 1. The [description] looks good.
   * ❓ 2. There is [consensus] that the contribution should go into to Flink.
   * ❓ 3. Needs [attention] from.
   * ❓ 4. The change fits into the overall [architecture].
   * ❓ 5. Overall code [quality] is good.
   
   Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
    The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
    - `@flinkbot approve all` to approve all aspects
    - `@flinkbot approve-until architecture` to approve everything until `architecture`
    - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
    - `@flinkbot disapprove architecture` to remove an approval you gave earlier
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] JingsongLi commented on pull request #13724: [FLINK-19581][orc] Introduce Orc ColumnarRow File source bulk Format

Posted by GitBox <gi...@apache.org>.
JingsongLi commented on pull request #13724:
URL: https://github.com/apache/flink/pull/13724#issuecomment-722180598


   > Thanks @JingsongLi , I left some minor comments. Another suggestion is to make the naming convention consistent between parquet and orc. For example, the base bulk format class of orc is `AbstractOrcFileInputFormat`, while the counterpart in parquet is named `ParquetVectorizedInputFormat`. Making them consistent should make the code easier to understand.
   
   I think it is better to add `abstract` to abstract class, but the name of `ParquetVectorizedInputFormat` is too long.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13724: [FLINK-19581][orc] Introduce Orc ColumnarRow File source bulk Format

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13724:
URL: https://github.com/apache/flink/pull/13724#issuecomment-713528928


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "bf777e65a831b6c6ee8f1c22ae8f6a232ee87deb",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "bf777e65a831b6c6ee8f1c22ae8f6a232ee87deb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f619476bce3df1f3cff0b0579a0888a98d5312cb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8015",
       "triggerID" : "f619476bce3df1f3cff0b0579a0888a98d5312cb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "012beed0dd7796ec1fc67a6e717431540461a7e0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8957",
       "triggerID" : "012beed0dd7796ec1fc67a6e717431540461a7e0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d25c91253ce75af6d2d87b0bae62bc7dbd575bb",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8979",
       "triggerID" : "3d25c91253ce75af6d2d87b0bae62bc7dbd575bb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "eff771f1557f8fe7dc60a92c22dd1db6a347e047",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "eff771f1557f8fe7dc60a92c22dd1db6a347e047",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * bf777e65a831b6c6ee8f1c22ae8f6a232ee87deb UNKNOWN
   * 3d25c91253ce75af6d2d87b0bae62bc7dbd575bb Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8979) 
   * eff771f1557f8fe7dc60a92c22dd1db6a347e047 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] StephanEwen commented on pull request #13724: [FLINK-19581][orc] Introduce Orc ColumnarRow File source bulk Format

Posted by GitBox <gi...@apache.org>.
StephanEwen commented on pull request #13724:
URL: https://github.com/apache/flink/pull/13724#issuecomment-723024204


   I see. In that case, +1 to merge this.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] lirui-apache commented on a change in pull request #13724: [FLINK-19581][orc] Introduce Orc ColumnarRow File source bulk Format

Posted by GitBox <gi...@apache.org>.
lirui-apache commented on a change in pull request #13724:
URL: https://github.com/apache/flink/pull/13724#discussion_r517827555



##########
File path: flink-formats/flink-orc/src/main/java/org/apache/flink/orc/OrcColumnarRowFileInputFormat.java
##########
@@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.orc;
+
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.connector.file.src.FileSourceSplit;
+import org.apache.flink.connector.file.src.util.Pool;
+import org.apache.flink.orc.shim.OrcShim;
+import org.apache.flink.orc.vector.ColumnBatchFactory;
+import org.apache.flink.orc.vector.OrcVectorizedBatchWrapper;
+import org.apache.flink.table.data.ColumnarRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.vector.ColumnVector;
+import org.apache.flink.table.data.vector.VectorizedColumnBatch;
+import org.apache.flink.table.filesystem.ColumnarRowIterator;
+import org.apache.flink.table.filesystem.PartitionFieldExtractor;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.flink.table.types.logical.RowType;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch;
+import org.apache.orc.TypeDescription;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.orc.OrcSplitReaderUtil.convertToOrcTypeWithPart;
+import static org.apache.flink.orc.OrcSplitReaderUtil.getNonPartNames;
+import static org.apache.flink.orc.OrcSplitReaderUtil.getSelectedOrcFields;
+import static org.apache.flink.orc.vector.AbstractOrcColumnVector.createFlinkVector;
+import static org.apache.flink.orc.vector.AbstractOrcColumnVector.createFlinkVectorFromConstant;
+
+/**
+ * An ORC reader that produces a stream of {@link ColumnarRowData} records.
+ *
+ * <p>This class can add extra fields through {@link ColumnBatchFactory}, for example,
+ * add partition fields, which can be extracted from path. Therefore, the {@link #getProducedType()}
+ * may be different and types of extra fields need to be added.
+ */
+public class OrcColumnarRowFileInputFormat<BatchT, SplitT extends FileSourceSplit> extends
+		AbstractOrcFileInputFormat<RowData, BatchT, SplitT> {
+
+	private static final long serialVersionUID = 1L;
+
+	private final ColumnBatchFactory<BatchT, SplitT> batchFactory;
+	private final RowType projectedOutputType;
+
+	public OrcColumnarRowFileInputFormat(
+			final OrcShim<BatchT> shim,
+			final Configuration hadoopConfig,
+			final TypeDescription schema,
+			final int[] selectedFields,
+			final List<OrcFilters.Predicate> conjunctPredicates,
+			final int batchSize,
+			final ColumnBatchFactory<BatchT, SplitT> batchFactory,
+			final RowType projectedOutputType) {
+		super(shim, hadoopConfig, schema, selectedFields, conjunctPredicates, batchSize);
+		this.batchFactory = batchFactory;
+		this.projectedOutputType = projectedOutputType;
+	}
+
+	@Override
+	public OrcReaderBatch<RowData, BatchT> createReaderBatch(
+			final SplitT split,
+			final OrcVectorizedBatchWrapper<BatchT> orcBatch,
+			final Pool.Recycler<OrcReaderBatch<RowData, BatchT>> recycler,
+			final int batchSize) {
+
+		final VectorizedColumnBatch flinkColumnBatch = batchFactory.create(split, orcBatch.getBatch());
+		return new VectorizedColumnReaderBatch<>(orcBatch, flinkColumnBatch, recycler);
+	}
+
+	@Override
+	public TypeInformation<RowData> getProducedType() {
+		return InternalTypeInfo.of(projectedOutputType);
+	}
+
+	// ------------------------------------------------------------------------
+
+	/**
+	 * One batch of ORC columnar vectors and Flink column vectors.
+	 */
+	private static final class VectorizedColumnReaderBatch<BatchT> extends OrcReaderBatch<RowData, BatchT> {
+
+		private final VectorizedColumnBatch flinkColumnBatch;
+		private final ColumnarRowIterator result;
+
+		VectorizedColumnReaderBatch(
+				final OrcVectorizedBatchWrapper<BatchT> orcBatch,
+				final VectorizedColumnBatch flinkColumnBatch,
+				final Pool.Recycler<OrcReaderBatch<RowData, BatchT>> recycler) {
+			super(orcBatch, recycler);
+			this.flinkColumnBatch = flinkColumnBatch;
+			this.result = new ColumnarRowIterator(new ColumnarRowData(flinkColumnBatch), this::recycle);
+		}
+
+		@Override
+		public RecordIterator<RowData> convertAndGetIterator(
+				final OrcVectorizedBatchWrapper<BatchT> orcBatch,
+				final long startingOffset) {
+			// no copying from the ORC column vectors to the Flink columns vectors necessary,
+			// because they point to the same data arrays internally design
+			int batchSize = orcBatch.size();
+			flinkColumnBatch.setNumRows(batchSize);
+			result.set(batchSize, startingOffset, 0);
+			return result;
+		}
+	}
+
+	/**
+	 * Create a partitioned {@link OrcColumnarRowFileInputFormat}, the partition columns can be
+	 * generated by split.
+	 */
+	public static <SplitT extends FileSourceSplit> OrcColumnarRowFileInputFormat<VectorizedRowBatch, SplitT> createPartitionedFormat(
+			OrcShim<VectorizedRowBatch> shim,
+			Configuration hadoopConfig,
+			RowType tableType,

Review comment:
       OK :rofl:




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13724: [FLINK-19581][orc] Introduce Orc ColumnarRow File source bulk Format

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13724:
URL: https://github.com/apache/flink/pull/13724#issuecomment-713528928


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "bf777e65a831b6c6ee8f1c22ae8f6a232ee87deb",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "bf777e65a831b6c6ee8f1c22ae8f6a232ee87deb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f619476bce3df1f3cff0b0579a0888a98d5312cb",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8015",
       "triggerID" : "f619476bce3df1f3cff0b0579a0888a98d5312cb",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * bf777e65a831b6c6ee8f1c22ae8f6a232ee87deb UNKNOWN
   * f619476bce3df1f3cff0b0579a0888a98d5312cb Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8015) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13724: [FLINK-19581][orc] Introduce Orc ColumnarRow File source bulk Format

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13724:
URL: https://github.com/apache/flink/pull/13724#issuecomment-713528928


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "bf777e65a831b6c6ee8f1c22ae8f6a232ee87deb",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "bf777e65a831b6c6ee8f1c22ae8f6a232ee87deb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f619476bce3df1f3cff0b0579a0888a98d5312cb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8015",
       "triggerID" : "f619476bce3df1f3cff0b0579a0888a98d5312cb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "012beed0dd7796ec1fc67a6e717431540461a7e0",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8957",
       "triggerID" : "012beed0dd7796ec1fc67a6e717431540461a7e0",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d25c91253ce75af6d2d87b0bae62bc7dbd575bb",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8979",
       "triggerID" : "3d25c91253ce75af6d2d87b0bae62bc7dbd575bb",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * bf777e65a831b6c6ee8f1c22ae8f6a232ee87deb UNKNOWN
   * 3d25c91253ce75af6d2d87b0bae62bc7dbd575bb Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8979) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13724: [FLINK-19581][orc] Introduce Orc ColumnarRow File source bulk Format

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13724:
URL: https://github.com/apache/flink/pull/13724#issuecomment-713528928


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "bf777e65a831b6c6ee8f1c22ae8f6a232ee87deb",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "bf777e65a831b6c6ee8f1c22ae8f6a232ee87deb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f619476bce3df1f3cff0b0579a0888a98d5312cb",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f619476bce3df1f3cff0b0579a0888a98d5312cb",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * bf777e65a831b6c6ee8f1c22ae8f6a232ee87deb UNKNOWN
   * f619476bce3df1f3cff0b0579a0888a98d5312cb UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] JingsongLi commented on a change in pull request #13724: [FLINK-19581][orc] Introduce Orc ColumnarRow File source bulk Format

Posted by GitBox <gi...@apache.org>.
JingsongLi commented on a change in pull request #13724:
URL: https://github.com/apache/flink/pull/13724#discussion_r517822079



##########
File path: flink-formats/flink-orc/src/main/java/org/apache/flink/orc/OrcColumnarRowFileInputFormat.java
##########
@@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.orc;
+
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.connector.file.src.FileSourceSplit;
+import org.apache.flink.connector.file.src.util.Pool;
+import org.apache.flink.orc.shim.OrcShim;
+import org.apache.flink.orc.vector.ColumnBatchFactory;
+import org.apache.flink.orc.vector.OrcVectorizedBatchWrapper;
+import org.apache.flink.table.data.ColumnarRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.vector.ColumnVector;
+import org.apache.flink.table.data.vector.VectorizedColumnBatch;
+import org.apache.flink.table.filesystem.ColumnarRowIterator;
+import org.apache.flink.table.filesystem.PartitionFieldExtractor;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.flink.table.types.logical.RowType;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch;
+import org.apache.orc.TypeDescription;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.orc.OrcSplitReaderUtil.convertToOrcTypeWithPart;
+import static org.apache.flink.orc.OrcSplitReaderUtil.getNonPartNames;
+import static org.apache.flink.orc.OrcSplitReaderUtil.getSelectedOrcFields;
+import static org.apache.flink.orc.vector.AbstractOrcColumnVector.createFlinkVector;
+import static org.apache.flink.orc.vector.AbstractOrcColumnVector.createFlinkVectorFromConstant;
+
+/**
+ * An ORC reader that produces a stream of {@link ColumnarRowData} records.
+ *
+ * <p>This class can add extra fields through {@link ColumnBatchFactory}, for example,
+ * add partition fields, which can be extracted from path. Therefore, the {@link #getProducedType()}
+ * may be different and types of extra fields need to be added.
+ */
+public class OrcColumnarRowFileInputFormat<BatchT, SplitT extends FileSourceSplit> extends
+		AbstractOrcFileInputFormat<RowData, BatchT, SplitT> {
+
+	private static final long serialVersionUID = 1L;
+
+	private final ColumnBatchFactory<BatchT, SplitT> batchFactory;
+	private final RowType projectedOutputType;
+
+	public OrcColumnarRowFileInputFormat(
+			final OrcShim<BatchT> shim,
+			final Configuration hadoopConfig,
+			final TypeDescription schema,
+			final int[] selectedFields,
+			final List<OrcFilters.Predicate> conjunctPredicates,
+			final int batchSize,
+			final ColumnBatchFactory<BatchT, SplitT> batchFactory,
+			final RowType projectedOutputType) {
+		super(shim, hadoopConfig, schema, selectedFields, conjunctPredicates, batchSize);
+		this.batchFactory = batchFactory;
+		this.projectedOutputType = projectedOutputType;
+	}
+
+	@Override
+	public OrcReaderBatch<RowData, BatchT> createReaderBatch(
+			final SplitT split,
+			final OrcVectorizedBatchWrapper<BatchT> orcBatch,
+			final Pool.Recycler<OrcReaderBatch<RowData, BatchT>> recycler,
+			final int batchSize) {
+
+		final VectorizedColumnBatch flinkColumnBatch = batchFactory.create(split, orcBatch.getBatch());
+		return new VectorizedColumnReaderBatch<>(orcBatch, flinkColumnBatch, recycler);
+	}
+
+	@Override
+	public TypeInformation<RowData> getProducedType() {
+		return InternalTypeInfo.of(projectedOutputType);
+	}
+
+	// ------------------------------------------------------------------------
+
+	/**
+	 * One batch of ORC columnar vectors and Flink column vectors.
+	 */
+	private static final class VectorizedColumnReaderBatch<BatchT> extends OrcReaderBatch<RowData, BatchT> {
+
+		private final VectorizedColumnBatch flinkColumnBatch;
+		private final ColumnarRowIterator result;
+
+		VectorizedColumnReaderBatch(
+				final OrcVectorizedBatchWrapper<BatchT> orcBatch,
+				final VectorizedColumnBatch flinkColumnBatch,
+				final Pool.Recycler<OrcReaderBatch<RowData, BatchT>> recycler) {
+			super(orcBatch, recycler);
+			this.flinkColumnBatch = flinkColumnBatch;
+			this.result = new ColumnarRowIterator(new ColumnarRowData(flinkColumnBatch), this::recycle);
+		}
+
+		@Override
+		public RecordIterator<RowData> convertAndGetIterator(
+				final OrcVectorizedBatchWrapper<BatchT> orcBatch,
+				final long startingOffset) {
+			// no copying from the ORC column vectors to the Flink columns vectors necessary,
+			// because they point to the same data arrays internally design
+			int batchSize = orcBatch.size();
+			flinkColumnBatch.setNumRows(batchSize);
+			result.set(batchSize, startingOffset, 0);
+			return result;
+		}
+	}
+
+	/**
+	 * Create a partitioned {@link OrcColumnarRowFileInputFormat}, the partition columns can be
+	 * generated by split.
+	 */
+	public static <SplitT extends FileSourceSplit> OrcColumnarRowFileInputFormat<VectorizedRowBatch, SplitT> createPartitionedFormat(
+			OrcShim<VectorizedRowBatch> shim,
+			Configuration hadoopConfig,
+			RowType tableType,

Review comment:
       And `AbstractOrcFileInputFormat` is a table-free class.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13724: [FLINK-19581][orc] Introduce Orc ColumnarRow File source bulk Format

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13724:
URL: https://github.com/apache/flink/pull/13724#issuecomment-713528928


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "bf777e65a831b6c6ee8f1c22ae8f6a232ee87deb",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "bf777e65a831b6c6ee8f1c22ae8f6a232ee87deb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f619476bce3df1f3cff0b0579a0888a98d5312cb",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8015",
       "triggerID" : "f619476bce3df1f3cff0b0579a0888a98d5312cb",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * bf777e65a831b6c6ee8f1c22ae8f6a232ee87deb UNKNOWN
   * f619476bce3df1f3cff0b0579a0888a98d5312cb Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8015) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] StephanEwen commented on pull request #13724: [FLINK-19581][orc] Introduce Orc ColumnarRow File source bulk Format

Posted by GitBox <gi...@apache.org>.
StephanEwen commented on pull request #13724:
URL: https://github.com/apache/flink/pull/13724#issuecomment-722365033


   This looks pretty good to me, we could merge it like it is.
   
   One idea for an improvement would be to not use the "skipRecordsCount" at all here. I fear this can lead to surprised with ORC due to pushed down predicates. If during after update, the predicate would be more selective, then ORC itself would filter more rows and we would skip too many later.
   
   What we could do is the following: The `VectorizedRowBatch` has the `int[] selected` array, which has the positions of the rows. We could also pass that array to the `ColumnarRowIterator`, instead of the `startingOffset`.
   When returning the next record, it would set that position to the result, rather than incrementing the skipCount.
   
   What do you think?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] JingsongLi commented on pull request #13724: [FLINK-19581][orc] Introduce Orc ColumnarRow File source bulk Format

Posted by GitBox <gi...@apache.org>.
JingsongLi commented on pull request #13724:
URL: https://github.com/apache/flink/pull/13724#issuecomment-722776712


   Hi @StephanEwen, `int[] selected` in `VectorizedRowBatch` is used by Hive vectorization computation engine. (Hive filter node can do zero-copy filter). ORC `RecordReader` does not use this field.
   
   About pushed down predicates, I have done some research and testing, and I think that at least Orc will guarantee the equivalence of `RecordReader.nextBatch` and `RecordReader.seekToRow`, no matter how many predicates there are. So I think it should be safe.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #13724: [FLINK-19581][orc] Introduce Orc ColumnarRow File source bulk Format

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #13724:
URL: https://github.com/apache/flink/pull/13724#issuecomment-713528928


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "bf777e65a831b6c6ee8f1c22ae8f6a232ee87deb",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "bf777e65a831b6c6ee8f1c22ae8f6a232ee87deb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f619476bce3df1f3cff0b0579a0888a98d5312cb",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8015",
       "triggerID" : "f619476bce3df1f3cff0b0579a0888a98d5312cb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "012beed0dd7796ec1fc67a6e717431540461a7e0",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8957",
       "triggerID" : "012beed0dd7796ec1fc67a6e717431540461a7e0",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * bf777e65a831b6c6ee8f1c22ae8f6a232ee87deb UNKNOWN
   * f619476bce3df1f3cff0b0579a0888a98d5312cb Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8015) 
   * 012beed0dd7796ec1fc67a6e717431540461a7e0 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8957) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] JingsongLi commented on a change in pull request #13724: [FLINK-19581][orc] Introduce Orc ColumnarRow File source bulk Format

Posted by GitBox <gi...@apache.org>.
JingsongLi commented on a change in pull request #13724:
URL: https://github.com/apache/flink/pull/13724#discussion_r517822283



##########
File path: flink-formats/flink-orc/src/main/java/org/apache/flink/orc/OrcFileSystemFormatFactory.java
##########
@@ -18,45 +18,45 @@
 
 package org.apache.flink.orc;
 
-import org.apache.flink.api.common.io.FileInputFormat;
-import org.apache.flink.api.common.io.InputFormat;
 import org.apache.flink.api.common.serialization.BulkWriter;
-import org.apache.flink.api.common.serialization.Encoder;
 import org.apache.flink.configuration.ConfigOption;
 import org.apache.flink.configuration.ReadableConfig;
-import org.apache.flink.core.fs.FileInputSplit;
-import org.apache.flink.core.fs.Path;
+import org.apache.flink.connector.file.src.FileSourceSplit;
+import org.apache.flink.connector.file.src.reader.BulkFormat;
+import org.apache.flink.orc.shim.OrcShim;
 import org.apache.flink.orc.vector.RowDataVectorizer;
 import org.apache.flink.orc.writer.OrcBulkWriterFactory;
+import org.apache.flink.table.connector.ChangelogMode;
+import org.apache.flink.table.connector.format.BulkDecodingFormat;
+import org.apache.flink.table.connector.format.EncodingFormat;
+import org.apache.flink.table.connector.sink.DynamicTableSink;
+import org.apache.flink.table.connector.source.DynamicTableSource;
 import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.vector.VectorizedColumnBatch;
 import org.apache.flink.table.expressions.Expression;
-import org.apache.flink.table.factories.FileSystemFormatFactory;
+import org.apache.flink.table.expressions.ResolvedExpression;
+import org.apache.flink.table.factories.BulkReaderFormatFactory;
+import org.apache.flink.table.factories.BulkWriterFormatFactory;
+import org.apache.flink.table.factories.DynamicTableFactory;
+import org.apache.flink.table.filesystem.FileSystemOptions;
+import org.apache.flink.table.filesystem.PartitionFieldExtractor;
 import org.apache.flink.table.types.DataType;
 import org.apache.flink.table.types.logical.LogicalType;
 import org.apache.flink.table.types.logical.RowType;
-import org.apache.flink.table.utils.PartitionPathUtils;
 
 import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch;
 import org.apache.orc.TypeDescription;
 
-import java.io.IOException;
 import java.util.ArrayList;
-import java.util.Arrays;
 import java.util.HashSet;
-import java.util.LinkedHashMap;
 import java.util.List;
-import java.util.Optional;
 import java.util.Properties;
 import java.util.Set;
 
-import static org.apache.flink.table.data.vector.VectorizedColumnBatch.DEFAULT_SIZE;
-import static org.apache.flink.table.filesystem.RowPartitionComputer.restorePartValueFromType;
-
 /**
- * Orc {@link FileSystemFormatFactory} for file system.
+ * Orc format factory for file system.
  */
-public class OrcFileSystemFormatFactory implements FileSystemFormatFactory {
+public class OrcFileSystemFormatFactory implements BulkReaderFormatFactory, BulkWriterFormatFactory {

Review comment:
       I think `OrcFileFormatFactory` is better




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] JingsongLi commented on a change in pull request #13724: [FLINK-19581][orc] Introduce Orc ColumnarRow File source bulk Format

Posted by GitBox <gi...@apache.org>.
JingsongLi commented on a change in pull request #13724:
URL: https://github.com/apache/flink/pull/13724#discussion_r517125719



##########
File path: flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/filesystem/PartitionFieldExtractor.java
##########
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.filesystem;
+
+import org.apache.flink.connector.file.src.FileSourceSplit;
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.flink.table.utils.PartitionPathUtils;
+
+import java.io.Serializable;
+import java.util.LinkedHashMap;
+
+/**
+ * Interface to extract partition field from split.
+ */
+@FunctionalInterface
+public interface PartitionFieldExtractor<T extends FileSourceSplit> extends Serializable {

Review comment:
       This will be introduced in https://github.com/apache/flink/pull/13919




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org