You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by je...@apache.org on 2019/03/08 23:12:30 UTC
[incubator-pinot] branch flexibleFiles created (now 21619e8)
This is an automated email from the ASF dual-hosted git repository.
jenniferdai pushed a change to branch flexibleFiles
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git.
at 21619e8 Adding flexible file paths
This branch includes the following new commits:
new 21619e8 Adding flexible file paths
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[incubator-pinot] 01/01: Adding flexible file paths
Posted by je...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jenniferdai pushed a commit to branch flexibleFiles
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
commit 21619e83c8cb02d73eea634617bf7f777cf83b5b
Author: Jennifer Dai <jd...@linkedin.com>
AuthorDate: Fri Mar 8 15:12:16 2019 -0800
Adding flexible file paths
---
.../src/main/java/org/apache/pinot/hadoop/job/SegmentCreationJob.java | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/pinot-hadoop/src/main/java/org/apache/pinot/hadoop/job/SegmentCreationJob.java b/pinot-hadoop/src/main/java/org/apache/pinot/hadoop/job/SegmentCreationJob.java
index 5b3ecad..6939fd7 100644
--- a/pinot-hadoop/src/main/java/org/apache/pinot/hadoop/job/SegmentCreationJob.java
+++ b/pinot-hadoop/src/main/java/org/apache/pinot/hadoop/job/SegmentCreationJob.java
@@ -104,6 +104,10 @@ public class SegmentCreationJob extends BaseSegmentJob {
@Override
protected boolean isDataFile(String fileName) {
+ // Other files may have different extensions, eg: orc can have no extension
+ if (_properties.getProperty(JobConfigConstants.RECORD_READER_PATH) != null) {
+ return true;
+ }
return fileName.endsWith(".avro") || fileName.endsWith(".csv") || fileName.endsWith(".json") || fileName
.endsWith(".thrift");
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org