You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2021/02/05 23:15:02 UTC
[GitHub] [druid] jihoonson commented on a change in pull request #10830: HTTP inputSource and firehose should support only HTTP and HTTPS; HDFS inputSource and firehose should support only HDFS
jihoonson commented on a change in pull request #10830:
URL: https://github.com/apache/druid/pull/10830#discussion_r571301390
##########
File path: extensions-core/hdfs-storage/src/main/java/org/apache/druid/inputsource/hdfs/HdfsInputSource.java
##########
@@ -101,20 +102,25 @@ public HdfsInputSource(
return paths;
}
- public static Collection<Path> getPaths(List<String> inputPaths, Configuration configuration) throws IOException
+ public static Collection<Path> getPaths(List<String> inputPathStrings, Configuration configuration) throws IOException
{
- if (inputPaths.isEmpty()) {
+ if (inputPathStrings.isEmpty()) {
return Collections.emptySet();
}
// Use FileInputFormat to read splits. To do this, we need to make a fake Job.
Job job = Job.getInstance(configuration);
// Add paths to the fake JobContext.
- for (String inputPath : inputPaths) {
+ for (String inputPath : inputPathStrings) {
FileInputFormat.addInputPaths(job, inputPath);
}
+ final Path[] inputPaths = FileInputFormat.getInputPaths(job);
+ if (Arrays.stream(inputPaths).anyMatch(path -> !"hdfs".equalsIgnoreCase(path.toUri().getScheme()))) {
+ throw new IllegalArgumentException("Input paths must be the HDFS path");
+ }
+
Review comment:
@suneet-s @a2l007 good point. I realized that this is a [documented feature](https://druid.apache.org/docs/0.20.0/ingestion/native-batch.html#hdfs-input-source) that should be supported. The concern here is that users can use whatever protocol they want if it's supported by the hdfs client. Instead of restricting it to only hdfs, I added new configs for http and hdfs inputSources so that users can configure what protocols they want to allow.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org