You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@camel.apache.org by "Kengo Seki (Jira)" <ji...@apache.org> on 2023/03/29 23:16:00 UTC
[jira] [Assigned] (CAMEL-18776) camel-hdfs - Fix HdfsNormalFileHandler to handle temporary file path correctly
[ https://issues.apache.org/jira/browse/CAMEL-18776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kengo Seki reassigned CAMEL-18776:
----------------------------------
Assignee: Kengo Seki
> camel-hdfs - Fix HdfsNormalFileHandler to handle temporary file path correctly
> ------------------------------------------------------------------------------
>
> Key: CAMEL-18776
> URL: https://issues.apache.org/jira/browse/CAMEL-18776
> Project: Camel
> Issue Type: Bug
> Components: camel-hdfs
> Reporter: Kengo Seki
> Assignee: Kengo Seki
> Priority: Minor
> Fix For: 3.14.7, 3.18.5, 3.20.0
>
>
> Currently, HDFS consumer doesn't seem to work with normal files.
> Given the following file on HDFS,
> {code}
> $ hdfs dfs -ls /tmp/input
> Found 1 items
> -rw-r--r-- 3 sekikn supergroup 4 2022-11-30 20:40 /tmp/input/foo.txt
> $ hdfs dfs -cat /tmp/input/foo.txt
> bar
> {code}
> I tried to run the following route so as to copy normal files from HDFS to local filesystem.
> {code}
> $ cat put.java
> import org.apache.camel.builder.RouteBuilder;
> public class put extends RouteBuilder {
> @Override
> public void configure() throws Exception {
> from("hdfs:192.168.2.2:9000/tmp/input/?repeatCount=1").log("${body}");
> }
> }
> {code}
> But I came across the following error.
> {code}
> $ camel run put.java
> ...
> 2022-11-30 20:46:25.151 WARN 2718863 --- [9000/tmp/input/] apache.camel.component.hdfs.HdfsConsumer : Failed polling endpoint: hdfs://192.168.2.2:9000/tmp/input/?repeatCount=1. Will try again at next poll. Caused by: [org.apache.camel.RuntimeCamelException - java.io.FileNotFoundException: /tmp/foo.txt.opened5427239805734101292.hdfs/foo.txt.opened (Not a directory)]
> org.apache.camel.RuntimeCamelException: java.io.FileNotFoundException: /tmp/foo.txt.opened5427239805734101292.hdfs/foo.txt.opened (Not a directory)
> at org.apache.camel.component.hdfs.HdfsNormalFileHandler.createInputStream(HdfsNormalFileHandler.java:98) ~[camel-hdfs-3.19.0.jar:3.19.0]
> at org.apache.camel.component.hdfs.HdfsNormalFileHandler.createInputStream(HdfsNormalFileHandler.java:38) ~[camel-hdfs-3.19.0.jar:3.19.0]
> at org.apache.camel.component.hdfs.HdfsFileType.createInputStream(HdfsFileType.java:46) ~[camel-hdfs-3.19.0.jar:3.19.0]
> at org.apache.camel.component.hdfs.HdfsInputStream.createInputStream(HdfsInputStream.java:71) ~[camel-hdfs-3.19.0.jar:3.19.0]
> at org.apache.camel.component.hdfs.HdfsConsumer.asHdfsFile(HdfsConsumer.java:246) ~[camel-hdfs-3.19.0.jar:3.19.0]
> ...
> Caused by: java.io.FileNotFoundException: /tmp/foo.txt.opened5427239805734101292.hdfs/foo.txt.opened (Not a directory)
> at java.io.FileInputStream.open0(Native Method) ~[?:?]
> at java.io.FileInputStream.open(FileInputStream.java:219) ~[?:?]
> at java.io.FileInputStream.<init>(FileInputStream.java:157) ~[?:?]
> at org.apache.camel.component.hdfs.HdfsNormalFileHandler.createInputStream(HdfsNormalFileHandler.java:94) ~[camel-hdfs-3.19.0.jar:3.19.0]
> ... 27 more
> {code}
> Temporary file was correctly created as follows, but HdfsNormalFileHandler seems to have mistaken its path. The correct one is "/tmp/foo.txt.opened5427239805734101292.hdfs", but there's unnecessary trailing string "/foo.txt.opened" in the log above.
> {code}
> $ ls /tmp/foo.txt*
> /tmp/foo.txt.opened5427239805734101292.hdfs
> $ cat /tmp/foo.txt.opened5427239805734101292.hdfs
> bar
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)