You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2020/10/14 03:27:09 UTC
[GitHub] [pulsar] handywc opened a new issue #8254: pulsar io hdfs could not execute
handywc opened a new issue #8254:
URL: https://github.com/apache/pulsar/issues/8254
**Describe the bug**
execute method：
sinkconfig.yaml configuration：
but the log shows some error:
if i change the configuration of directory to “/tmp/bar”, it will write the file to the local file system
and even I keep produce messages to the topic, but the local file still is not writen.
pulsar 2.6.0
puslar-io-hdfs2-2.6.0.nar
could you give some advice，please？
Thank you.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] BewareMyPower commented on issue #8254: pulsar io hdfs could not execute
Posted by GitBox <gi...@apache.org>.
BewareMyPower commented on issue #8254:
URL: https://github.com/apache/pulsar/issues/8254#issuecomment-709755159
Anyway, the reason why HDFS sink wrote to local file system is that the configuration didn't work and the Hadoop `FileSystem` use `file:///` as the default file system.
But it's weird that local runner works but functions worker doesn't work. Could you give some help? @wolfstudy @tuteng @congbobo184
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] BewareMyPower commented on issue #8254: pulsar io hdfs could not execute
Posted by GitBox <gi...@apache.org>.
BewareMyPower commented on issue #8254:
URL: https://github.com/apache/pulsar/issues/8254#issuecomment-723571315
@handywc see https://stackoverflow.com/questions/32231105/why-is-hsync-not-flushing-my-hdfs-file
The default HDFS sink doesn't specify the `UPDATE_LENGTH` flag to update the metadata of namenode, see https://github.com/apache/pulsar/blob/d7f65451dadc573fc2bb75dbb03cce705ed04d0a/pulsar-io/hdfs2/src/main/java/org/apache/pulsar/io/hdfs2/sink/HdfsSyncThread.java#L73
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] BewareMyPower commented on issue #8254: pulsar io hdfs could not execute
Posted by GitBox <gi...@apache.org>.
BewareMyPower commented on issue #8254:
URL: https://github.com/apache/pulsar/issues/8254#issuecomment-708858891
`localrun` is using `bin/function-localrunner` to run your sink, while `create` is run your sink in the functions worker.
It looks like that you deploy the functions worker with broker, so it may be different with running local runner. Could you try to [deploy your functions worker separately](https://pulsar.apache.org/docs/en/functions-worker/#run-functions-worker-separately)?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] congbobo184 commented on issue #8254: pulsar io hdfs could not execute
Posted by GitBox <gi...@apache.org>.
congbobo184 commented on issue #8254:
URL: https://github.com/apache/pulsar/issues/8254#issuecomment-717646203
@handywc can you add the hadoop.tmp.dir = /tmp/bar configuration in core-site.xml and test again?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] congbobo184 commented on issue #8254: pulsar io hdfs could not execute
Posted by GitBox <gi...@apache.org>.
congbobo184 commented on issue #8254:
URL: https://github.com/apache/pulsar/issues/8254#issuecomment-717756191
@handywc can you change the config hadoop.tmp.dir = tmp/bar and change the directory = tmp/bar?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] congbobo184 edited a comment on issue #8254: pulsar io hdfs could not execute
Posted by GitBox <gi...@apache.org>.
congbobo184 edited a comment on issue #8254:
URL: https://github.com/apache/pulsar/issues/8254#issuecomment-717645920
can you add the hadoop.tmp.dir = /tmp/bar configuration in core-site.xml and test again? @handynwc
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] congbobo184 commented on issue #8254: pulsar io hdfs could not execute
Posted by GitBox <gi...@apache.org>.
congbobo184 commented on issue #8254:
URL: https://github.com/apache/pulsar/issues/8254#issuecomment-713269927
can i have a look for you core-site.xml and hdfs-site.xml？
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] handywc commented on issue #8254: pulsar io hdfs could not execute
Posted by GitBox <gi...@apache.org>.
handywc commented on issue #8254:
URL: https://github.com/apache/pulsar/issues/8254#issuecomment-717754413
I add the port, but still like before.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] congbobo184 edited a comment on issue #8254: pulsar io hdfs could not execute
Posted by GitBox <gi...@apache.org>.
congbobo184 edited a comment on issue #8254:
URL: https://github.com/apache/pulsar/issues/8254#issuecomment-717756191
@handywc can you change the config hadoop.tmp.dir = tmp/bar and change the directory = tmp/bar then test again
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] handywc commented on issue #8254: pulsar io hdfs could not execute
Posted by GitBox <gi...@apache.org>.
handywc commented on issue #8254:
URL: https://github.com/apache/pulsar/issues/8254#issuecomment-708231778
hi，tks for your reply.
I copy the core-site.xml and the hdfs-site.xml from the namenode of hadoop cluster，and the hosts file is set.
I will write a example code to verfiy if the xml configuration is correct or not later.
tks again.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] congbobo184 commented on issue #8254: pulsar io hdfs could not execute
Posted by GitBox <gi...@apache.org>.
congbobo184 commented on issue #8254:
URL: https://github.com/apache/pulsar/issues/8254#issuecomment-717654717
@handywc the fs.defaultFS configuration in your core-site.xml is hdfs://dims32, should we add the port for this? like hdfs://dims32:port
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] handywc commented on issue #8254: pulsar io hdfs could not execute
Posted by GitBox <gi...@apache.org>.
handywc commented on issue #8254:
URL: https://github.com/apache/pulsar/issues/8254#issuecomment-717753828
I add the port, but still like before.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] handywc commented on issue #8254: pulsar io hdfs could not execute
Posted by GitBox <gi...@apache.org>.
handywc commented on issue #8254:
URL: https://github.com/apache/pulsar/issues/8254#issuecomment-718376348
@congbobo184 The pulsar01 doesn't have the core-site.xml. It writes to hdfs after when I copy the file to pulsar01.
Thank you.
But I have another questions.
I set the configuration of syncinterval and maxpendingrecords, but it doesn't seem to work.
I produce five messages
But the file in hdfs is still empty unless I delete the sink .
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] handywc commented on issue #8254: pulsar io hdfs could not execute
Posted by GitBox <gi...@apache.org>.
handywc commented on issue #8254:
URL: https://github.com/apache/pulsar/issues/8254#issuecomment-717768683
Thank you for your patience.
I change the setting, but it's not ok.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] congbobo184 edited a comment on issue #8254: pulsar io hdfs could not execute
Posted by GitBox <gi...@apache.org>.
congbobo184 edited a comment on issue #8254:
URL: https://github.com/apache/pulsar/issues/8254#issuecomment-717654717
@handywc the fs.defaultFS configuration in your core-site.xml is `hdfs://dims32`, should we add the port for this? like hdfs://dims32:port
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] handywc commented on issue #8254: pulsar io hdfs could not execute
Posted by GitBox <gi...@apache.org>.
handywc commented on issue #8254:
URL: https://github.com/apache/pulsar/issues/8254#issuecomment-717649072
hi，thanks for your reply.
I add the hadoop.tmp.dir = /tmp/bar configuration in core-site.xml and test again, but still like before.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] congbobo184 commented on issue #8254: pulsar io hdfs could not execute
Posted by GitBox <gi...@apache.org>.
congbobo184 commented on issue #8254:
URL: https://github.com/apache/pulsar/issues/8254#issuecomment-717808659
@handywc I think the pulsar01 don't have core-site.xml, so it will work by local file system. may be you should copy the core-site.xml to pulsar01. You can try and I don't know if that's true. and then I look up the code, if you set hadoop.tmp.dir = /tmp/bar and the directory should be "", you can choose one.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] handywc commented on issue #8254: pulsar io hdfs could not execute
Posted by GitBox <gi...@apache.org>.
handywc commented on issue #8254:
URL: https://github.com/apache/pulsar/issues/8254#issuecomment-708847010
hi
when i execute the command:
It can write to hdfs successfully.
but i execute the command:
It writes to local file system.
The different is between "localrun" and "create".
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] congbobo184 commented on issue #8254: pulsar io hdfs could not execute
Posted by GitBox <gi...@apache.org>.
congbobo184 commented on issue #8254:
URL: https://github.com/apache/pulsar/issues/8254#issuecomment-717645920
can you add the hadoop.tmp.dir = /tmp/bar configuration in core-site.xml and test again?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] handywc commented on issue #8254: pulsar io hdfs could not execute
Posted by GitBox <gi...@apache.org>.
handywc commented on issue #8254:
URL: https://github.com/apache/pulsar/issues/8254#issuecomment-717633569
hi，I upload core-site.xml and hdfs-site.xml.
By the way, the two files is copied from my hadoop cluster.
[hdfs-site&core-site.zip](https://github.com/apache/pulsar/files/5448931/hdfs-site.core-site.zip)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] BewareMyPower commented on issue #8254: pulsar io hdfs could not execute
Posted by GitBox <gi...@apache.org>.
BewareMyPower commented on issue #8254:
URL: https://github.com/apache/pulsar/issues/8254#issuecomment-732518152
@BewareMyPower Yeah, you're right. I think it may needs a flag for user to specify whether to set the `UPDATE_LENGTH` flag.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] handywc commented on issue #8254: pulsar io hdfs could not execute
Posted by GitBox <gi...@apache.org>.
handywc commented on issue #8254:
URL: https://github.com/apache/pulsar/issues/8254#issuecomment-731955716
@BewareMyPower hi，tks for your reply.
So the default HDFS sink is actually not available, right?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] BewareMyPower commented on issue #8254: pulsar io hdfs could not execute
Posted by GitBox <gi...@apache.org>.
BewareMyPower commented on issue #8254:
URL: https://github.com/apache/pulsar/issues/8254#issuecomment-708165443
See https://github.com/apache/pulsar/blob/3a298f3404d597e6a94de981c5fbe570264dcba1/pulsar-io/hdfs2/src/main/java/org/apache/pulsar/io/hdfs2/sink/HdfsAbstractSink.java#L92-L99
`path` is created from your `directory` and other config params.
The HDFS sink just uses a Hadoop client to create `FileSystem` and open a file under the configured `directory` to write or append. So you should not add any prefix to the path of `directory`.
> if i change the configuration of directory to “/tmp/bar”, it will write the file to the local file system
It means there's something wrong with your `hdfs-site.xml`, it cannot load the Hadoop file system, so it use local file system instead.
You can write a simple Hadoop client example to verify if the issue is related to the `pdfs-site.xml`, like:
```java
Configuration conf = new Configuration();
conf.addResource(new Path(/* path of your pdfs-site.xml */));
FileSystem fs = FileSystem.get(conf);
FSDataOutputStream stream = fs.create(/* file path */);
// Then use `stream` to write some data
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] congbobo184 removed a comment on issue #8254: pulsar io hdfs could not execute
Posted by GitBox <gi...@apache.org>.
congbobo184 removed a comment on issue #8254:
URL: https://github.com/apache/pulsar/issues/8254#issuecomment-717645920
can you add the hadoop.tmp.dir = /tmp/bar configuration in core-site.xml and test again? @handynwc
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] handywc removed a comment on issue #8254: pulsar io hdfs could not execute
Posted by GitBox <gi...@apache.org>.
handywc removed a comment on issue #8254:
URL: https://github.com/apache/pulsar/issues/8254#issuecomment-717754040
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] handywc commented on issue #8254: pulsar io hdfs could not execute
Posted by GitBox <gi...@apache.org>.
handywc commented on issue #8254:
URL: https://github.com/apache/pulsar/issues/8254#issuecomment-717754040
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] handywc removed a comment on issue #8254: pulsar io hdfs could not execute
Posted by GitBox <gi...@apache.org>.
handywc removed a comment on issue #8254:
URL: https://github.com/apache/pulsar/issues/8254#issuecomment-717753828
I add the port, but still like before.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] handywc commented on issue #8254: pulsar io hdfs could not execute
Posted by GitBox <gi...@apache.org>.
handywc commented on issue #8254:
URL: https://github.com/apache/pulsar/issues/8254#issuecomment-709672512
hi
I have deployed functions worker separately.
But still like before. I change the admin-url setting to functions worker's IP and Port.
It also writes to local file system.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org