You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@opendal.apache.org by "shbhmrzd (via GitHub)" <gi...@apache.org> on 2024/01/16 08:43:12 UTC
[I] Support 307 redirects for webhdfs multi write (Needed for hadoop 2.6) [incubator-opendal]
shbhmrzd opened a new issue, #3991:
URL: https://github.com/apache/incubator-opendal/issues/3991
Details at https://discord.com/channels/1081052318650339399/1196707603254874112/1196707611412795452
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support 307 redirects for webhdfs multi write (Needed for hadoop 2.6) [opendal]
Posted by "shbhmrzd (via GitHub)" <gi...@apache.org>.
shbhmrzd commented on issue #3991:
URL: https://github.com/apache/opendal/issues/3991#issuecomment-1899903309
The error I get for Hadoop 2.6 - https://github.com/apache/hadoop/blob/release-2.6.1/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java#L2072
This is refactored and not part of Hadoop 2.7
https://github.com/apache/hadoop/blame/branch-2.7.0/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
Adding versions based on this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support 307 redirects for webhdfs multi write (Needed for hadoop 2.6) [incubator-opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #3991:
URL: https://github.com/apache/incubator-opendal/issues/3991#issuecomment-1895382128
> Sure, I have another setup of HDFS 3.2, let me check on that. Also can you please help me with the version of HDFS used for test jobs in OpenDAL?
We are testing against version `3.1.3`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support 307 redirects for webhdfs multi write (Needed for hadoop 2.6) [incubator-opendal]
Posted by "shbhmrzd (via GitHub)" <gi...@apache.org>.
shbhmrzd commented on issue #3991:
URL: https://github.com/apache/incubator-opendal/issues/3991#issuecomment-1895362539
Sure, I have another setup of HDFS 3.2, let me check on that.
Also can you please help me with the version of HDFS used for test jobs in OpenDAL?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support 307 redirects for webhdfs multi write (Needed for hadoop 2.6) [incubator-opendal]
Posted by "shbhmrzd (via GitHub)" <gi...@apache.org>.
shbhmrzd commented on issue #3991:
URL: https://github.com/apache/incubator-opendal/issues/3991#issuecomment-1898627464
I have reached out to Brahma Reddy Battula, the reporter for https://issues.apache.org/jira/browse/HDFS-6641 to find the exact HDFS version in which concat was fixed. He is an Hadoop PMC.
This could help us document exactly the least version OpenDAL will be able to support.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support 307 redirects for webhdfs multi write (Needed for hadoop 2.6) [incubator-opendal]
Posted by "shbhmrzd (via GitHub)" <gi...@apache.org>.
shbhmrzd commented on issue #3991:
URL: https://github.com/apache/incubator-opendal/issues/3991#issuecomment-1895437764
I tested with 3.2 and here are my findings
1. No failure in the app run, it does not panic
2. If the target file path is `/webhdfs_test/sample_file.txt`, and the parent directory does not exist then the target file is also not created.
3. But there is a concatenated file present in `atomic_write_dir`
4. If I create `webhdfs_test` manually then app run results in sample file being created.
If revert my changes and test on `main` OpenDAL with hdfs 3.2
Then I observe similar behaviour
1. App does not crash
2. If target folder does not exist, the file is not created and the temp file remains in `atomic_write_dir`
3. If target folder exists, the file is created
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support 307 redirects for webhdfs multi write (Needed for hadoop 2.6) [incubator-opendal]
Posted by "shbhmrzd (via GitHub)" <gi...@apache.org>.
shbhmrzd commented on issue #3991:
URL: https://github.com/apache/incubator-opendal/issues/3991#issuecomment-1895505901
It seems hdfs 2.6 cannot be supported with this implementation.
It would be good to have a mention in doc for the least version of hdfs being supported.
Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support 307 redirects for webhdfs multi write (Needed for hadoop 2.6) [incubator-opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #3991:
URL: https://github.com/apache/incubator-opendal/issues/3991#issuecomment-1895452714
> * If target folder does not exist, the file is not created and the temp file remains in `atomic_write_dir`
Oh, that's bad. I don't know why `rename` didn't return an error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support 307 redirects for webhdfs multi write (Needed for hadoop 2.6) [incubator-opendal]
Posted by "shbhmrzd (via GitHub)" <gi...@apache.org>.
shbhmrzd commented on issue #3991:
URL: https://github.com/apache/incubator-opendal/issues/3991#issuecomment-1898095097
> > It would be good to have a mention in doc for the least version of hdfs being supported.
>
> Would you like to submit a PR for this?
Sure, I will raise in some time. Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support 307 redirects for webhdfs multi write (Needed for hadoop 2.6) [incubator-opendal]
Posted by "shbhmrzd (via GitHub)" <gi...@apache.org>.
shbhmrzd commented on issue #3991:
URL: https://github.com/apache/incubator-opendal/issues/3991#issuecomment-1895333521
> > Makes sense. But now I am getting error
> > ```
> > called `Result::unwrap()` on an `Err` value: Custom { kind: Other, error: Unexpected (permanent) at Writer::close => WebHdfsError { exception: "HadoopIllegalArgumentException", message: "The last block in /atomic_write_dir2/96c10cfe-dade-460e-906d-a7567ec978cd is not full; last block size = 31 but file block size = 134217728", java_class_name: "org.apache.hadoop.HadoopIllegalArgumentException" }
> >
> > Context:
> > uri: http://namenode:50070/webhdfs/v1/atomic_write_dir2/96c10cfe-dade-460e-906d-a7567ec978cd?op=CONCAT&sources=/atomic_write_dir2/2c9ad7ab-9c4c-41b7-a54a-070968ba41bc
> > response: Parts { status: 400, version: HTTP/1.1, headers: {"cache-control": "no-cache", "expires": "Wed, 17 Jan 2024 04:41:10 GMT", "expires": "Wed, 17 Jan 2024 04:41:10 GMT", "date": "Wed, 17 Jan 2024 04:41:10 GMT", "date": "Wed, 17 Jan 2024 04:41:10 GMT", "pragma": "no-cache", "pragma": "no-cache", "content-type": "application/json", "x-frame-options": "SAMEORIGIN", "transfer-encoding": "chunked"} }
> > service: webhdfs
> > path: webhdfs_test2/sink_file.txt
> > ```
>
> The `now` here means using #3999?
Yes, the rest of the #3999 changes.
Is it possible that the two files being merged are small files and hdfs finds it tough to merge two smaller blocks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support 307 redirects for webhdfs multi write (Needed for hadoop 2.6) [incubator-opendal]
Posted by "shbhmrzd (via GitHub)" <gi...@apache.org>.
shbhmrzd commented on issue #3991:
URL: https://github.com/apache/incubator-opendal/issues/3991#issuecomment-1895227054
Makes sense. But now I am getting error
```
called `Result::unwrap()` on an `Err` value: Custom { kind: Other, error: Unexpected (permanent) at Writer::close => WebHdfsError { exception: "HadoopIllegalArgumentException", message: "The last block in /atomic_write_dir2/96c10cfe-dade-460e-906d-a7567ec978cd is not full; last block size = 31 but file block size = 134217728", java_class_name: "org.apache.hadoop.HadoopIllegalArgumentException" }
Context:
uri: http://namenode:50070/webhdfs/v1/atomic_write_dir2/96c10cfe-dade-460e-906d-a7567ec978cd?op=CONCAT&sources=/atomic_write_dir2/2c9ad7ab-9c4c-41b7-a54a-070968ba41bc
response: Parts { status: 400, version: HTTP/1.1, headers: {"cache-control": "no-cache", "expires": "Wed, 17 Jan 2024 04:41:10 GMT", "expires": "Wed, 17 Jan 2024 04:41:10 GMT", "date": "Wed, 17 Jan 2024 04:41:10 GMT", "date": "Wed, 17 Jan 2024 04:41:10 GMT", "pragma": "no-cache", "pragma": "no-cache", "content-type": "application/json", "x-frame-options": "SAMEORIGIN", "transfer-encoding": "chunked"} }
service: webhdfs
path: webhdfs_test2/sink_file.txt
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support 307 redirects for webhdfs multi write (Needed for hadoop 2.6) [incubator-opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #3991:
URL: https://github.com/apache/incubator-opendal/issues/3991#issuecomment-1893584388
> The first request at https://github.com/apache/incubator-opendal/blob/main/core/src/services/webhdfs/backend.rs#L262
> returns with status code 201 and also creates the file. That is why the response body is empty.
Ok, I got it. We follow the redirect and created an empty file instead which is not expected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support 307 redirects for webhdfs multi write (Needed for hadoop 2.6) [incubator-opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #3991:
URL: https://github.com/apache/incubator-opendal/issues/3991#issuecomment-1897805708
> It would be good to have a mention in doc for the least version of hdfs being supported.
Would you like to submit a PR for this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support 307 redirects for webhdfs multi write (Needed for hadoop 2.6) [incubator-opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #3991:
URL: https://github.com/apache/incubator-opendal/issues/3991#issuecomment-1895230580
> Makes sense. But now I am getting error
>
> ```
> called `Result::unwrap()` on an `Err` value: Custom { kind: Other, error: Unexpected (permanent) at Writer::close => WebHdfsError { exception: "HadoopIllegalArgumentException", message: "The last block in /atomic_write_dir2/96c10cfe-dade-460e-906d-a7567ec978cd is not full; last block size = 31 but file block size = 134217728", java_class_name: "org.apache.hadoop.HadoopIllegalArgumentException" }
>
> Context:
> uri: http://namenode:50070/webhdfs/v1/atomic_write_dir2/96c10cfe-dade-460e-906d-a7567ec978cd?op=CONCAT&sources=/atomic_write_dir2/2c9ad7ab-9c4c-41b7-a54a-070968ba41bc
> response: Parts { status: 400, version: HTTP/1.1, headers: {"cache-control": "no-cache", "expires": "Wed, 17 Jan 2024 04:41:10 GMT", "expires": "Wed, 17 Jan 2024 04:41:10 GMT", "date": "Wed, 17 Jan 2024 04:41:10 GMT", "date": "Wed, 17 Jan 2024 04:41:10 GMT", "pragma": "no-cache", "pragma": "no-cache", "content-type": "application/json", "x-frame-options": "SAMEORIGIN", "transfer-encoding": "chunked"} }
> service: webhdfs
> path: webhdfs_test2/sink_file.txt
> ```
The `now` here means using https://github.com/apache/incubator-opendal/pull/3999?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support 307 redirects for webhdfs multi write (Needed for hadoop 2.6) [incubator-opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #3991:
URL: https://github.com/apache/incubator-opendal/issues/3991#issuecomment-1895344112
> Yes, the rest of the #3999 changes.
> Is it possible that the two files being merged are small files and hdfs finds it tough to merge two smaller blocks
I'm not sure are we on the same picture. Test on PR 3999 failed for
```shell
---- behavior::test_blocking_writer_with_append ----
Unexpected (permanent) at read, context: { uri: http://127.0.0.1:9870/webhdfs/v1/dda279c3-1e16-414a-8bbb-2d8227bfce50/6535823f-6205-42a1-97ef-2219140dcfb3?op=OPEN&offset=0&length=13025904, response: Parts { status: 307, version: HTTP/1.1, headers: {"date": "Wed, 17 Jan 2024 05:14:38 GMT", "date": "Wed, 17 Jan 2024 05:14:38 GMT", "cache-control": "no-cache", "expires": "Wed, 17 Jan 2024 05:14:38 GMT", "pragma": "no-cache", "x-frame-options": "SAMEORIGIN", "location": "http://localhost:9864/webhdfs/v1/dda279c3-1e16-414a-8bbb-2d8227bfce50/6535823f-6205-42a1-97ef-2219140dcfb3?op=OPEN&namenoderpcaddress=a36c2a44b7e7:8020&length=13025904&offset=0", "content-type": "application/octet-stream", "content-length": "0"} }, service: webhdfs, path: 6535823f-6205-42a1-97ef-2219140dcfb3, range: 0-13025903 }
```
Are you refering to the error happen on hdfs 2.6?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support 307 redirects for webhdfs multi write (Needed for hadoop 2.6) [incubator-opendal]
Posted by "shbhmrzd (via GitHub)" <gi...@apache.org>.
shbhmrzd commented on issue #3991:
URL: https://github.com/apache/incubator-opendal/issues/3991#issuecomment-1895484221
I am facing the same issue as mentioned in https://issues.apache.org/jira/browse/HDFS-6641 with hdfs 2.6
It seems it was later handled better in the newer versions.
Found another occurrence of issues with concat for another project - https://github.com/pravega/pravega/issues/852
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support 307 redirects for webhdfs multi write (Needed for hadoop 2.6) [incubator-opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #3991:
URL: https://github.com/apache/incubator-opendal/issues/3991#issuecomment-1895222270
I feel like we can add a new feature called `enable_redirect`. This feature is added for users that using old webhdfs which doesn't support `noredirect`. If user enable this feature, we won't send `noredirect` to services and handles the redirect by hand.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support 307 redirects for webhdfs multi write (Needed for hadoop 2.6) [incubator-opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #3991:
URL: https://github.com/apache/incubator-opendal/issues/3991#issuecomment-1898630159
> I have reached out to Brahma Reddy Battula, the reporter for https://issues.apache.org/jira/browse/HDFS-6641 to find the exact HDFS version in which concat was fixed. He is an Hadoop PMC. This could help us document exactly the least version OpenDAL will be able to support.
Perfect. Thanks a lot!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support 307 redirects for webhdfs multi write (Needed for hadoop 2.6) [opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo closed issue #3991: Support 307 redirects for webhdfs multi write (Needed for hadoop 2.6)
URL: https://github.com/apache/opendal/issues/3991
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support 307 redirects for webhdfs multi write (Needed for hadoop 2.6) [incubator-opendal]
Posted by "shbhmrzd (via GitHub)" <gi...@apache.org>.
shbhmrzd commented on issue #3991:
URL: https://github.com/apache/incubator-opendal/issues/3991#issuecomment-1894931227
For disabling redirect, I modify the http client to set redirect false.
https://github.com/apache/incubator-opendal/pull/3999
But now the failure is with error
```
thread 'main' panicked at src/main.rs:42:35:
called `Result::unwrap()` on an `Err` value: Custom { kind: Other, error: Unexpected (permanent) at Writer::close => WebHdfsError { exception: "HadoopIllegalArgumentException", message: "The last block in /atomic_write_dir2/96c10cfe-dade-460e-906d-a7567ec978cd is not full; last block size = 31 but file block size = 134217728", java_class_name: "org.apache.hadoop.HadoopIllegalArgumentException" }
Context:
uri: http://namenode:50070/webhdfs/v1/atomic_write_dir2/96c10cfe-dade-460e-906d-a7567ec978cd?op=CONCAT&sources=/atomic_write_dir2/2c9ad7ab-9c4c-41b7-a54a-070968ba41bc
response: Parts { status: 400, version: HTTP/1.1, headers: {"cache-control": "no-cache", "expires": "Wed, 17 Jan 2024 04:41:10 GMT", "expires": "Wed, 17 Jan 2024 04:41:10 GMT", "date": "Wed, 17 Jan 2024 04:41:10 GMT", "date": "Wed, 17 Jan 2024 04:41:10 GMT", "pragma": "no-cache", "pragma": "no-cache", "content-type": "application/json", "x-frame-options": "SAMEORIGIN", "transfer-encoding": "chunked"} }
service: webhdfs
path: webhdfs_test2/sink_file.txt
Backtrace:
0: std::backtrace_rs::backtrace::libunwind::trace
at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/../../backtrace/src/backtrace/libunwind.rs:104:5
1: std::backtrace_rs::backtrace::trace_unsynchronized
at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
2: std::backtrace::Backtrace::create
at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/backtrace.rs:331:13
3: opendal::types::error::Error::new
at /Users/sraizada/.cargo/git/checkouts/incubator-opendal-b5114b6bfa1ac4c6/f571aa2/core/src/types/error.rs:338:24
4: opendal::services::webhdfs::error::parse_error_msg
at /Users/sraizada/.cargo/git/checkouts/incubator-opendal-b5114b6bfa1ac4c6/f571aa2/core/src/services/webhdfs/error.rs:68:19
5: opendal::services::webhdfs::error::parse_error::{{closure}}
at /Users/sraizada/.cargo/git/checkouts/incubator-opendal-b5114b6bfa1ac4c6/f571aa2/core/src/services/webhdfs/error.rs:46:5
6: <opendal::services::webhdfs::writer::WebhdfsWriter as opendal::raw::oio::write::block_write::BlockWrite>::complete_block::{{closure}}
at /Users/sraizada/.cargo/git/checkouts/incubator-opendal-b5114b6bfa1ac4c6/f571aa2/core/src/services/webhdfs/writer.rs:115:46
7: <core::pin::Pin<P> as core::future::future::Future>::poll
at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/future/future.rs:125:9
8: <opendal::raw::oio::write::block_write::BlockWriter<W> as opendal::raw::oio::write::api::Write>::poll_close::{{closure}}
at /Users/sraizada/.cargo/git/checkouts/incubator-opendal-b5114b6bfa1ac4c6/f571aa2/core/src/raw/oio/write/block_write.rs:282:74
9: <opendal::raw::oio::write::block_write::BlockWriter<W> as opendal::raw::oio::write::api::Write>::poll_close
at /Users/sraizada/.cargo/git/checkouts/incubator-opendal-b5114b6bfa1ac4c6/f571aa2/core/src/raw/oio/write/block_write.rs:288:47
10: <opendal::raw::enum_utils::TwoWays<ONE,TWO> as opendal::raw::oio::write::api::Write>::poll_close
at /Users/sraizada/.cargo/git/checkouts/incubator-opendal-b5114b6bfa1ac4c6/f571aa2/core/src/raw/enum_utils.rs:116:29
11: <opendal::layers::error_context::ErrorContextWrapper<T> as opendal::raw::oio::write::api::Write>::poll_close
at /Users/sraizada/.cargo/git/checkouts/incubator-opendal-b5114b6bfa1ac4c6/f571aa2/core/src/layers/error_context.rs:420:9
12: <opendal::layers::complete::CompleteWriter<W> as opendal::raw::oio::write::api::Write>::poll_close
at /Users/sraizada/.cargo/git/checkouts/incubator-opendal-b5114b6bfa1ac4c6/f571aa2/core/src/layers/complete.rs:728:16
13: <opendal::raw::enum_utils::TwoWays<ONE,TWO> as opendal::raw::oio::write::api::Write>::poll_close
at /Users/sraizada/.cargo/git/checkouts/incubator-opendal-b5114b6bfa1ac4c6/f571aa2/core/src/raw/enum_utils.rs:116:29
14: <alloc::boxed::Box<T> as opendal::raw::oio::write::api::Write>::poll_close
at /Users/sraizada/.cargo/git/checkouts/incubator-opendal-b5114b6bfa1ac4c6/f571aa2/core/src/raw/oio/write/api.rs:125:9
15: <opendal::raw::oio::write::api::CloseFuture<W> as core::future::future::Future>::poll
at /Users/sraizada/.cargo/git/checkouts/incubator-opendal-b5114b6bfa1ac4c6/f571aa2/core/src/raw/oio/write/api.rs:197:9
16: opendal::types::writer::Writer::close::{{closure}}
at /Users/sraizada/.cargo/git/checkouts/incubator-opendal-b5114b6bfa1ac4c6/f571aa2/core/src/types/writer.rs:208:28
17: opendal_test::multi_write_example::{{closure}}
at ./src/main.rs:59:15
18: opendal_test::main::{{closure}}
at ./src/main.rs:42:29
19: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
at /Users/sraizada/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.0/src/runtime/park.rs:282:63
20: tokio::runtime::coop::with_budget
at /Users/sraizada/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.0/src/runtime/coop.rs:107:5
21: tokio::runtime::coop::budget
at /Users/sraizada/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.0/src/runtime/coop.rs:73:5
22: tokio::runtime::park::CachedParkThread::block_on
at /Users/sraizada/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.0/src/runtime/park.rs:282:31
23: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
at /Users/sraizada/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.0/src/runtime/context/blocking.rs:66:9
24: tokio::runtime::scheduler::multi_thread::MultiThread::block_on::{{closure}}
at /Users/sraizada/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.0/src/runtime/scheduler/multi_thread/mod.rs:87:13
25: tokio::runtime::context::runtime::enter_runtime
at /Users/sraizada/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.0/src/runtime/context/runtime.rs:65:16
26: tokio::runtime::scheduler::multi_thread::MultiThread::block_on
at /Users/sraizada/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.0/src/runtime/scheduler/multi_thread/mod.rs:86:9
27: tokio::runtime::runtime::Runtime::block_on
at /Users/sraizada/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.0/src/runtime/runtime.rs:350:45
28: opendal_test::main
at ./src/main.rs:43:5
29: core::ops::function::FnOnce::call_once
at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:250:5
30: std::sys_common::backtrace::__rust_begin_short_backtrace
at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/sys_common/backtrace.rs:154:18
31: std::rt::lang_start::{{closure}}
at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:167:18
32: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:284:13
33: std::panicking::try::do_call
at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552:40
34: std::panicking::try
at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
35: std::panic::catch_unwind
at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
36: std::rt::lang_start_internal::{{closure}}
at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:48
37: std::panicking::try::do_call
at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552:40
38: std::panicking::try
at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
39: std::panic::catch_unwind
at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
40: std::rt::lang_start_internal
at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:20
41: std::rt::lang_start
at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:166:17
42: _main
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support 307 redirects for webhdfs multi write (Needed for hadoop 2.6) [incubator-opendal]
Posted by "shbhmrzd (via GitHub)" <gi...@apache.org>.
shbhmrzd commented on issue #3991:
URL: https://github.com/apache/incubator-opendal/issues/3991#issuecomment-1895347932
Yes, I am not referring to the tests. I made the change for no redirect and corresponding change needs to be done in all methods to rely on the header to get location. I will make that change.
For my testing currently, I am just testing multi write on 2.6 which was initially failing with first error for empty response and now failing during concat with the above HadoopIllegalArgumentException.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support 307 redirects for webhdfs multi write (Needed for hadoop 2.6) [incubator-opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #3991:
URL: https://github.com/apache/incubator-opendal/issues/3991#issuecomment-1895355161
Have you tried running the code on a newer HDFS version, such as 3.3? I'm concerned that the same limitations apply across different HDFS versions. If hdfs has requirement on the blocks of concat, we might need to set the `write_multi_min_size` and set buffer for writer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support 307 redirects for webhdfs multi write (Needed for hadoop 2.6) [incubator-opendal]
Posted by "shbhmrzd (via GitHub)" <gi...@apache.org>.
shbhmrzd commented on issue #3991:
URL: https://github.com/apache/incubator-opendal/issues/3991#issuecomment-1893581055
I tried adding logs and observe a strange behaviour with my hdfs cluster.
The first request at https://github.com/apache/incubator-opendal/blob/main/core/src/services/webhdfs/backend.rs#L262
returns with status code 201 and also creates the file. That is why the response body is empty.
Now since the file is created, the subsequent request to add data to the file must go through smoothly. I make another request by using the same namenode url by making changes to the code in backend.
```rust
let mut location: String;
if status == StatusCode::CREATED {
let hdfs_location = resp.headers().get("location").unwrap().to_str().unwrap().to_string();
println!("hdfs_location - {:?}", hdfs_location);
// hdfs_location - "hdfs://my-hdfs/atomic_write_dir2/a2c5f1b7-34e9-4bd7-bf22-e7d4513b6a4c"
location = url.clone();
println!("location - {:?}", location);
}else{
let bs = resp.into_body().bytes().await?;
println!("Response body {:?}", bs);
let resp =
serde_json::from_slice::<LocationResponse>(&bs).map_err(new_json_deserialize_error)?;
location = resp.location;
}
```
But this time there is a redirect and it fails at
https://github.com/apache/incubator-opendal/blob/main/core/src/services/webhdfs/writer.rs#L83
```
Context:
uri: http://my-namenode:50070/webhdfs/v1/atomic_write_dir2/a2c5f1b7-34e9-4bd7-bf22-e7d4513b6a4c?op=CREATE&overwrite=true&noredirect=true
response: Parts { status: 307, version: HTTP/1.1, headers: {"cache-control": "no-cache", "expires": "Tue, 16 Jan 2024 10:51:50 GMT", "expires": "Tue, 16 Jan 2024 10:51:50 GMT", "date": "Tue, 16 Jan 2024 10:51:50 GMT", "date": "Tue, 16 Jan 2024 10:51:50 GMT", "pragma": "no-cache", "pragma": "no-cache", "content-type": "application/octet-stream", "x-frame-options": "SAMEORIGIN", "location": "http://redirect-url:50075/webhdfs/v1/atomic_write_dir2/a2c5f1b7-34e9-4bd7-bf22-e7d4513b6a4c?op=CREATE&namenoderpcaddress=tlb-qe-1min-hdfs&overwrite=true", "content-length": "0"} }
service: webhdfs
path: webhdfs_test2/sink_file.txt
```
Based on the docs, the create file must have redirect.
https://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Create_and_Write_to_a_File
Not sure why the subsequent data write leads to redirect.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org