You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@opendal.apache.org by "Xuanwo (via GitHub)" <gi...@apache.org> on 2023/09/12 12:38:47 UTC
[GitHub] [incubator-opendal] Xuanwo opened a new issue, #3040: etcd test is flaky for test_blocking_rename_self
Xuanwo opened a new issue, #3040:
URL: https://github.com/apache/incubator-opendal/issues/3040
```shell
running 1 test
2023-09-12T12:36:31.357430Z DEBUG opendal::services: service=etcd operation=blocking_write path=81ae3294-1e21-47f9-af79-0703033bc7d8 -> started
at core/src/layers/logging.rs:726
2023-09-12T12:36:31.357626Z DEBUG opendal::services: service=etcd operation=blocking_write path=81ae3294-1e21-47f9-af79-0703033bc7d8 -> start writing
at core/src/layers/logging.rs:737
2023-09-12T12:36:31.569737Z ERROR opendal::services: service=etcd operation=BlockingWriter::close path=81ae3294-1e21-47f9-af79-0703033bc7d8 written=1485803B -> data close failed: Unexpected (temporary) at Writer::close => grpc request error: status: Unauthenticated, message: "etcdserver: invalid auth token", details: [], metadata: MetadataMap { headers: {"access-control-allow-headers": "accept, content-type, authorization", "access-control-allow-methods": "POST, GET, OPTIONS, PUT, DELETE", "access-control-allow-origin": "*", "content-type": "application/grpc", "trailer": "Grpc-Status", "trailer": "Grpc-Message", "trailer": "Grpc-Status-Details-Bin"} }
Context:
service: etcd
path: 81ae3294-1e21-47f9-af79-0703033bc7d8
Source:
grpc request error: status: Unauthenticated, message: "etcdserver: invalid auth token", details: [], metadata: MetadataMap { headers: {"access-control-allow-headers": "accept, content-type, authorization", "access-control-allow-methods": "POST, GET, OPTIONS, PUT, DELETE", "access-control-allow-origin": "*", "content-type": "application/grpc", "trailer": "Grpc-Status", "trailer": "Grpc-Message", "trailer": "Grpc-Status-Details-Bin"} }
Backtrace:
0: opendal::types::error::Error::new
at ./src/types/error.rs:338:24
1: opendal::services::etcd::backend::<impl core::convert::From<etcd_client::error::Error> for opendal::types::error::Error>::from
at ./src/services/etcd/backend.rs:320:9
2: <core::result::Result<T,F> as core::ops::try_trait::FromResidual<core::result::Result<core::convert::Infallible,E>>>::from_residual
at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/core/src/result.rs:1961:27
3: <opendal::services::etcd::backend::Adapter as opendal::raw::adapters::kv::api::Adapter>::set::{{closure}}
at ./src/services/etcd/backend.rs:292:17
4: <core::pin::Pin<P> as core::future::future::Future>::poll
at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/core/src/future/future.rs:125:9
5: <opendal::raw::adapters::kv::backend::KvWriter<S> as opendal::raw::oio::write::api::Write>::poll_close::{{closure}}
at ./src/raw/adapters/kv/backend.rs:451:64
6: <core::pin::Pin<P> as core::future::future::Future>::poll
at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/core/src/future/future.rs:125:9
7: futures_util::future::future::FutureExt::poll_unpin
at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/future/future/mod.rs:562:9
8: <opendal::raw::adapters::kv::backend::KvWriter<S> as opendal::raw::oio::write::api::Write>::poll_close
at ./src/raw/adapters/kv/backend.rs:439:28
9: <opendal::layers::error_context::ErrorContextWrapper<T> as opendal::raw::oio::write::api::Write>::poll_close
at ./src/layers/error_context.rs:423:9
10: <opendal::layers::complete::CompleteWriter<W> as opendal::raw::oio::write::api::Write>::poll_close
at ./src/layers/complete.rs:765:16
11: <opendal::layers::blocking::BlockingWrapper<I> as opendal::raw::oio::write::api::BlockingWrite>::close::{{closure}}
at ./src/layers/blocking.rs:207:36
12: <futures_util::future::poll_fn::PollFn<F> as core::future::future::Future>::poll
at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/future/poll_fn.rs:56:9
13: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/park.rs:283:63
14: tokio::runtime::coop::with_budget
at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/coop.rs:107:5
15: tokio::runtime::coop::budget
at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/coop.rs:73:5
16: tokio::runtime::park::CachedParkThread::block_on
at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/park.rs:283:31
17: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/context/blocking.rs:66:9
18: tokio::runtime::handle::Handle::block_on::{{closure}}
at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/handle.rs:308:13
19: tokio::runtime::context::runtime::enter_runtime
at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/context/runtime.rs:65:16
20: tokio::runtime::handle::Handle::block_on
at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/handle.rs:307:9
21: <opendal::layers::blocking::BlockingWrapper<I> as opendal::raw::oio::write::api::BlockingWrite>::close
at ./src/layers/blocking.rs:206:9
22: <opendal::layers::logging::LoggingWriter<W> as opendal::raw::oio::write::api::BlockingWrite>::close
at ./src/layers/logging.rs:1389:15
23: <opendal::layers::retry::RetryWrapper<R,I> as opendal::raw::oio::write::api::BlockingWrite>::close::{{closure}}
at ./src/layers/retry.rs:1034:14
24: backon::blocking_retry::BlockingRetry<B,T,E,F,RF,NF>::call
at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/backon-0.4.1/src/blocking_retry.rs:175:26
25: <opendal::layers::retry::RetryWrapper<R,I> as opendal::raw::oio::write::api::BlockingWrite>::close
at ./src/layers/retry.rs:1034:9
26: opendal::types::operator::blocking_operator::BlockingOperator::write_with::{{closure}}
at ./src/types/operator/blocking_operator.rs:569:17
27: core::ops::function::FnOnce::call_once
at /rustc/5[680](https://github.com/apache/incubator-opendal/actions/runs/6159457461/job/16714336842#step:6:681)fa18feaa87f3ff04063800aec256c3d4b4be/library/core/src/ops/function.rs:250:5
28: opendal::types::operator::operator_functions::OperatorFunction<T,R>::call
at ./src/types/operator/operator_functions.rs:65:9
29: opendal::types::operator::operator_functions::FunctionWrite::call
at ./src/types/operator/operator_functions.rs:142:9
30: opendal::types::operator::blocking_operator::BlockingOperator::write
at ./src/types/operator/blocking_operator.rs:405:9
31: behavior::blocking_rename::test_blocking_rename_self
at ./tests/behavior/blocking_rename.rs:115:5
32: core::ops::function::FnOnce::call_once
at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/core/src/ops/function.rs:250:5
33: behavior::utils::build_blocking_trial::{{closure}}
at ./tests/behavior/utils.rs:170:9
34: libtest_mimic::Trial::test::{{closure}}
at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/libtest-mimic-0.6.1/src/lib.rs:112:54
35: core::ops::function::FnOnce::call_once{{vtable.shim}}
at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/core/src/ops/function.rs:250:5
36: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/alloc/src/boxed.rs:1993:9
37: libtest_mimic::run_single::{{closure}}
at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/libtest-mimic-0.6.1/src/lib.rs:505:43
38: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/core/src/panic/unwind_safe.rs:271:9
39: std::panicking::try::do_call
at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/std/src/panicking.rs:500:40
40: __rust_try
41: std::panicking::try
at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/std/src/panicking.rs:464:19
42: std::panic::catch_unwind
at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/std/src/panic.rs:142:14
43: libtest_mimic::run_single
at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/libtest-mimic-0.6.1/src/lib.rs:505:5
44: libtest_mimic::run::{{closure}}
at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/libtest-mimic-0.6.1/src/lib.rs:476:35
45: <F as threadpool::FnBox>::call_box
at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/threadpool-1.8.1/src/lib.rs:95:9
46: threadpool::spawn_in_pool::{{closure}}
at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/threadpool-1.8.1/src/lib.rs:769:17
47: std::sys_common::backtrace::__rust_begin_short_backtrace
at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/std/src/sys_common/backtrace.rs:135:18
48: std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}
at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/std/src/thread/mod.rs:529:17
49: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/core/src/panic/unwind_safe.rs:271:9
50: std::panicking::try::do_call
at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/std/src/panicking.rs:500:40
51: __rust_try
52: std::panicking::try
at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/std/src/panicking.rs:464:19
53: std::panic::catch_unwind
at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/std/src/panic.rs:142:14
54: std::thread::Builder::spawn_unchecked_::{{closure}}
at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/std/src/thread/mod.rs:528:30
55: core::ops::function::FnOnce::call_once{{vtable.shim}}
at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/core/src/ops/function.rs:250:5
56: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/alloc/src/boxed.rs:1993:9
57: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/alloc/src/boxed.rs:1993:9
58: std::sys::unix::thread::Thread::new::thread_start
at /rustc/5680fa18feaa87f3ff04063800aec256c3d4b4be/library/std/src/sys/unix/thread.rs:108:17
59: <unknown>
60: <unknown>
at core/src/layers/logging.rs:1403
2023-09-12T12:36:31.5[698](https://github.com/apache/incubator-opendal/actions/runs/6159457461/job/16714336842#step:6:699)85Z WARN opendal::service: operation=BlockingWriter::close path=81ae3294-1e21-47f9-af79-0[703](https://github.com/apache/incubator-opendal/actions/runs/6159457461/job/16714336842#step:6:704)033bc7d8 -> retry after 1s: error=Unexpected (temporary) at Writer::close, context: { service: etcd, path: 81ae3294-1e21-47f9-af79-0703033bc7d8 } => grpc request error: status: Unauthenticated, message: "etcdserver: invalid auth token", details: [], metadata: MetadataMap { headers: {"access-control-allow-headers": "accept, content-type, authorization", "access-control-allow-methods": "POST, GET, OPTIONS, PUT, DELETE", "access-control-allow-origin": "*", "content-type": "application/grpc", "trailer": "Grpc-Status", "trailer": "Grpc-Message", "trailer": "Grpc-Status-Details-Bin"} }, source: grpc request error: status: Unauthenticated, message: "etcdserver: invalid auth toke
n", details: [], metadata: MetadataMap { headers: {"access-control-allow-headers": "accept, content-type, authorization", "access-control-allow-methods": "POST, GET, OPTIONS, PUT, DELETE", "access-control-allow-origin": "*", "content-type": "application/grpc", "trailer": "Grpc-Status", "trailer": "Grpc-Message", "trailer": "Grpc-Status-Details-Bin"} }
at core/src/layers/retry.rs:259
2023-09-12T12:36:32.573783Z WARN opendal::layers::complete: writer has not been closed or aborted, must be a bug
at core/src/layers/complete.rs:[705](https://github.com/apache/incubator-opendal/actions/runs/6159457461/job/16714336842#step:6:706)
test services_etcd::test_blocking_rename_self ... FAILED
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-opendal] Xuanwo closed issue #3040: etcd test is flaky for test_blocking_rename_self
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo closed issue #3040: etcd test is flaky for test_blocking_rename_self
URL: https://github.com/apache/incubator-opendal/issues/3040
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-opendal] Xuanwo commented on issue #3040: etcd test is flaky for test_blocking_rename_self
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #3040:
URL: https://github.com/apache/incubator-opendal/issues/3040#issuecomment-1715681894
Seems etcd-client doesn't do token refresh for us.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-opendal] Xuanwo commented on issue #3040: etcd test is flaky for test_blocking_rename_self
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #3040:
URL: https://github.com/apache/incubator-opendal/issues/3040#issuecomment-1715647318
cc @G-XD, would you like to take a look? Thanks in advance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-opendal] Xuanwo commented on issue #3040: etcd test is flaky for test_blocking_rename_self
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #3040:
URL: https://github.com/apache/incubator-opendal/issues/3040#issuecomment-1715787128
Well, the root cause is our kv API not implement correctly. But adding a connection pool is also a good idea :rofl:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-opendal] Xuanwo commented on issue #3040: etcd test is flaky for test_blocking_rename_self
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #3040:
URL: https://github.com/apache/incubator-opendal/issues/3040#issuecomment-1715797559
> which kv APi not implemented correctly?
The future state machine inside kv adapter:
https://github.com/apache/incubator-opendal/pull/3041/files#diff-e33ce9dc785c78f1663af0614d80c879d02da13f139918a5f95063df5836e0f4L439
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-opendal] oowl commented on issue #3040: etcd test is flaky for test_blocking_rename_self
Posted by "oowl (via GitHub)" <gi...@apache.org>.
oowl commented on issue #3040:
URL: https://github.com/apache/incubator-opendal/issues/3040#issuecomment-1715793222
> Well, the root cause is our kv API not implemented correctly. But adding a connection pool is also a good idea 🤣
which kv APi not implemented correctly?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org