You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@opendal.apache.org by "Zheaoli (via GitHub)" <gi...@apache.org> on 2023/10/31 14:47:02 UTC
[PR] chore(test): Fix sftp test failed in python [incubator-opendal]
Zheaoli opened a new pull request, #3436:
URL: https://github.com/apache/incubator-opendal/pull/3436
Signed-off-by: Manjusaka <me...@manjusaka.me>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] chore(test): Fix sftp test failed in python [incubator-opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #3436:
URL: https://github.com/apache/incubator-opendal/pull/3436#discussion_r1377780624
##########
.github/services/sftp/sftp/action.yml:
##########
@@ -24,7 +24,28 @@ runs:
- name: Setup Sftp
shell: bash
working-directory: fixtures/sftp
- run: docker-compose -f docker-compose-sftp.yml up -d
+ run: |
+ docker-compose -f docker-compose-sftp.yml up -d
+ set +ex
Review Comment:
So we can't implement that in `health_check`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] chore(test): Fix sftp test failed in python [incubator-opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #3436:
URL: https://github.com/apache/incubator-opendal/pull/3436#discussion_r1377794299
##########
.github/services/sftp/sftp/action.yml:
##########
@@ -24,7 +24,28 @@ runs:
- name: Setup Sftp
shell: bash
working-directory: fixtures/sftp
- run: docker-compose -f docker-compose-sftp.yml up -d
+ run: |
+ docker-compose -f docker-compose-sftp.yml up -d
+ set +ex
Review Comment:
Maybe we can use `docker compose up -d --wait`?
```shell
--wait Wait for services to be
running|healthy. Implies detached mode.
--wait-timeout int Maximum duration to wait for the
project to be running|healthy.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] chore(test): Fix sftp test failed in python [incubator-opendal]
Posted by "Zheaoli (via GitHub)" <gi...@apache.org>.
Zheaoli commented on PR #3436:
URL: https://github.com/apache/incubator-opendal/pull/3436#issuecomment-1787442150
cc @Xuanwo Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] chore(test): Fix sftp test failed in python [incubator-opendal]
Posted by "Zheaoli (via GitHub)" <gi...@apache.org>.
Zheaoli commented on PR #3436:
URL: https://github.com/apache/incubator-opendal/pull/3436#issuecomment-1787533325
@Xuanwo PTAL
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] chore(test): Fix sftp test failed in python [incubator-opendal]
Posted by "Zheaoli (via GitHub)" <gi...@apache.org>.
Zheaoli commented on code in PR #3436:
URL: https://github.com/apache/incubator-opendal/pull/3436#discussion_r1377794770
##########
.github/services/sftp/sftp/action.yml:
##########
@@ -24,7 +24,28 @@ runs:
- name: Setup Sftp
shell: bash
working-directory: fixtures/sftp
- run: docker-compose -f docker-compose-sftp.yml up -d
+ run: |
+ docker-compose -f docker-compose-sftp.yml up -d
+ set +ex
Review Comment:
> that we have the same setup in both CI and locally.
We already have the same test, just need extra check for automation test in CI.
Actually, if you want to run the test and compose by using script locally, you need extra health check to make sure the server is running before running your test too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] chore(test): Fix sftp test failed in python [incubator-opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #3436:
URL: https://github.com/apache/incubator-opendal/pull/3436#discussion_r1377770681
##########
.github/services/sftp/sftp/action.yml:
##########
@@ -24,7 +24,28 @@ runs:
- name: Setup Sftp
shell: bash
working-directory: fixtures/sftp
- run: docker-compose -f docker-compose-sftp.yml up -d
+ run: |
+ docker-compose -f docker-compose-sftp.yml up -d
+ set +ex
Review Comment:
I don't like this since it's hard for local development. Can we implement in docker-compose's health check instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] chore(test): Fix sftp test failed in python [incubator-opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #3436:
URL: https://github.com/apache/incubator-opendal/pull/3436#discussion_r1377762395
##########
.github/services/sftp/sftp_with_default_root/action.yml:
##########
@@ -35,6 +35,26 @@ runs:
--ulimit nofile=65536:65536 \
-p 2222:22 -d atmoz/sftp \
foo::::upload
+ set +ex
+ username="foo"
+ server="127.0.0.1"
+ port=2222
+ know_hosts_strategy="no"
+ identity_file="${{ github.workspace }}/target/ssh/id_rsa"
+ while :; do
+ (
+ sftp -oPort=$port -o StrictHostKeyChecking=$know_hosts_strategy -o IdentityFile=$identity_file $username@$server << EOF
Review Comment:
> Carry your pr or open a new PR?
Both LGTM. (In fact, you can send a PR to `polish-sftp` branch)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] chore(test): Fix sftp test failed in python [incubator-opendal]
Posted by "Zheaoli (via GitHub)" <gi...@apache.org>.
Zheaoli commented on code in PR #3436:
URL: https://github.com/apache/incubator-opendal/pull/3436#discussion_r1377761026
##########
.github/services/sftp/sftp_with_default_root/action.yml:
##########
@@ -35,6 +35,26 @@ runs:
--ulimit nofile=65536:65536 \
-p 2222:22 -d atmoz/sftp \
foo::::upload
+ set +ex
+ username="foo"
+ server="127.0.0.1"
+ port=2222
+ know_hosts_strategy="no"
+ identity_file="${{ github.workspace }}/target/ssh/id_rsa"
+ while :; do
+ (
+ sftp -oPort=$port -o StrictHostKeyChecking=$know_hosts_strategy -o IdentityFile=$identity_file $username@$server << EOF
Review Comment:
Carry your pr or open a new PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] chore(test): Fix sftp test failed in python [incubator-opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on PR #3436:
URL: https://github.com/apache/incubator-opendal/pull/3436#issuecomment-1787434216
The base has been changed to https://github.com/apache/incubator-opendal/pull/3430, you can continue your work now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] chore(test): Fix sftp test failed in python [incubator-opendal]
Posted by "Zheaoli (via GitHub)" <gi...@apache.org>.
Zheaoli commented on code in PR #3436:
URL: https://github.com/apache/incubator-opendal/pull/3436#discussion_r1377789469
##########
.github/services/sftp/sftp/action.yml:
##########
@@ -24,7 +24,28 @@ runs:
- name: Setup Sftp
shell: bash
working-directory: fixtures/sftp
- run: docker-compose -f docker-compose-sftp.yml up -d
+ run: |
+ docker-compose -f docker-compose-sftp.yml up -d
+ set +ex
Review Comment:
Like postgresql
```text
╭─ manjusaka@manjusaka-garuda ~/Documents/projects/opendal jupyter 3.12.0 manjusaka/fix-sftp-test *1 23:34:00
╰─ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
1e70063710ea postgres:13 "docker-entrypoint.s…" 8 seconds ago Up 7 seconds (health: starting) 0.0.0.0:5432->5432/tcp, :::5432->5432/tcp postgresql-postgres-db-1
729b4e400e39 mongo:latest "docker-entrypoint.s…" 8 days ago Up 3 hours 0.0.0.0:27017->27017/tcp, :::27017->27017/tcp mongo-mongo-1
```
The server still in starting status after 7s when `docker compose up -d` command finished
##########
.github/services/sftp/sftp/action.yml:
##########
@@ -24,7 +24,28 @@ runs:
- name: Setup Sftp
shell: bash
working-directory: fixtures/sftp
- run: docker-compose -f docker-compose-sftp.yml up -d
+ run: |
+ docker-compose -f docker-compose-sftp.yml up -d
+ set +ex
Review Comment:
Like postgresql
```text
╭─ manjusaka@manjusaka-garuda ~/Documents/projects/opendal jupyter 3.12.0 manjusaka/fix-sftp-test *1 23:34:00
╰─ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
1e70063710ea postgres:13 "docker-entrypoint.s…" 8 seconds ago Up 7 seconds (health: starting) 0.0.0.0:5432->5432/tcp, :::5432->5432/tcp postgresql-postgres-db-1
729b4e400e39 mongo:latest "docker-entrypoint.s…" 8 days ago Up 3 hours 0.0.0.0:27017->27017/tcp, :::27017->27017/tcp mongo-mongo-1
```
The server still in starting status after 7s when `docker compose up -d` command finished
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] chore(test): Fix sftp test failed in python [incubator-opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo merged PR #3436:
URL: https://github.com/apache/incubator-opendal/pull/3436
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] chore(test): Fix sftp test failed in python [incubator-opendal]
Posted by "Zheaoli (via GitHub)" <gi...@apache.org>.
Zheaoli commented on code in PR #3436:
URL: https://github.com/apache/incubator-opendal/pull/3436#discussion_r1377797407
##########
.github/services/sftp/sftp/action.yml:
##########
@@ -24,7 +24,28 @@ runs:
- name: Setup Sftp
shell: bash
working-directory: fixtures/sftp
- run: docker-compose -f docker-compose-sftp.yml up -d
+ run: |
+ docker-compose -f docker-compose-sftp.yml up -d
+ set +ex
Review Comment:
> Maybe we can use `docker compose up -d --wait`?
>
> ```shell
> --wait Wait for services to be
> running|healthy. Implies detached mode.
> --wait-timeout int Maximum duration to wait for the
> project to be running|healthy.
> ```
Seems good, I'll make a test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] chore(test): Fix sftp test failed in python [incubator-opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #3436:
URL: https://github.com/apache/incubator-opendal/pull/3436#discussion_r1377780624
##########
.github/services/sftp/sftp/action.yml:
##########
@@ -24,7 +24,28 @@ runs:
- name: Setup Sftp
shell: bash
working-directory: fixtures/sftp
- run: docker-compose -f docker-compose-sftp.yml up -d
+ run: |
+ docker-compose -f docker-compose-sftp.yml up -d
+ set +ex
Review Comment:
So we can't implement that in `health_check`? I want to make sure that we have the same setup in both CI and locally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] chore(test): Fix sftp test failed in python [incubator-opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #3436:
URL: https://github.com/apache/incubator-opendal/pull/3436#discussion_r1377792332
##########
.github/services/sftp/sftp/action.yml:
##########
@@ -24,7 +24,28 @@ runs:
- name: Setup Sftp
shell: bash
working-directory: fixtures/sftp
- run: docker-compose -f docker-compose-sftp.yml up -d
+ run: |
+ docker-compose -f docker-compose-sftp.yml up -d
+ set +ex
Review Comment:
Is there a standard method to confirm that the container has started following `docker compose up -d`? I prefer remove those scripts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] chore(test): Fix sftp test failed in python [incubator-opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #3436:
URL: https://github.com/apache/incubator-opendal/pull/3436#discussion_r1377762395
##########
.github/services/sftp/sftp_with_default_root/action.yml:
##########
@@ -35,6 +35,26 @@ runs:
--ulimit nofile=65536:65536 \
-p 2222:22 -d atmoz/sftp \
foo::::upload
+ set +ex
+ username="foo"
+ server="127.0.0.1"
+ port=2222
+ know_hosts_strategy="no"
+ identity_file="${{ github.workspace }}/target/ssh/id_rsa"
+ while :; do
+ (
+ sftp -oPort=$port -o StrictHostKeyChecking=$know_hosts_strategy -o IdentityFile=$identity_file $username@$server << EOF
Review Comment:
> Carry your pr or open a new PR?
Both LGTM. (In fact, you can send a PR to `polish-sftp` branch, let me help you to do this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] chore(test): Fix sftp test failed in python [incubator-opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #3436:
URL: https://github.com/apache/incubator-opendal/pull/3436#discussion_r1377758122
##########
.github/services/sftp/sftp_with_default_root/action.yml:
##########
@@ -35,6 +35,26 @@ runs:
--ulimit nofile=65536:65536 \
-p 2222:22 -d atmoz/sftp \
foo::::upload
+ set +ex
+ username="foo"
+ server="127.0.0.1"
+ port=2222
+ know_hosts_strategy="no"
+ identity_file="${{ github.workspace }}/target/ssh/id_rsa"
+ while :; do
+ (
+ sftp -oPort=$port -o StrictHostKeyChecking=$know_hosts_strategy -o IdentityFile=$identity_file $username@$server << EOF
Review Comment:
Can you submit this fix to https://github.com/apache/incubator-opendal/pull/3430 and implement as the health check in `docker-compose`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] chore(test): Fix sftp test failed in python [incubator-opendal]
Posted by "Zheaoli (via GitHub)" <gi...@apache.org>.
Zheaoli commented on code in PR #3436:
URL: https://github.com/apache/incubator-opendal/pull/3436#discussion_r1377778335
##########
.github/services/sftp/sftp/action.yml:
##########
@@ -24,7 +24,28 @@ runs:
- name: Setup Sftp
shell: bash
working-directory: fixtures/sftp
- run: docker-compose -f docker-compose-sftp.yml up -d
+ run: |
+ docker-compose -f docker-compose-sftp.yml up -d
+ set +ex
Review Comment:
No. We use `-d` flag for `docker compose`, that's means all docker operations will be run in the background.
This is not used for local development,; people will run `docker ps` manually when the test fails. The piece of script is just for CI only.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] chore(test): Fix sftp test failed in python [incubator-opendal]
Posted by "Zheaoli (via GitHub)" <gi...@apache.org>.
Zheaoli commented on code in PR #3436:
URL: https://github.com/apache/incubator-opendal/pull/3436#discussion_r1377784157
##########
.github/services/sftp/sftp/action.yml:
##########
@@ -24,7 +24,28 @@ runs:
- name: Setup Sftp
shell: bash
working-directory: fixtures/sftp
- run: docker-compose -f docker-compose-sftp.yml up -d
+ run: |
+ docker-compose -f docker-compose-sftp.yml up -d
+ set +ex
Review Comment:
Yes, the health_check in `docker-compose` is been designed to help the Docker daemon detect the dead container and restart it. Not suitable for this circumstance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org