You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@couchdb.apache.org by GitBox <gi...@apache.org> on 2020/04/21 03:39:40 UTC

[GitHub] [couchdb] YC opened a new issue #2797: Cluster set up with API fails with unknown_error

YC opened a new issue #2797:
URL: https://github.com/apache/couchdb/issues/2797


   ## Description
   Encountered an issue when trying to automate CouchDB 3.0 cluster setup with cluster setup API, described at https://docs.couchdb.org/en/stable/setup/cluster.html#the-cluster-setup-api. 
   
   If the `/` path is not accessed before finish_cluster is called, `{"error":"unknown_error","reason":"undef","ref":1124911208}` is given. 
   
   ## Steps to Reproduce
   Pre-existing `data` directories are deleted and `docker-compose up -d` is executed on all nodes
   Docker compose configuration
   ```
   version: '3'
   services:
     couchdb:
       image: couchdb:3
       environment:
         - COUCHDB_USER=user
         - COUCHDB_PASSWORD=pass
         - COUCHDB_SECRET=secret
         - NODENAME=<ip of node>
       ports:
         - "5984:5984"
         - "4369:4369"
         - "9100:9100"
       volumes:
         - ./data:/opt/couchdb/data
   ```
   Assuming a 2 node setup:
   On setup node: the following commands are executed:
   ```
   curl http://user:pass@localhost:5984/_cluster_setup
   curl --request POST \
     --url http://user:pass@localhost:5984/_cluster_setup \
     --header 'content-type: application/json' \
     --data '{
   	"action": "enable_cluster",
   	"bind_address": "0.0.0.0",
   	"username": "user",
   	"password": "pass",
   	"port": 5984,
   	"node_count": 2,
   	"remote_node": "<ip of remote node>",
   	"remote_current_user": "user",
   	"remote_current_password": "pass"
   }'
   curl --request POST \
     --url http://user:pass@localhost:5984/_cluster_setup \
     --header 'content-type: application/json' \
     --data '{
       "action": "add_node",
       "host": "<ip of remote node>",
       "port": 5984,
       "username": "user",
       "password": "pass",
       "singlenode": false
   }'
   # curl http://localhost:5984/
   curl --request POST \
     --url http://user:pass@localhost:5984/_cluster_setup \
     --header 'content-type: application/json' \
     --data '{ "action": "finish_cluster" }'
   curl http://user:pass@localhost:5984/_cluster_setup
   ```
   Output (after running commands above, with `/` request omitted):
   ```
   {"state":"cluster_enabled"}
   {"ok":true}
   {"ok":true}
   {"error":"unknown_error","reason":"undef","ref":1124911208}
   {"state":"cluster_enabled"}
   ```
   
   ## Expected Behaviour
   The finish_cluster request should complete and the state should be cluster_finished. 
   
   ## Your Environment
   `{"couchdb":"Welcome","version":"3.0.0","git_sha":"03a77db6c","uuid":"80f48d682bf369f142e1bb3632cf20bf","features":["access-ready","partitioned","pluggable-storage-engines","reshard","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}`
   
   ## Additional Context
   If a request to `/` is made before `finish_cluster`, `finish_cluster` returns response `{"ok": true}` and setup state becomes `cluster_finished`. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [couchdb] janl commented on issue #2797: Cluster set up with API fails with unknown_error

Posted by GitBox <gi...@apache.org>.
janl commented on issue #2797:
URL: https://github.com/apache/couchdb/issues/2797#issuecomment-617111407


   Thanks!
   
   first thing I noticed in the setup_node log is that the order of requests doesn’t match your listing with the curl commands above:
   
   1. `GET /_cluster_setup 200`: L229
   2. `POST /_cluster_setup {add_node: 10.0.2.2} 201`: L232
   3. `POST /_cluster_setup {finish_cluster} 201`: L243
   4. `POST /_cluster_setup 500`: L260
   
   That last one doesn’t dump out what `action` it is, but I’d assume it is the second `add_node` call, and that won’t work after the `finish_cluster` in 3.
   
   Can you verify the order of things?
   
   I agree that a better error message would help, if this is the case, so happy to keep this issue open for this regardless.
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [couchdb] janl commented on issue #2797: Cluster set up with API fails with unknown_error

Posted by GitBox <gi...@apache.org>.
janl commented on issue #2797:
URL: https://github.com/apache/couchdb/issues/2797#issuecomment-617111899


   Actually, it might be that 4. is a second call to `finish_cluster` which is also not correct, but also should not throw a weird error.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [couchdb] YC edited a comment on issue #2797: Cluster set up with API fails with unknown_error

Posted by GitBox <gi...@apache.org>.
YC edited a comment on issue #2797:
URL: https://github.com/apache/couchdb/issues/2797#issuecomment-617215014


   Here are 2 sets of test logs. 
   For the first set, the first line of run.sh was commented whereas it was uncommented in the second run. 
   Uploaded wrong run.sh, replacing...
   
   Set 1:
   [run_output.txt](https://github.com/apache/couchdb/files/4510481/run_output.txt)
   [setup_log.txt](https://github.com/apache/couchdb/files/4510482/setup_log.txt)
   [other_log.txt](https://github.com/apache/couchdb/files/4510480/other_log.txt)
   
   Set 2:
   [run_output2.txt](https://github.com/apache/couchdb/files/4510484/run_output2.txt)
   [setup_log2.txt](https://github.com/apache/couchdb/files/4510485/setup_log2.txt)
   [other_log2.txt](https://github.com/apache/couchdb/files/4510483/other_log2.txt)
   
   I noticed that the second run (with the request to `/`) succeeded after having the error `Request to create N=3 DB but only 2 node(s)` while the first still? had `[<<"log:error/2">>,<<"setup:sync_config/3 L265">>,<<"setup:finish_cluster/1 L203">>,<<"setup_httpd:handle_action/2 L99">>,<<"setup_httpd:handle_setup_req/1 L24">>,<<"chttpd:handle_req_after_auth/2 L322">>,<<"chttpd:process_request/1 L305">>,<<"chttpd:handle_request_int/1 L243">>]`. 
   
   Regarding the patch, I'm not quite sure about how I can apply it easily, given that I'm using docker. 
   If `setup.erl` is compiled when building from source, perhaps it's best if the issue is closed for now. 
   I can just GET `/` for now and investigate further once I have more time. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [couchdb] YC edited a comment on issue #2797: Cluster set up with API fails with unknown_error

Posted by GitBox <gi...@apache.org>.
YC edited a comment on issue #2797:
URL: https://github.com/apache/couchdb/issues/2797#issuecomment-617075616


   [setup_node.txt](https://github.com/apache/couchdb/files/4509110/setup_node.txt)
   [other_node.txt](https://github.com/apache/couchdb/files/4509108/other_node.txt)
   
   Please find attached. 
   Note that the logs are from a virtualbox vm (setup node) and my local machine (other node). 
   Apologies if I've done anything incorrectly as I'm new to couch. 
   Thanks. 
   
   Edit: On second thoughts, it may be better to have two actual nodes, given that 'other node' cannot reach 'setup node'. Let me know if you want me to attempt to grab the logs from a real environment. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [couchdb] janl commented on issue #2797: Cluster set up with API fails with unknown_error

Posted by GitBox <gi...@apache.org>.
janl commented on issue #2797:
URL: https://github.com/apache/couchdb/issues/2797#issuecomment-617062273


   heya, great report, thanks!
   
   Can you share your couch.log on all connected notes for the commands above. Ideally at `debug` level


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [couchdb] janl commented on issue #2797: Cluster set up with API fails with unknown_error

Posted by GitBox <gi...@apache.org>.
janl commented on issue #2797:
URL: https://github.com/apache/couchdb/issues/2797#issuecomment-617113858


   applying the patch in https://github.com/apache/couchdb/pull/2798/files should give you a better error message.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [couchdb] janl edited a comment on issue #2797: Cluster set up with API fails with unknown_error

Posted by GitBox <gi...@apache.org>.
janl edited a comment on issue #2797:
URL: https://github.com/apache/couchdb/issues/2797#issuecomment-617111407


   Thanks!
   
   first thing I noticed in the setup_node log is that the order of requests doesn’t match your listing with the curl commands above:
   
   1. `GET /_cluster_setup 200`: L229
   2. `POST /_cluster_setup {add_node: 10.0.2.2} 201`: L232
   3. `POST /_cluster_setup {finish_cluster} 201`: L243
   4. `POST /_cluster_setup 500`: L260
   
   That last one doesn’t dump out what `action` it is, but I’d assume it is the second `add_node` call, and that won’t work after the `finish_cluster` in 3.
   
   Can you verify the order of things?
   
   I agree that a better error message would help, if this is the case, so happy to keep this issue open for this regardless.
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [couchdb] YC edited a comment on issue #2797: Cluster set up with API fails with unknown_error

Posted by GitBox <gi...@apache.org>.
YC edited a comment on issue #2797:
URL: https://github.com/apache/couchdb/issues/2797#issuecomment-617215014


   Here are 2 sets of test logs. 
   For the first set, the first line of run.sh was commented whereas it was uncommented in the second run. 
   [run.sh](https://github.com/apache/couchdb/files/4510611/run.txt)
   
   Set 1:
   [run_output.txt](https://github.com/apache/couchdb/files/4510481/run_output.txt)
   [setup_log.txt](https://github.com/apache/couchdb/files/4510482/setup_log.txt)
   [other_log.txt](https://github.com/apache/couchdb/files/4510480/other_log.txt)
   
   Set 2:
   [run_output2.txt](https://github.com/apache/couchdb/files/4510484/run_output2.txt)
   [setup_log2.txt](https://github.com/apache/couchdb/files/4510485/setup_log2.txt)
   [other_log2.txt](https://github.com/apache/couchdb/files/4510483/other_log2.txt)
   
   I noticed that the second run (with the request to `/`) succeeded after having the error `Request to create N=3 DB but only 2 node(s)` while the first still? had `[<<"log:error/2">>,<<"setup:sync_config/3 L265">>,<<"setup:finish_cluster/1 L203">>,<<"setup_httpd:handle_action/2 L99">>,<<"setup_httpd:handle_setup_req/1 L24">>,<<"chttpd:handle_req_after_auth/2 L322">>,<<"chttpd:process_request/1 L305">>,<<"chttpd:handle_request_int/1 L243">>]`. 
   
   Regarding the patch, I'm not quite sure about how I can apply it easily, given that I'm using docker. 
   If `setup.erl` is compiled when building from source, perhaps it's best if the issue is closed for now. 
   I can just GET `/` for now and investigate further once I have more time. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [couchdb] YC edited a comment on issue #2797: Cluster set up with API fails with unknown_error

Posted by GitBox <gi...@apache.org>.
YC edited a comment on issue #2797:
URL: https://github.com/apache/couchdb/issues/2797#issuecomment-617215014






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [couchdb] YC edited a comment on issue #2797: Cluster set up with API fails with unknown_error

Posted by GitBox <gi...@apache.org>.
YC edited a comment on issue #2797:
URL: https://github.com/apache/couchdb/issues/2797#issuecomment-617075616


   [setup_node.txt](https://github.com/apache/couchdb/files/4509110/setup_node.txt)
   [other_node.txt](https://github.com/apache/couchdb/files/4509108/other_node.txt)
   
   Please find attached. 
   Note that the logs are from a virtualbox vm (setup node) and my local machine (other node). 
   Apologies if I've done anything incorrectly as I'm new to couch. 
   Thanks. 
   
   Edit: On second thoughts, it may be better to have two actual nodes, given that 'other node' cannot reach 'setup node'. Let me know if you want me to grab the logs from a real environment. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [couchdb] YC commented on issue #2797: Cluster set up with API fails with unknown_error

Posted by GitBox <gi...@apache.org>.
YC commented on issue #2797:
URL: https://github.com/apache/couchdb/issues/2797#issuecomment-617119726


   Yes, my bad, the script with the commands don't represent what I've written above. 
   Instead, it's:
   1. GET _cluster_setup
   2. POST _cluster_setup, "enable_cluster"
   3. POST _cluster_setup, "add_node"
   4. POST _cluster_setup, "finish_cluster"
   The 5th request, the GET was not executed. 
   The strange thing is if I add a GET request to `/` between 3 and 4, the error seems to go away. 
   
   I'll try to get a better set of logs for you. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [couchdb] mgsweet commented on issue #2797: Cluster set up with API fails with unknown_error

Posted by GitBox <gi...@apache.org>.
mgsweet commented on issue #2797:
URL: https://github.com/apache/couchdb/issues/2797#issuecomment-629634725


   It spends me three days to fix this problem... Follow almost every config and command made by @YC. The official document is very ambiguous when it comes to which node should do what. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [couchdb] YC edited a comment on issue #2797: Cluster set up with API fails with unknown_error

Posted by GitBox <gi...@apache.org>.
YC edited a comment on issue #2797:
URL: https://github.com/apache/couchdb/issues/2797#issuecomment-617119726


   Yes, my bad, the script with the commands don't represent what I've written above. 
   Instead, it's:
   1. GET _cluster_setup
   2. POST _cluster_setup, "enable_cluster"
   3. POST _cluster_setup, "add_node"
   4. POST _cluster_setup, "finish_cluster"
   
   The 5th request, the GET was not executed. 
   The strange thing is if I add a GET request to `/` between 3 and 4, the error seems to go away. 
   
   I'll try to get a better set of logs for you. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [couchdb] YC commented on issue #2797: Cluster set up with API fails with unknown_error

Posted by GitBox <gi...@apache.org>.
YC commented on issue #2797:
URL: https://github.com/apache/couchdb/issues/2797#issuecomment-617215014


   Here are 2 sets of test logs. 
   For the first set, the first line of run.sh was commented whereas it was uncommented in the second run. 
   [run.sh](https://github.com/apache/couchdb/files/4510491/run.txt)
   
   Set 1:
   [run_output.txt](https://github.com/apache/couchdb/files/4510481/run_output.txt)
   [setup_log.txt](https://github.com/apache/couchdb/files/4510482/setup_log.txt)
   [other_log.txt](https://github.com/apache/couchdb/files/4510480/other_log.txt)
   
   Set 2:
   [run_output2.txt](https://github.com/apache/couchdb/files/4510484/run_output2.txt)
   [setup_log2.txt](https://github.com/apache/couchdb/files/4510485/setup_log2.txt)
   [other_log2.txt](https://github.com/apache/couchdb/files/4510483/other_log2.txt)
   
   I noticed that the second run succeeded after having the error `Request to create N=3 DB but only 2 node(s)` while the first still? had `[<<"log:error/2">>,<<"setup:sync_config/3 L265">>,<<"setup:finish_cluster/1 L203">>,<<"setup_httpd:handle_action/2 L99">>,<<"setup_httpd:handle_setup_req/1 L24">>,<<"chttpd:handle_req_after_auth/2 L322">>,<<"chttpd:process_request/1 L305">>,<<"chttpd:handle_request_int/1 L243">>]`. 
   
   Regarding the patch, I'm not quite sure about how I can apply it easily, given that I'm using docker. 
   If `setup.erl` is compiled when building from source, perhaps it's best if the issue is closed for now. 
   I can just GET `/` for now and investigate further once I have more time. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [couchdb] YC commented on issue #2797: Cluster set up with API fails with unknown_error

Posted by GitBox <gi...@apache.org>.
YC commented on issue #2797:
URL: https://github.com/apache/couchdb/issues/2797#issuecomment-617075616


   [setup_node.txt](https://github.com/apache/couchdb/files/4509110/setup_node.txt)
   [other_node.txt](https://github.com/apache/couchdb/files/4509108/other_node.txt)
   
   Please find attached. 
   Note that the logs are from a virtualbox vm (setup node) and my local machine (other node). 
   Apologies if I've done anything incorrectly as I'm new to couch. 
   Thanks. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [couchdb] wohali commented on issue #2797: Cluster set up with API fails with unknown_error

Posted by GitBox <gi...@apache.org>.
wohali commented on issue #2797:
URL: https://github.com/apache/couchdb/issues/2797#issuecomment-629669941


   We welcome pull requests to improve documentation.
   
   If you are looking for general support with using CouchDB, please try one of these other options:
   
   * The user mailing list. Signup instructions are [here](http://couchdb.apache.org/#mailing-lists)
   * The Slack/IRC chat room. Joining instructions are [here](http://couchdb.apache.org/#chat)
   
   If that results in a clear bug that needs to be resolved, we're happy to track it on GitHub in a new issue.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org