You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@shenyu.apache.org by GitBox <gi...@apache.org> on 2021/10/08 09:22:35 UTC

[GitHub] [incubator-shenyu] longhujing opened a new issue #2200: [BUG] DividePlugin插件, 如果Url中包含特殊字符如#等会导致请求错误

longhujing opened a new issue #2200:
URL: https://github.com/apache/incubator-shenyu/issues/2200


   ### Is there an existing issue for this?
   
   - [X] I have searched the existing issues
   
   ### Current Behavior
   
   http://localhost:9195/queryItem?keyword=#&page=1&pageSize=1
   不使用Soul网关，走本地相同的条件在PostMan是OK的，但是用Soul网关就会出现错误
   {
        "code": 500001,
        "message": "页号不能为空"
   }
   
   ### Expected Behavior
   
   能够像本地请求一样返回正确的结果而不是报错
   
   ### Steps To Reproduce
   
   1. 随意写一个分页的HTTP接口
   2. 参数中包含#等一个或多个特殊字符
   3. 业务端无法正确获取到页号信息
   
   
   http://localhost:9195/queryItem?keyword=#&page=1&pageSize=1
   @GetMapping(values = "/queryItem")
   public Object queryItem(String keyword, Integer page, Integer pageSize) {
           System.out.println(keyword);
           System.out.println(page);
           System.out.println(size);
   }
   
   ### Environment
   
   ```markdown
   ShenYu version(s): 2.3.0
   ```
   
   
   ### Debug logs
   
   _No response_
   
   ### Anything else?
   
   这个问题看了下是DividePlugin插件中URL拼参数造成的，将
   String query = exchange.getRequest().getURI().getQuery();
   替换成
   String query = exchange.getRequest().getURI().getRawQuery();
   
   就可以了，目前我验证是OK的，大佬们可以提出一些其他的改进意见


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@shenyu.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-shenyu] qicz commented on issue #2200: [BUG] DividePlugin插件, 如果Url中包含特殊字符如#等会导致请求错误

Posted by GitBox <gi...@apache.org>.
qicz commented on issue #2200:
URL: https://github.com/apache/incubator-shenyu/issues/2200#issuecomment-939769975


   https://stackoverflow.com/questions/2295223/how-to-find-out-if-string-has-already-been-url-encoded
   
   > Decode, compare to original. If it does differ, original is encoded. If it doesn't differ, original isn't encoded. 
   
   我参考这个，先在跟踪一下。
   
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@shenyu.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-shenyu] longhujing edited a comment on issue #2200: [BUG] DividePlugin插件, 如果Url中包含特殊字符如#等会导致请求错误

Posted by GitBox <gi...@apache.org>.
longhujing edited a comment on issue #2200:
URL: https://github.com/apache/incubator-shenyu/issues/2200#issuecomment-939197335


   抱歉，昨天我提出的解决方案其实存在一些问题，我在进行debug的时候，发现问题的根本原因是这样的
   
   在WebClientPlugin里面构造出来的URL是这样的
   http://172.20.218.131:8080/test?name=#&page=1&pageSize=1
   事实上我们期望的是这样的
   http://172.20.218.131:8080/test?name=%23&page=1&pageSize=1，也就是构造出来的URL需要对特殊字符进行转义处理，但是这里并没有，就会导致应用解析错误
   
   ![image](https://user-images.githubusercontent.com/54448899/136638072-57be40db-88b5-43e0-8211-99fabff73527.png)
   
   然后我昨天提的，将getQuery()改成getRawQuery()，这样拿到的query参数就是已经转义过的，但是在构造新的URL的时候出现了意料之外的结果
   ![image](https://user-images.githubusercontent.com/54448899/136638288-52b62ab1-fa44-4864-a224-4623f2989158.png)
   这里又对%进行了一次转义，转成了%25，于是
   本来期望的是这样的：http://172.20.218.131:8080/test?name=%23&page=1&pageSize=1
   结果给我变成了这样：http://172.20.218.131:8080/test?name=%2523&page=1&pageSize=1
   
   所以这里应该重写uri的构造方法，也就是替换掉这一段
   WebClient.RequestBodySpec requestBodySpec = webClient.method(method).uri(urlPath);
   
   使用自定义的uri构造方式，像这样
           WebClient.RequestBodySpec requestBodySpec = webClient
                   .method(method)
                   .uri(urlPath, uriBuilder -> uriBuilder.replaceQueryParams(exchange.getRequest().getQueryParams()).build());
   ![image](https://user-images.githubusercontent.com/54448899/136638413-6a11fc05-69c7-4c57-9dee-b61b92caadae.png)
   
   这样构造出来的URL就符合期望了，而且这里是直接替换掉url上面的请求参数，因此我认为DividePlugin里面追加query参数的步骤都是可以省略的
   ![image](https://user-images.githubusercontent.com/54448899/136638463-de2cbd70-ea03-411d-b4c4-5dad25d7b807.png)
   
   对于昨天提出的没有经过详细验证的方案我很抱歉，不好意思挖坑了🤣
   
   @qicz 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@shenyu.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-shenyu] qicz commented on issue #2200: [BUG] DividePlugin插件, 如果Url中包含特殊字符如#等会导致请求错误

Posted by GitBox <gi...@apache.org>.
qicz commented on issue #2200:
URL: https://github.com/apache/incubator-shenyu/issues/2200#issuecomment-939210896


   @longhujing  
   
   > 这里又对%进行了一次转义，转成了%25，于是
   本来期望的是这样的：http://172.20.218.131:8080/test?name=%23&page=1&pageSize=1
   结果给我变成了这样：http://172.20.218.131:8080/test?name=%2523&page=1&pageSize=1
   
   事实上，就是这样的处理的
   ![image](https://user-images.githubusercontent.com/2174082/136641372-a17889b4-fbe1-4ada-998b-ad4366e89930.png)
   
   同时，使用`getRawQuery`是可以正确获取数据的。 这是webclient发送数据的时候对数据进行了编码， 服务端在数据参数mapping时进行了解码。
   
   所以你的目的是拿到对的数据，还是你要的这个地址（http://172.20.218.131:8080/test?name=%23&page=1&pageSize=1）做其他的事？
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@shenyu.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-shenyu] longhujing commented on issue #2200: [BUG] DividePlugin插件, 如果Url中包含特殊字符如#等会导致请求错误

Posted by GitBox <gi...@apache.org>.
longhujing commented on issue #2200:
URL: https://github.com/apache/incubator-shenyu/issues/2200#issuecomment-939197335


   抱歉，昨天我提出的解决方案其实存在一些问题，我在进行debug的时候，发现问题的根本原因是这样的
   
   在WebClientPlugin里面构造出来的URL是这样的
   http://172.20.218.131:8080/test?name=#&page=1&pageSize=1
   事实上我们期望的是这样的
   http://172.20.218.131:8080/test?name=%23&page=1&pageSize=1，也就是构造出来的URL需要对特殊字符进行转义处理，但是这里并没有，就会导致应用解析错误
   
   ![image](https://user-images.githubusercontent.com/54448899/136638072-57be40db-88b5-43e0-8211-99fabff73527.png)
   
   然后我昨天提的，将getQuery()改成getRawQuery()，这样拿到的query参数就是已经转义过的，但是在构造新的URL的时候出现了意料之外的结果
   ![image](https://user-images.githubusercontent.com/54448899/136638288-52b62ab1-fa44-4864-a224-4623f2989158.png)
   这里又对%进行了一次转义，转成了%25，于是
   本来期望的是这样的：http://172.20.218.131:8080/test?name=%23&page=1&pageSize=1
   结果给我变成了这样：http://172.20.218.131:8080/test?name=%2523&page=1&pageSize=1
   
   所以这里应该重写uri的构造方法，也就是替换掉这一段
   WebClient.RequestBodySpec requestBodySpec = webClient.method(method).uri(urlPath);
   
   使用自定义的uri构造方式，像这样
           WebClient.RequestBodySpec requestBodySpec = webClient
                   .method(method)
                   .uri(urlPath, uriBuilder -> uriBuilder.replaceQueryParams(exchange.getRequest().getQueryParams()).build());
   ![image](https://user-images.githubusercontent.com/54448899/136638413-6a11fc05-69c7-4c57-9dee-b61b92caadae.png)
   
   这样构造出来的URL就符合期望了，而且这里是直接替换掉url上面的请求参数，因此我认为DividePlugin里面追加query参数的步骤都是可以省略的
   ![image](https://user-images.githubusercontent.com/54448899/136638463-de2cbd70-ea03-411d-b4c4-5dad25d7b807.png)
   
   另外，我在测试的时候，如果参数是%的话还是有问题，不使用Soul也是有问题的，这个可能是Tomcat的问题吧...
   
   对于昨天提出的没有经过详细验证的方案我很抱歉，不好意思挖坑了🤣
   
   @qicz 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@shenyu.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-shenyu] qicz commented on issue #2200: [BUG] DividePlugin插件, 如果Url中包含特殊字符如#等会导致请求错误

Posted by GitBox <gi...@apache.org>.
qicz commented on issue #2200:
URL: https://github.com/apache/incubator-shenyu/issues/2200#issuecomment-939285424


   @yu199195 @tydhot 看看大佬们的意见吧。
   
   我尝试了好几个方式，的确有些问题。 使用uribuild对webclient可以，但是对nettyclient我找了，没有对应的实现。
   
   我今天我还尝试去把request的queryParam拿出来自己拼接，发现queryParam是decode过的。 对%23这种的处理还是有问题。 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@shenyu.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-shenyu] tuohai666 closed issue #2200: [BUG] DividePlugin插件, 如果Url中包含特殊字符如#等会导致请求错误

Posted by GitBox <gi...@apache.org>.
tuohai666 closed issue #2200:
URL: https://github.com/apache/incubator-shenyu/issues/2200


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@shenyu.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-shenyu] qicz commented on issue #2200: [BUG] DividePlugin插件, 如果Url中包含特殊字符如#等会导致请求错误

Posted by GitBox <gi...@apache.org>.
qicz commented on issue #2200:
URL: https://github.com/apache/incubator-shenyu/issues/2200#issuecomment-939250386


   经过反复验证和测试，发现`getQuery`对中文处理友好，但对带有特殊字符，如%23，这种处理不友好。因为`getQuery`会信息`decode`处理。而`getRawQuery`刚好相反。所以个人建议，针对包含特殊字符的参数，传参的时候自己进行URI编码，获取参数之后，再自己解码，可能是更好的选择。


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@shenyu.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-shenyu] longhujing edited a comment on issue #2200: [BUG] DividePlugin插件, 如果Url中包含特殊字符如#等会导致请求错误

Posted by GitBox <gi...@apache.org>.
longhujing edited a comment on issue #2200:
URL: https://github.com/apache/incubator-shenyu/issues/2200#issuecomment-939279558


   我试过你这种做法，我之前是这样弄的：
   1. 我通过getQuery()拿到路径参数，路径参数包含一些特殊字符比如#等
   2. 然后我对特殊字符进行转义，把#转化成%23
   3. 但是到了最后构造请求的时候变成了 %2523，导致我下一层应用获取到的数据就变成了%23，实际上我期望的应该是#，总不能让我下一层应用做一次转码吧
   示例如下：
   ![image](https://user-images.githubusercontent.com/54448899/136655683-785a76a4-1d78-4299-b7e9-fe6bdd3cddfd.png)
   ![image](https://user-images.githubusercontent.com/54448899/136655704-56f6a1ce-f8a7-4b84-85fa-e28c40a64ed5.png)
   ![image](https://user-images.githubusercontent.com/54448899/136655775-24f135e7-152b-43e0-bf78-1eab3f947fb6.png)
   
   
   而且，更重要的是，在不接入Shenyu网关之前，前端都无需做任何的改动（比如你说的对特殊字符编码之类的），但是接入了Shenyu网关之后却要多做一次转化，我觉得这是非常不合理的。
   
   目前我个人更倾向于在网关层屏蔽掉这个问题 @qicz 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@shenyu.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-shenyu] yu199195 commented on issue #2200: [BUG] DividePlugin插件, 如果Url中包含特殊字符如#等会导致请求错误

Posted by GitBox <gi...@apache.org>.
yu199195 commented on issue #2200:
URL: https://github.com/apache/incubator-shenyu/issues/2200#issuecomment-947414511


   are you origin url is ：
   ```
    /queryItem?keyword=#&page=1&pageSize=1 
   ```
   
   not is  this ?
   `queryItem?keyword=%23&page=1&pageSize=1`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@shenyu.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-shenyu] yu199195 edited a comment on issue #2200: [BUG] DividePlugin插件, 如果Url中包含特殊字符如#等会导致请求错误

Posted by GitBox <gi...@apache.org>.
yu199195 edited a comment on issue #2200:
URL: https://github.com/apache/incubator-shenyu/issues/2200#issuecomment-947414511


   are you original url is ：
   ```
    /queryItem?keyword=#&page=1&pageSize=1 
   ```
   
   not is  this ?
   `queryItem?keyword=%23&page=1&pageSize=1`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@shenyu.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-shenyu] qicz edited a comment on issue #2200: [BUG] DividePlugin插件, 如果Url中包含特殊字符如#等会导致请求错误

Posted by GitBox <gi...@apache.org>.
qicz edited a comment on issue #2200:
URL: https://github.com/apache/incubator-shenyu/issues/2200#issuecomment-939200028


   @longhujing OK，我昨天针对你说的情况进行了对应的调整和测试，但因为没有全面场景验证，只是简单模拟你提到的场景。我今天再看看，如果这个地方可以设计成通用的话，我继续调整一下。
   
   不过，就我的了解，uriencode和decode应该是不需要特别处理的，框架都会自行处理，以springboot来说，在进行mapping的时候是会处理的。


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@shenyu.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-shenyu] qicz edited a comment on issue #2200: [BUG] DividePlugin插件, 如果Url中包含特殊字符如#等会导致请求错误

Posted by GitBox <gi...@apache.org>.
qicz edited a comment on issue #2200:
URL: https://github.com/apache/incubator-shenyu/issues/2200#issuecomment-939200028


   @longhujing OK，我昨天针对你说的情况进行了对应的调整和测试，但因为没有全面场景验证，只是简单模拟你提到的场景。我今天再看看，如果这个地方可以设计成通用的话，我继续调整一下。


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@shenyu.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-shenyu] longhujing commented on issue #2200: [BUG] DividePlugin插件, 如果Url中包含特殊字符如#等会导致请求错误

Posted by GitBox <gi...@apache.org>.
longhujing commented on issue #2200:
URL: https://github.com/apache/incubator-shenyu/issues/2200#issuecomment-939279558


   我试过你这种做法，我之前是这样弄的：
   1. 我通过getQuery()拿到路径参数，路径参数包含一些特殊字符比如#等
   2. 然后我对特殊字符进行转义，把#转化成%25
   3. 但是到了最后构造请求的时候变成了 %2325，导致我下一层应用获取到的数据就变成了%25，实际上我期望的应该是#，总不能让我下一层应用做一次转码吧
   示例如下：
   ![image](https://user-images.githubusercontent.com/54448899/136655683-785a76a4-1d78-4299-b7e9-fe6bdd3cddfd.png)
   ![image](https://user-images.githubusercontent.com/54448899/136655704-56f6a1ce-f8a7-4b84-85fa-e28c40a64ed5.png)
   ![image](https://user-images.githubusercontent.com/54448899/136655775-24f135e7-152b-43e0-bf78-1eab3f947fb6.png)
   
   
   而且，更重要的是，在不接入Shenyu网关之前，前端都无需做任何的改动（比如你说的对特殊字符编码之类的），但是接入了Shenyu网关之后却要多做一次转化，我觉得这是非常不合理的。
   
   目前我个人更倾向于在网关层屏蔽掉这个问题 @qicz 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@shenyu.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-shenyu] qicz commented on issue #2200: [BUG] DividePlugin插件, 如果Url中包含特殊字符如#等会导致请求错误

Posted by GitBox <gi...@apache.org>.
qicz commented on issue #2200:
URL: https://github.com/apache/incubator-shenyu/issues/2200#issuecomment-939200028


   @longhujing OK，我昨天针对你说的情况进行了对应的调整和测试，但因为没有全面场景验证，只是简单模拟你提到的场景。我今天再看看，如果这个地方可以设计成通用的话，我继续调整一下。


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@shenyu.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org