You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/06/22 23:35:54 UTC
[GitHub] [pulsar] raghuvelagala opened a new issue, #16186: Es-sink not publishing to indexName with date append
raghuvelagala opened a new issue, #16186:
URL: https://github.com/apache/pulsar/issues/16186
**Describe the bug**
A clear and concise description of what the bug is.
Pulsar version: apache-pulsar-2.10.0
ES version: 7.17.0
Running an es-sink connector with sink-config field "indexName": "my-index-%{+yyyy-MM-dd}" as suggested in the documentation. https://pulsar.apache.org/fr/docs/next/io-elasticsearch-sink/
However whenever the connector consumes a message and pushes to ES, the date is not being appended to the index name (the date-format seems to be used as-is and not replaced) :
org.opensearch.OpenSearchStatusException: OpenSearch exception [type=invalid_index_name_exception, reason=Invalid index name [my_index-%{+yyyy-MM-dd}], must be lowercase]
at org.opensearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:202) ~[?:?]
at org.opensearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:2075) ~[?:?]
at org.opensearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:2052) ~[?:?]
at org.opensearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1775) ~[?:?]
at org.opensearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1728) ~[?:?]
at org.opensearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1696) ~[?:?]
at org.opensearch.client.RestHighLevelClient.index(RestHighLevelClient.java:961) ~[?:?]
at org.apache.pulsar.io.elasticsearch.ElasticSearchClient.indexDocument(ElasticSearchClient.java:298) ~[?:?]
at org.apache.pulsar.io.elasticsearch.ElasticSearchSink.write(ElasticSearchSink.java:110) ~[?:?]
at org.apache.pulsar.functions.instance.JavaInstanceRunnable.sendOutputMessage(JavaInstanceRunnable.java:380) ~[org.apache.pulsar-pulsar-functions-instance-2.10.0.jar:2.10.0]
at org.apache.pulsar.functions.instance.JavaInstanceRunnable.handleResult(JavaInstanceRunnable.java:363) ~[org.apache.pulsar-pulsar-functions-instance-2.10.0.jar:2.10.0]
at org.apache.pulsar.functions.instance.JavaInstanceRunnable.run(JavaInstanceRunnable.java:304) ~[org.apache.pulsar-pulsar-functions-instance-2.10.0.jar:2.10.0]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_322]
Suppressed: org.opensearch.client.ResponseException: method [POST], host [http://XXXXXXXXXXX:9200], URI [/my_index-%25%7B+yyyy-MM-dd%7D/_doc?timeout=1m], status line [HTTP/1.1 400 Bad Request
**To Reproduce**
Steps to reproduce the behavior:
Set ES indexName with date as in the documentation: 'logs-%{+yyyy-MM-dd};
**Expected behavior**
A clear and concise description of what you expected to happen.
An index with current date created in Elasticsearch. E.g. my-index-2022-06-22
**Screenshots**
If applicable, add screenshots to help explain your problem.
**Desktop (please complete the following information):**
- OS: [e.g. iOS]
**Additional context**
Add any other context about the problem here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] silrag commented on issue #16186: Es-sink not publishing to indexName with date append
Posted by GitBox <gi...@apache.org>.
silrag commented on issue #16186:
URL: https://github.com/apache/pulsar/issues/16186#issuecomment-1163775422
@fantapsody I see you originally contributed to this feature in [Issue 13019]. Am I using the right version with the feature available/enabled? Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] silrag commented on issue #16186: Es-sink not publishing to indexName with date append
Posted by GitBox <gi...@apache.org>.
silrag commented on issue #16186:
URL: https://github.com/apache/pulsar/issues/16186#issuecomment-1176988326
Thanks, the new version works for me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] fantapsody commented on issue #16186: Es-sink not publishing to indexName with date append
Posted by GitBox <gi...@apache.org>.
fantapsody commented on issue #16186:
URL: https://github.com/apache/pulsar/issues/16186#issuecomment-1165397657
Hi @silrag , I just checked the code, and the PR was cherry-picked to the branch 2.10 after the release of 2.10.0: https://github.com/apache/pulsar/tree/v2.10.0/pulsar-io/elastic-search/src/main/java/org/apache/pulsar/io/elasticsearch
You can wait for the release of 2.10.1 which should include this commit but has not been released yet, or you can build the sink from the master or the 2.10 branch by yourself and try it out. You may also try [a newer 2.10 build](https://github.com/streamnative/pulsar/releases/tag/v2.10.0.7) from StreamNative.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] fantapsody commented on issue #16186: Es-sink not publishing to indexName with date append
Posted by GitBox <gi...@apache.org>.
fantapsody commented on issue #16186:
URL: https://github.com/apache/pulsar/issues/16186#issuecomment-1176224607
@silrag The newly released 2.10.1 should include the commit, please give it a try: https://github.com/apache/pulsar/releases/tag/v2.10.1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] silrag closed issue #16186: Es-sink not publishing to indexName with date append
Posted by GitBox <gi...@apache.org>.
silrag closed issue #16186: Es-sink not publishing to indexName with date append
URL: https://github.com/apache/pulsar/issues/16186
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org