You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "EthanWang95 (via GitHub)" <gi...@apache.org> on 2023/03/14 05:44:54 UTC
[GitHub] [incubator-seatunnel] EthanWang95 opened a new issue, #4350: titleES import automatically creates a timestamp
EthanWang95 opened a new issue, #4350:
URL: https://github.com/apache/incubator-seatunnel/issues/4350
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### What happened
In version 2.1, importing data to ES can automatically generate a timestamp according to the ${now} parameter, but it seems to be invalid in version 2.3.
### SeaTunnel Version
2.3.0
### SeaTunnel Config
```conf
env {
# seatunnel defined streaming batch duration in seconds
## [spark.app.name](http://spark.app.name) = "SeaTunnel"
#spark.sql.catalogImplementation = "hive"
spark.executor.instances = 10
spark.executor.cores = 20
spark.executor.memory = "40g"
}
source {
jdbc {
driver = "org.postgresql.Driver"
url = "jdbc:postgresql:/****:5432/cdp_edge"
user = "dbadmin"
password = "**"
table = "cdp_rep_master"
result_table_name = "cdp_rep_master"
#jdbc.SSL = "true"
#jdbc.SSLKeyStorePath = "/data/ludp_jks/keystore.jks"
query = "SELECT *,zbpli as rowkey from cml.cdp_rep_master where to_timestamp(update_timestamp,'yyyyMMddhh24miss') > current_date - interval '7 day'"
}
}
transform {
# split data by specific delimiter
# you can also use other filter plugins, such as sql
# sql {
# sql = "select * from accesslog where request_time > 1000"
#}
}
sink {
# choose stdout output plugin to output data to console
# Console {}
elasticsearch {
hosts = ["*******:9201"]
index = "cdp_rep_master_uat_${now}_01"
index_time_format = "yyyyMMdd"
index_type = "seatunnel"
}
}
```
### Running Command
```shell
nohup ./bin/start-seatunnel-spark-connector-v2.sh --master yarn --deploy-mode client --config config/toes/prod.cml.cdp_rep_master_append.conf &
```
### Error Exception
```log
The automatically generated index name is cdp_rep_master_uat_${now}_01 ,the parameter is not valid.
```
### Flink or Spark Version
_No response_
### Java or Scala Version
_No response_
### Screenshots
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] laglangyue commented on issue #4350: titleES import automatically creates a timestamp
Posted by "laglangyue (via GitHub)" <gi...@apache.org>.
laglangyue commented on issue #4350:
URL: https://github.com/apache/incubator-seatunnel/issues/4350#issuecomment-1474801424
well, maybe you use connector-v1 in version 2.1, I think it not supports in es-connector, because I don't find any code to handler the expression `${now}`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] github-actions[bot] commented on issue #4350: titleES import automatically creates a timestamp
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #4350:
URL: https://github.com/apache/incubator-seatunnel/issues/4350#issuecomment-1518440996
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] iture123 commented on issue #4350: titleES import automatically creates a timestamp
Posted by "iture123 (via GitHub)" <gi...@apache.org>.
iture123 commented on issue #4350:
URL: https://github.com/apache/incubator-seatunnel/issues/4350#issuecomment-1475253296
Connector-v2 ES sink do not support the expression ${now},but you can use source field name as a variable in index(e.g.
cdp_rep_master_uat_${rowkey} ). https://seatunnel.apache.org/docs/2.3.0/connector-v2/sink/Elasticsearch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] iture123 commented on issue #4350: titleES import automatically creates a timestamp
Posted by "iture123 (via GitHub)" <gi...@apache.org>.
iture123 commented on issue #4350:
URL: https://github.com/apache/incubator-seatunnel/issues/4350#issuecomment-1475256000
I think it is better to define some system built-in variables using in index,It can be expressed as ${sys:xx(args)},e.g. ${sys:now(yyyyMMdd)}.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org