You are viewing a plain text version of this content. The canonical link for it is here.
Posted to devnull@infra.apache.org by "twalthr (GitHub)" <gi...@apache.org> on 2018/08/24 11:56:45 UTC
[GitHub] [flink] twalthr opened pull request #6611: [FLINK-3875]
[connectors] Add an upsert table sink factory for Elasticsearch
## What is the purpose of the change
This PR adds full support for Elasticsearch to be used with Table & SQL API as well as SQL Client.
## Brief change log
This PR includes:
- Elasticsearch 6 upsert table sink
- Elasticsearch 6 table factory
- Elasticsearch table descriptors & validators
- Unit tests
- SQL Client end-to-end test
- Website documentation
## Verifying this change
- Unit tests for descriptors, factory, and sink
- SQL Client end-to-end test extended with Elasticsearch sink
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): yes
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): yes
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? yes
- If yes, how is the feature documented? docs
[ Full content available at: https://github.com/apache/flink/pull/6611 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [flink] twalthr commented on pull request #6611:
[FLINK-3875] [connectors] Add an upsert table sink factory for Elasticsearch
Posted by "twalthr (GitHub)" <gi...@apache.org>.
If this were not runtime code, I would agree. But I would use a good old for loop here just to be sure that we don't create a lot of helper objects underneath etc.
[ Full content available at: https://github.com/apache/flink/pull/6611 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [flink] dawidwys commented on pull request #6611:
[FLINK-3875] [connectors] Add an upsert table sink factory for Elasticsearch
Posted by "dawidwys (GitHub)" <gi...@apache.org>.
+1 for `protocol`
[ Full content available at: https://github.com/apache/flink/pull/6611 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [flink] fhueske commented on pull request #6611:
[FLINK-3875] [connectors] Add an upsert table sink factory for Elasticsearch
Posted by "fhueske (GitHub)" <gi...@apache.org>.
`schema` -> `rest-schema` or `protocol`?
[ Full content available at: https://github.com/apache/flink/pull/6611 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [flink] fhueske commented on pull request #6611:
[FLINK-3875] [connectors] Add an upsert table sink factory for Elasticsearch
Posted by "fhueske (GitHub)" <gi...@apache.org>.
`kafka` -> `elasticsearch`
[ Full content available at: https://github.com/apache/flink/pull/6611 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [flink] dawidwys commented on pull request #6611:
[FLINK-3875] [connectors] Add an upsert table sink factory for Elasticsearch
Posted by "dawidwys (GitHub)" <gi...@apache.org>.
In case there is no key fields in the query (e.g. in case of simple SELECT * FROM source) we will be constantly updating a single row.
I think in this case we should work in "append" mode to allow e.g. ETL case when we want to move data from one e.g. Kafka to ES.
[ Full content available at: https://github.com/apache/flink/pull/6611 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [flink] dawidwys commented on pull request #6611:
[FLINK-3875] [connectors] Add an upsert table sink factory for Elasticsearch
Posted by "dawidwys (GitHub)" <gi...@apache.org>.
How about link to json format in the dependencies table?
[ Full content available at: https://github.com/apache/flink/pull/6611 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [flink] dawidwys commented on pull request #6611:
[FLINK-3875] [connectors] Add an upsert table sink factory for Elasticsearch
Posted by "dawidwys (GitHub)" <gi...@apache.org>.
Could we keep all of those properties from the start in `DescriptorProperties` and in here just merge them into the `properties` parameter?
This way we would save some boiler plate code for checking against null. What do you think?
[ Full content available at: https://github.com/apache/flink/pull/6611 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [flink] twalthr commented on issue #6611: [FLINK-3875]
[connectors] Add an upsert table sink factory for Elasticsearch
Posted by "twalthr (GitHub)" <gi...@apache.org>.
Thank you @fhueske and @dawidwys for your reviews. I hope I could address all of them. I also rebased the PR. I just saw that the PR still removes some unrelated class. I will out this in a hotfix commit when merging.
[ Full content available at: https://github.com/apache/flink/pull/6611 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [flink] twalthr commented on pull request #6611:
[FLINK-3875] [connectors] Add an upsert table sink factory for Elasticsearch
Posted by "twalthr (GitHub)" <gi...@apache.org>.
I think we can also remove it. As the comment suggests it doesn't check much.
[ Full content available at: https://github.com/apache/flink/pull/6611 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [flink] asfgit closed pull request #6611: [FLINK-3875]
[connectors] Add an upsert table sink factory for Elasticsearch
Posted by "asfgit (GitHub)" <gi...@apache.org>.
[ pull request closed by asfgit ]
[ Full content available at: https://github.com/apache/flink/pull/6611 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [flink] twalthr commented on pull request #6611:
[FLINK-3875] [connectors] Add an upsert table sink factory for Elasticsearch
Posted by "twalthr (GitHub)" <gi...@apache.org>.
I think we can also remove it. As the comment suggests it doesn't check much. Update modes are optional for custom factories anyway.
[ Full content available at: https://github.com/apache/flink/pull/6611 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [flink] twalthr commented on pull request #6611:
[FLINK-3875] [connectors] Add an upsert table sink factory for Elasticsearch
Posted by "twalthr (GitHub)" <gi...@apache.org>.
You never know where a service is located. I think exposing this property is useful that's why Elasticsearch is also exposing it.
[ Full content available at: https://github.com/apache/flink/pull/6611 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [flink] dawidwys commented on pull request #6611:
[FLINK-3875] [connectors] Add an upsert table sink factory for Elasticsearch
Posted by "dawidwys (GitHub)" <gi...@apache.org>.
Same as above.
[ Full content available at: https://github.com/apache/flink/pull/6611 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [flink] twalthr commented on issue #6611: [FLINK-3875]
[connectors] Add an upsert table sink factory for Elasticsearch
Posted by "twalthr (GitHub)" <gi...@apache.org>.
CC @tzulitai @dawidwys
[ Full content available at: https://github.com/apache/flink/pull/6611 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [flink] dawidwys commented on pull request #6611:
[FLINK-3875] [connectors] Add an upsert table sink factory for Elasticsearch
Posted by "dawidwys (GitHub)" <gi...@apache.org>.
Could we use `MemorySize` for this? It is quite cumbersome to switch to it later. Especially if previously the value was specified in Mb.
[ Full content available at: https://github.com/apache/flink/pull/6611 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [flink] dawidwys commented on pull request #6611:
[FLINK-3875] [connectors] Add an upsert table sink factory for Elasticsearch
Posted by "dawidwys (GitHub)" <gi...@apache.org>.
Personally would prefer java 8's stream version of this function:
private String createKey(Row row) {
return Arrays.stream(keyFieldIndices).mapToObj(i -> {
final Object value = row.getField(i);
if (value == null) {
return keyNullLiteral;
} else {
return value.toString();
}
}).collect(Collectors.joining("keyDelimiter"));
}
It is more readable at least for me.
[ Full content available at: https://github.com/apache/flink/pull/6611 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [flink] dawidwys commented on pull request #6611:
[FLINK-3875] [connectors] Add an upsert table sink factory for Elasticsearch
Posted by "dawidwys (GitHub)" <gi...@apache.org>.
Does it have any other goal than setting rest api version? Wouldn't it make more sense to choose from api version?
[ Full content available at: https://github.com/apache/flink/pull/6611 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [flink] dawidwys commented on pull request #6611:
[FLINK-3875] [connectors] Add an upsert table sink factory for Elasticsearch
Posted by "dawidwys (GitHub)" <gi...@apache.org>.
Does it make sense to perform this validation here? This validation most probably has to be always repeated in the `*TableFactory`. Or am I missing something?
[ Full content available at: https://github.com/apache/flink/pull/6611 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [flink] dawidwys commented on pull request #6611:
[FLINK-3875] [connectors] Add an upsert table sink factory for Elasticsearch
Posted by "dawidwys (GitHub)" <gi...@apache.org>.
It is no longer `(in MB)`, by default if no unit is provided it will be treated as bytes. I would also add info about the minimal restriction here.
[ Full content available at: https://github.com/apache/flink/pull/6611 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org
[GitHub] [flink] twalthr commented on issue #6611: [FLINK-3875]
[connectors] Add an upsert table sink factory for Elasticsearch
Posted by "twalthr (GitHub)" <gi...@apache.org>.
Thank you @dawidwys. Travis gave green light. I will merge this...
[ Full content available at: https://github.com/apache/flink/pull/6611 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org