You are viewing a plain text version of this content. The canonical link for it is here.
Posted to devnull@infra.apache.org by "twalthr (GitHub)" <gi...@apache.org> on 2018/08/24 11:56:45 UTC

[GitHub] [flink] twalthr opened pull request #6611: [FLINK-3875] [connectors] Add an upsert table sink factory for Elasticsearch

## What is the purpose of the change

This PR adds full support for Elasticsearch to be used with Table & SQL API as well as SQL Client.

## Brief change log

This PR includes:
  - Elasticsearch 6 upsert table sink
  - Elasticsearch 6 table factory
  - Elasticsearch table descriptors & validators
  - Unit tests
  - SQL Client end-to-end test
  - Website documentation

## Verifying this change

- Unit tests for descriptors, factory, and sink
- SQL Client end-to-end test extended with Elasticsearch sink

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): yes
  - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
  - The serializers: no
  - The runtime per-record code paths (performance sensitive): yes
  - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: no
  - The S3 file system connector: no

## Documentation

  - Does this pull request introduce a new feature?  yes
  - If yes, how is the feature documented? docs


[ Full content available at: https://github.com/apache/flink/pull/6611 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org

[GitHub] [flink] twalthr commented on pull request #6611: [FLINK-3875] [connectors] Add an upsert table sink factory for Elasticsearch

Posted by "twalthr (GitHub)" <gi...@apache.org>.
If this were not runtime code, I would agree. But I would use a good old for loop here just to be sure that we don't create a lot of helper objects underneath etc.

[ Full content available at: https://github.com/apache/flink/pull/6611 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org

[GitHub] [flink] dawidwys commented on pull request #6611: [FLINK-3875] [connectors] Add an upsert table sink factory for Elasticsearch

Posted by "dawidwys (GitHub)" <gi...@apache.org>.
+1 for `protocol`

[ Full content available at: https://github.com/apache/flink/pull/6611 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org

[GitHub] [flink] fhueske commented on pull request #6611: [FLINK-3875] [connectors] Add an upsert table sink factory for Elasticsearch

Posted by "fhueske (GitHub)" <gi...@apache.org>.
`schema` -> `rest-schema` or `protocol`?

[ Full content available at: https://github.com/apache/flink/pull/6611 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org

[GitHub] [flink] fhueske commented on pull request #6611: [FLINK-3875] [connectors] Add an upsert table sink factory for Elasticsearch

Posted by "fhueske (GitHub)" <gi...@apache.org>.
`kafka` -> `elasticsearch`

[ Full content available at: https://github.com/apache/flink/pull/6611 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org

[GitHub] [flink] dawidwys commented on pull request #6611: [FLINK-3875] [connectors] Add an upsert table sink factory for Elasticsearch

Posted by "dawidwys (GitHub)" <gi...@apache.org>.
In case there is no key fields in the query (e.g. in case of simple SELECT * FROM source) we will be constantly updating a single row.

I think in this case we should work in "append" mode to allow e.g. ETL case when we want to move data from one e.g. Kafka to ES.

[ Full content available at: https://github.com/apache/flink/pull/6611 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org

[GitHub] [flink] dawidwys commented on pull request #6611: [FLINK-3875] [connectors] Add an upsert table sink factory for Elasticsearch

Posted by "dawidwys (GitHub)" <gi...@apache.org>.
How about link to json format in the dependencies table?

[ Full content available at: https://github.com/apache/flink/pull/6611 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org

[GitHub] [flink] dawidwys commented on pull request #6611: [FLINK-3875] [connectors] Add an upsert table sink factory for Elasticsearch

Posted by "dawidwys (GitHub)" <gi...@apache.org>.
Could we keep all of those properties from the start in `DescriptorProperties` and in here just merge them into the `properties` parameter? 

This way we would save some boiler plate code for checking against null. What do you think?

[ Full content available at: https://github.com/apache/flink/pull/6611 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org

[GitHub] [flink] twalthr commented on issue #6611: [FLINK-3875] [connectors] Add an upsert table sink factory for Elasticsearch

Posted by "twalthr (GitHub)" <gi...@apache.org>.
Thank you @fhueske and @dawidwys for your reviews. I hope I could address all of them. I also rebased the PR. I just saw that the PR still removes some unrelated class. I will out this in a hotfix commit when merging.

[ Full content available at: https://github.com/apache/flink/pull/6611 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org

[GitHub] [flink] twalthr commented on pull request #6611: [FLINK-3875] [connectors] Add an upsert table sink factory for Elasticsearch

Posted by "twalthr (GitHub)" <gi...@apache.org>.
I think we can also remove it. As the comment suggests it doesn't check much.

[ Full content available at: https://github.com/apache/flink/pull/6611 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org

[GitHub] [flink] asfgit closed pull request #6611: [FLINK-3875] [connectors] Add an upsert table sink factory for Elasticsearch

Posted by "asfgit (GitHub)" <gi...@apache.org>.
[ pull request closed by asfgit ]

[ Full content available at: https://github.com/apache/flink/pull/6611 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org

[GitHub] [flink] twalthr commented on pull request #6611: [FLINK-3875] [connectors] Add an upsert table sink factory for Elasticsearch

Posted by "twalthr (GitHub)" <gi...@apache.org>.
I think we can also remove it. As the comment suggests it doesn't check much. Update modes are optional for custom factories anyway.

[ Full content available at: https://github.com/apache/flink/pull/6611 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org

[GitHub] [flink] twalthr commented on pull request #6611: [FLINK-3875] [connectors] Add an upsert table sink factory for Elasticsearch

Posted by "twalthr (GitHub)" <gi...@apache.org>.
You never know where a service is located. I think exposing this property is useful that's why Elasticsearch is also exposing it. 

[ Full content available at: https://github.com/apache/flink/pull/6611 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org

[GitHub] [flink] dawidwys commented on pull request #6611: [FLINK-3875] [connectors] Add an upsert table sink factory for Elasticsearch

Posted by "dawidwys (GitHub)" <gi...@apache.org>.
Same as above.

[ Full content available at: https://github.com/apache/flink/pull/6611 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org

[GitHub] [flink] twalthr commented on issue #6611: [FLINK-3875] [connectors] Add an upsert table sink factory for Elasticsearch

Posted by "twalthr (GitHub)" <gi...@apache.org>.
CC @tzulitai @dawidwys 

[ Full content available at: https://github.com/apache/flink/pull/6611 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org

[GitHub] [flink] dawidwys commented on pull request #6611: [FLINK-3875] [connectors] Add an upsert table sink factory for Elasticsearch

Posted by "dawidwys (GitHub)" <gi...@apache.org>.
Could we use `MemorySize` for this? It is quite cumbersome to switch to it later. Especially if previously the value was specified in Mb.

[ Full content available at: https://github.com/apache/flink/pull/6611 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org

[GitHub] [flink] dawidwys commented on pull request #6611: [FLINK-3875] [connectors] Add an upsert table sink factory for Elasticsearch

Posted by "dawidwys (GitHub)" <gi...@apache.org>.
Personally would prefer java 8's stream version of this function:

	private String createKey(Row row) {
		return Arrays.stream(keyFieldIndices).mapToObj(i -> {
			final Object value = row.getField(i);
			if (value == null) {
				return keyNullLiteral;
			} else {
				return value.toString();
			}
		}).collect(Collectors.joining("keyDelimiter"));
	}

It is more readable at least for me.

[ Full content available at: https://github.com/apache/flink/pull/6611 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org

[GitHub] [flink] dawidwys commented on pull request #6611: [FLINK-3875] [connectors] Add an upsert table sink factory for Elasticsearch

Posted by "dawidwys (GitHub)" <gi...@apache.org>.
Does it have any other goal than setting rest api version? Wouldn't it make more sense to choose from api version?

[ Full content available at: https://github.com/apache/flink/pull/6611 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org

[GitHub] [flink] dawidwys commented on pull request #6611: [FLINK-3875] [connectors] Add an upsert table sink factory for Elasticsearch

Posted by "dawidwys (GitHub)" <gi...@apache.org>.
Does it make sense to perform this validation here? This validation most probably has to be always repeated in the `*TableFactory`. Or am I missing something?

[ Full content available at: https://github.com/apache/flink/pull/6611 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org

[GitHub] [flink] dawidwys commented on pull request #6611: [FLINK-3875] [connectors] Add an upsert table sink factory for Elasticsearch

Posted by "dawidwys (GitHub)" <gi...@apache.org>.
It is no longer `(in MB)`, by default if no unit is provided it will be treated as bytes. I would also add info about the minimal restriction here. 

[ Full content available at: https://github.com/apache/flink/pull/6611 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org

[GitHub] [flink] twalthr commented on issue #6611: [FLINK-3875] [connectors] Add an upsert table sink factory for Elasticsearch

Posted by "twalthr (GitHub)" <gi...@apache.org>.
Thank you @dawidwys. Travis gave green light. I will merge this...

[ Full content available at: https://github.com/apache/flink/pull/6611 ]
This message was relayed via gitbox.apache.org for devnull@infra.apache.org