You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Flavio Pompermaier <po...@okkam.it> on 2016/07/18 09:23:17 UTC
Elasticsearch connector and number of shards
Hi to all,
we tried to use the streaming ES connector of Flink 1.1-SNAPSHOT and we
wanted to set the number of shards when creating a new index. Is that
possible at the moment?
Best,
Flavio
Re: Elasticsearch connector and number of shards
Posted by Flavio Pompermaier <po...@okkam.it>.
I've just added a JIRA improvement ticket for this (
https://issues.apache.org/jira/browse/FLINK-4491).
Best,
Flavio
On Wed, Jul 20, 2016 at 4:21 PM, Maximilian Michels <mx...@apache.org> wrote:
> The connector doesn't cover this use case. Through the API you need to
> use the IndicesAdminClient:
> https://www.elastic.co/guide/en/elasticsearch/client/java-
> api/current/java-admin-indices.html
>
> Otherwise Elasticsearch creates an index with shards automatically. We
> could add support for configuring shards in the future.
>
> On Mon, Jul 18, 2016 at 11:42 AM, Flavio Pompermaier
> <po...@okkam.it> wrote:
> > Indeed, we've tried with the parameter index.number_of_shards but it
> didn't
> > work so I fear that this parameter is not handled by the current
> > implementation..am I wrong?
> >
> > On Mon, Jul 18, 2016 at 11:37 AM, Ufuk Celebi <uc...@apache.org> wrote:
> >>
> >> I've never used the Elasticsearch sink, but the docs say:
> >>
> >> "Note how a Map of Strings is used to configure the Sink. The
> >> configuration keys are documented in the Elasticsearch
> >> documentationhere. Especially important is the cluster.name parameter
> >> that must correspond to the name of your cluster."
> >>
> >> The config keys for index creation are found here:
> >>
> >> https://www.elastic.co/guide/en/elasticsearch/reference/
> current/indices-create-index.html
> >>
> >> Does this help? If not, maybe @Aljoscha can chime in here.
> >>
> >> – Ufuk
> >>
> >>
> >> On Mon, Jul 18, 2016 at 11:23 AM, Flavio Pompermaier
> >> <po...@okkam.it> wrote:
> >> > Hi to all,
> >> >
> >> > we tried to use the streaming ES connector of Flink 1.1-SNAPSHOT and
> we
> >> > wanted to set the number of shards when creating a new index. Is that
> >> > possible at the moment?
> >> >
> >> > Best,
> >> > Flavio
> >
> >
>
Re: Elasticsearch connector and number of shards
Posted by Maximilian Michels <mx...@apache.org>.
The connector doesn't cover this use case. Through the API you need to
use the IndicesAdminClient:
https://www.elastic.co/guide/en/elasticsearch/client/java-api/current/java-admin-indices.html
Otherwise Elasticsearch creates an index with shards automatically. We
could add support for configuring shards in the future.
On Mon, Jul 18, 2016 at 11:42 AM, Flavio Pompermaier
<po...@okkam.it> wrote:
> Indeed, we've tried with the parameter index.number_of_shards but it didn't
> work so I fear that this parameter is not handled by the current
> implementation..am I wrong?
>
> On Mon, Jul 18, 2016 at 11:37 AM, Ufuk Celebi <uc...@apache.org> wrote:
>>
>> I've never used the Elasticsearch sink, but the docs say:
>>
>> "Note how a Map of Strings is used to configure the Sink. The
>> configuration keys are documented in the Elasticsearch
>> documentationhere. Especially important is the cluster.name parameter
>> that must correspond to the name of your cluster."
>>
>> The config keys for index creation are found here:
>>
>> https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html
>>
>> Does this help? If not, maybe @Aljoscha can chime in here.
>>
>> – Ufuk
>>
>>
>> On Mon, Jul 18, 2016 at 11:23 AM, Flavio Pompermaier
>> <po...@okkam.it> wrote:
>> > Hi to all,
>> >
>> > we tried to use the streaming ES connector of Flink 1.1-SNAPSHOT and we
>> > wanted to set the number of shards when creating a new index. Is that
>> > possible at the moment?
>> >
>> > Best,
>> > Flavio
>
>
Re: Elasticsearch connector and number of shards
Posted by Flavio Pompermaier <po...@okkam.it>.
Indeed, we've tried with the parameter *index.number_of_shards* but it
didn't work so I fear that this parameter is not handled by the current
implementation..am I wrong?
On Mon, Jul 18, 2016 at 11:37 AM, Ufuk Celebi <uc...@apache.org> wrote:
> I've never used the Elasticsearch sink, but the docs say:
>
> "Note how a Map of Strings is used to configure the Sink. The
> configuration keys are documented in the Elasticsearch
> documentationhere. Especially important is the cluster.name parameter
> that must correspond to the name of your cluster."
>
> The config keys for index creation are found here:
>
> https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html
>
> Does this help? If not, maybe @Aljoscha can chime in here.
>
> – Ufuk
>
>
> On Mon, Jul 18, 2016 at 11:23 AM, Flavio Pompermaier
> <po...@okkam.it> wrote:
> > Hi to all,
> >
> > we tried to use the streaming ES connector of Flink 1.1-SNAPSHOT and we
> > wanted to set the number of shards when creating a new index. Is that
> > possible at the moment?
> >
> > Best,
> > Flavio
>
Re: Elasticsearch connector and number of shards
Posted by Ufuk Celebi <uc...@apache.org>.
I've never used the Elasticsearch sink, but the docs say:
"Note how a Map of Strings is used to configure the Sink. The
configuration keys are documented in the Elasticsearch
documentationhere. Especially important is the cluster.name parameter
that must correspond to the name of your cluster."
The config keys for index creation are found here:
https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html
Does this help? If not, maybe @Aljoscha can chime in here.
– Ufuk
On Mon, Jul 18, 2016 at 11:23 AM, Flavio Pompermaier
<po...@okkam.it> wrote:
> Hi to all,
>
> we tried to use the streaming ES connector of Flink 1.1-SNAPSHOT and we
> wanted to set the number of shards when creating a new index. Is that
> possible at the moment?
>
> Best,
> Flavio