Status of Elastic Search integration tests

[Karl Wright <da...@gmail.com>]

Hello all,

I set up a branch (branches/CONNECTORS-1639) to work on the elasticsearch
test problem for JDK 11.  The branch downloads an ES and a mapper
attachment but it turns out that the mapper attachment is apparently
incompatible with the current (7.x) version of ES.  Does anyone know
whether the mapper attachment is still supported?  If so, where can I find
it in the Maven repo?

Karl

Re: Status of Elastic Search integration tests

[Karl Wright <da...@gmail.com>]

Can you suggest what components I need to download and configure for the
test, then?  Step-by-step directions would be very helpful, e.g. "download
es version x, and this component from this maven repo URL, and then unpack
here, and create this config file..."

The mapper attachment I download from the maven repo seems to not work with
the ES version I am downloading, which is why I'm stuck.  The new plugin
you pointed me at says:

Installationedit
<https://github.com/elastic/elasticsearch/edit/master/docs/plugins/install_remove.asciidoc>

Version 8.0.0 of the Elastic Stack has not yet been released.

On Wed, Apr 29, 2020 at 7:49 AM Michael Cizmar <mi...@michaelcizmar.com>
wrote:

> Right.  On my radar is refactoring of this to use the Elastic Java SDK.  If
> we use that then, in my view, the encoding of the document would be the
> responsibility of the SDK and one less thing to test.  (The Java SDK is
> somewhat complicated as well because they tend to rewrite underlying
> transmission pieces)
>
> For testing purposes currently we can install the mapper attachment and
> create an ingestion workflow to hand that.
>
> On Wed, Apr 29, 2020 at 6:44 AM Karl Wright <da...@gmail.com> wrote:
>
> > The connector itself encodes binary documents and sends them to ES,
> > purportedly for the mapper attachment to process and convert to text.
> The
> > test used to exercise that.
> >
> > Perhaps it's worth reviewing the connector code itself to see what is
> > outdated/legacy, and only test the parts that are not outdated?
> >
> > Specifically, my concern is that we need to support binary document
> > transmission to ES, and ES obviously needs to handle those for the
> > integration to work properly.
> >
> > Karl
> >
> >
> > On Wed, Apr 29, 2020 at 7:27 AM Michael Cizmar <
> michael@michaelcizmar.com>
> > wrote:
> >
> > > There's been some changes to Elasticsearch like reducing the document
> > types
> > > and ingestion/mapper.  The mapper attachment I believe has been
> > > deprecated in favor of:
> > >
> > >
> > >
> >
> https://www.elastic.co/guide/en/elasticsearch/plugins/master/ingest-attachment.html
> > >
> > > This should be incorporated into a pipeline.  Do we need something like
> > > this in our integration test?  I don't think it's the responsibility of
> > the
> > > output connector to handle this.
> > >
> > > On Wed, Apr 29, 2020 at 5:25 AM Karl Wright <da...@gmail.com>
> wrote:
> > >
> > > > Hello all,
> > > >
> > > > I set up a branch (branches/CONNECTORS-1639) to work on the
> > elasticsearch
> > > > test problem for JDK 11.  The branch downloads an ES and a mapper
> > > > attachment but it turns out that the mapper attachment is apparently
> > > > incompatible with the current (7.x) version of ES.  Does anyone know
> > > > whether the mapper attachment is still supported?  If so, where can I
> > > find
> > > > it in the Maven repo?
> > > >
> > > > Karl
> > > >
> > >
> >
>

Re: Status of Elastic Search integration tests

[Michael Cizmar <mi...@michaelcizmar.com>]

Right.  On my radar is refactoring of this to use the Elastic Java SDK.  If
we use that then, in my view, the encoding of the document would be the
responsibility of the SDK and one less thing to test.  (The Java SDK is
somewhat complicated as well because they tend to rewrite underlying
transmission pieces)

For testing purposes currently we can install the mapper attachment and
create an ingestion workflow to hand that.

On Wed, Apr 29, 2020 at 6:44 AM Karl Wright <da...@gmail.com> wrote:

> The connector itself encodes binary documents and sends them to ES,
> purportedly for the mapper attachment to process and convert to text.  The
> test used to exercise that.
>
> Perhaps it's worth reviewing the connector code itself to see what is
> outdated/legacy, and only test the parts that are not outdated?
>
> Specifically, my concern is that we need to support binary document
> transmission to ES, and ES obviously needs to handle those for the
> integration to work properly.
>
> Karl
>
>
> On Wed, Apr 29, 2020 at 7:27 AM Michael Cizmar <mi...@michaelcizmar.com>
> wrote:
>
> > There's been some changes to Elasticsearch like reducing the document
> types
> > and ingestion/mapper.  The mapper attachment I believe has been
> > deprecated in favor of:
> >
> >
> >
> https://www.elastic.co/guide/en/elasticsearch/plugins/master/ingest-attachment.html
> >
> > This should be incorporated into a pipeline.  Do we need something like
> > this in our integration test?  I don't think it's the responsibility of
> the
> > output connector to handle this.
> >
> > On Wed, Apr 29, 2020 at 5:25 AM Karl Wright <da...@gmail.com> wrote:
> >
> > > Hello all,
> > >
> > > I set up a branch (branches/CONNECTORS-1639) to work on the
> elasticsearch
> > > test problem for JDK 11.  The branch downloads an ES and a mapper
> > > attachment but it turns out that the mapper attachment is apparently
> > > incompatible with the current (7.x) version of ES.  Does anyone know
> > > whether the mapper attachment is still supported?  If so, where can I
> > find
> > > it in the Maven repo?
> > >
> > > Karl
> > >
> >
>

Re: Status of Elastic Search integration tests

[Karl Wright <da...@gmail.com>]

The connector itself encodes binary documents and sends them to ES,
purportedly for the mapper attachment to process and convert to text.  The
test used to exercise that.

Perhaps it's worth reviewing the connector code itself to see what is
outdated/legacy, and only test the parts that are not outdated?

Specifically, my concern is that we need to support binary document
transmission to ES, and ES obviously needs to handle those for the
integration to work properly.

Karl


On Wed, Apr 29, 2020 at 7:27 AM Michael Cizmar <mi...@michaelcizmar.com>
wrote:

> There's been some changes to Elasticsearch like reducing the document types
> and ingestion/mapper.  The mapper attachment I believe has been
> deprecated in favor of:
>
>
> https://www.elastic.co/guide/en/elasticsearch/plugins/master/ingest-attachment.html
>
> This should be incorporated into a pipeline.  Do we need something like
> this in our integration test?  I don't think it's the responsibility of the
> output connector to handle this.
>
> On Wed, Apr 29, 2020 at 5:25 AM Karl Wright <da...@gmail.com> wrote:
>
> > Hello all,
> >
> > I set up a branch (branches/CONNECTORS-1639) to work on the elasticsearch
> > test problem for JDK 11.  The branch downloads an ES and a mapper
> > attachment but it turns out that the mapper attachment is apparently
> > incompatible with the current (7.x) version of ES.  Does anyone know
> > whether the mapper attachment is still supported?  If so, where can I
> find
> > it in the Maven repo?
> >
> > Karl
> >
>

Re: Status of Elastic Search integration tests

[Michael Cizmar <mi...@michaelcizmar.com>]

There's been some changes to Elasticsearch like reducing the document types
and ingestion/mapper.  The mapper attachment I believe has been
deprecated in favor of:

https://www.elastic.co/guide/en/elasticsearch/plugins/master/ingest-attachment.html

This should be incorporated into a pipeline.  Do we need something like
this in our integration test?  I don't think it's the responsibility of the
output connector to handle this.

On Wed, Apr 29, 2020 at 5:25 AM Karl Wright <da...@gmail.com> wrote:

> Hello all,
>
> I set up a branch (branches/CONNECTORS-1639) to work on the elasticsearch
> test problem for JDK 11.  The branch downloads an ES and a mapper
> attachment but it turns out that the mapper attachment is apparently
> incompatible with the current (7.x) version of ES.  Does anyone know
> whether the mapper attachment is still supported?  If so, where can I find
> it in the Maven repo?
>
> Karl
>