You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by jainshasha <ja...@gmail.com> on 2020/09/07 07:20:02 UTC
Elastic Search sink showing -1 for numOutputRows
Hi,
Using structured spark streaming and sink the data into ElasticSearch.
In the stats emit for each batch the "numOutputRows" showing -1 for
ElasticSearch sink always
whereas when i see other sinks like Kafka it shows either 0 or some values
when it emit data.
What could be the reason for showing -1 for ElasticSearch ?
--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
Re: Elastic Search sink showing -1 for numOutputRows
Posted by jainshasha <ja...@gmail.com>.
Thanks Jungtaek Lim-2 for replying.
May i knw the reference of the API version for sink for both types (DSv1 and
DSv2) in code ?
Where could i see it ? Under what module of spark code ?
--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
Re: Elastic Search sink showing -1 for numOutputRows
Posted by Jungtaek Lim <ka...@gmail.com>.
I don't know about ES sink. The availability of "numOutputRows" depends on
the API version the sink is implementing (DSv1 vs DSv2), so you may be
better to ask a question to the author of ES sink and confirm the case.
On Tue, Sep 8, 2020 at 5:15 AM jainshasha <ja...@gmail.com> wrote:
> Hi,
>
> Using structured spark streaming and sink the data into ElasticSearch.
> In the stats emit for each batch the "numOutputRows" showing -1 for
> ElasticSearch sink always
> whereas when i see other sinks like Kafka it shows either 0 or some values
> when it emit data.
>
> What could be the reason for showing -1 for ElasticSearch ?
>
>
>
> --
> Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>
Re: Elastic Search sink showing -1 for numOutputRows
Posted by jainshasha <ja...@gmail.com>.
Hi,
Using structured spark streaming and sink the data into ElasticSearch.
In the stats emit for each batch the "numOutputRows" showing -1 for
ElasticSearch sink always
whereas when i see other sinks like Kafka it shows either 0 or some values
when it emit data.
What could be the reason for showing -1 for ElasticSearch ?
--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org