Re: Beam CassandraIO

[Alexey Romanenko <ar...@gmail.com>]

- dev@beam.apache.org
+ user@beam.apache.org 

Hi Enzo,

Can you make sure that all your workers were properly added and listed in Spark WebUI?

Did you specify a “ --master spark://HOST:PORT” option while running your Beam job with a SparkRunner?

PS: Please, use user@beam.apache.org mailing list for such type of questions.

—
Alexey

> On 2 Feb 2023, at 03:18, Enzo Bonggio <en...@medialab.la> wrote:
> 
> I have a spark standalone installed in two machines but once I send spark-submit, it will only execute in one executer. Is that the way that it suppose to work? 
> I thought that I could read from Cassandra with multiple machines 

Re: Beam CassandraIO

[Vincent Marquez <vi...@gmail.com>]

*~Vincent*


On Thu, Feb 2, 2023 at 3:01 AM Alexey Romanenko <ar...@gmail.com>
wrote:

> - dev@beam.apache.org
> + user@beam.apache.org
>
> Hi Enzo,
>
> Can you make sure that all your workers were properly added and listed in
> Spark WebUI?
>
> Did you specify a “ --master spark://HOST:PORT” option while running your
> Beam job with a SparkRunner?
>
> PS: Please, use user@beam.apache.org mailing list for such type of
> questions.
>
> —
> Alexey
>
> > On 2 Feb 2023, at 03:18, Enzo Bonggio <en...@medialab.la> wrote:
> >
> > I have a spark standalone installed in two machines but once I send
> spark-submit, it will only execute in one executer. Is that the way that it
> suppose to work?
> > I thought that I could read from Cassandra with multiple machines
>
>
Hi Enzo, I've used the CassandraIO connector quite a bit on Dataflow and
found it scaled out well to multiple machines.  I've not used it on Spark,
so I can't say if this is a spark specific issue.

Could you provide more details on how your pipeline works, and how you can
tell it's only executing on 'one executor'?