You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@beam.apache.org by Tom Barber <to...@spicule.co.uk> on 2019/09/16 23:30:54 UTC

Running Python Wordcount issues

Hello folks,

Trying to get started running the python word count example against the
portable runner using the beam 2.15 download.

In on terminal I have:

gradle :runners:spark:job-server:runShadow

And another:

gradle portableWordCount -PjobEndpoint=localhost:8099
-PenvironmentType=LOOPBACK

But when it starts it says:

Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File
"/home/ubuntu/beam-release-2.15.0/sdks/python/apache_beam/examples/wordcount.py",
line 135, in <module>
run()
File
"/home/ubuntu/beam-release-2.15.0/sdks/python/apache_beam/examples/wordcount.py",
line 115, in run
result.wait_until_finish()
File
"/home/ubuntu/beam-release-2.15.0/sdks/python/apache_beam/runners/portability/portable_runner.py",
line 446, in wait_until_finish
self._job_id, self._state, self._last_error_message()))
RuntimeError: Pipeline
BeamApp-ubuntu-0916232552-dd0af6d7_90675fc0-c39c-4ca7-93fc-c7c7e980f546
failed in state FAILED: java.lang.ClassCastException:
org.apache.beam.sdk.coders.LengthPrefixCoder cannot be cast to
org.apache.beam.sdk.coders.KvCoder

And I have no clue where to start looking to fix that as I’d assume the
demos should just run?

Tom

Spicule Limited is registered in England & Wales. Company Number:
09954122. Registered office: First Floor, Telecom House, 125-135 Preston
Road, Brighton, England, BN1 6AF. VAT No. 251478891.

All engagements
are subject to Spicule Terms and Conditions of Business. This email and its
contents are intended solely for the individual to whom it is addressed and
may contain information that is confidential, privileged or otherwise
protected from disclosure, distributing or copying. Any views or opinions
presented in this email are solely those of the author and do not
necessarily represent those of Spicule Limited. The company accepts no
liability for any damage caused by any virus transmitted by this email. If
you have received this message in error, please notify us immediately by
reply email before deleting it from your system. Service of legal notice
cannot be effected on Spicule Limited by email.

Re: Running Python Wordcount issues

Posted by Tom Barber <to...@spicule.co.uk>.

Alright well I’ve got 2.16 checked out and building so I’ll test it and
feedback.

Thanks!

On 17 September 2019 at 00:43:57, Benjamin Tan (benjamintanweihao@gmail.com)
wrote:

I can confirm. I used the master branch to get this to work. I tried the
"release-2.16.0" branch but that didn't work either.

On 2019/09/16 23:30:54, Tom Barber <to...@spicule.co.uk> wrote:
> Hello folks,
>
> Trying to get started running the python word count example against the
> portable runner using the beam 2.15 download.
>
> In on terminal I have:
>
> gradle :runners:spark:job-server:runShadow
>
> And another:
>
> gradle portableWordCount -PjobEndpoint=localhost:8099
> -PenvironmentType=LOOPBACK
>
>
> But when it starts it says:
>
> Traceback (most recent call last):
> File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
> "__main__", fname, loader, pkg_name)
> File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
> exec code in run_globals
> File
>
"/home/ubuntu/beam-release-2.15.0/sdks/python/apache_beam/examples/wordcount.py",

> line 135, in <module>
> run()
> File
>
"/home/ubuntu/beam-release-2.15.0/sdks/python/apache_beam/examples/wordcount.py",

> line 115, in run
> result.wait_until_finish()
> File
>
"/home/ubuntu/beam-release-2.15.0/sdks/python/apache_beam/runners/portability/portable_runner.py",

> line 446, in wait_until_finish
> self._job_id, self._state, self._last_error_message()))
> RuntimeError: Pipeline
> BeamApp-ubuntu-0916232552-dd0af6d7_90675fc0-c39c-4ca7-93fc-c7c7e980f546
> failed in state FAILED: java.lang.ClassCastException:
> org.apache.beam.sdk.coders.LengthPrefixCoder cannot be cast to
> org.apache.beam.sdk.coders.KvCoder
>
> And I have no clue where to start looking to fix that as I’d assume the
> demos should just run?
>
> Tom
>
> --
>
>
> Spicule Limited is registered in England & Wales. Company Number:
> 09954122. Registered office: First Floor, Telecom House, 125-135 Preston
> Road, Brighton, England, BN1 6AF. VAT No. 251478891.
>
>
>
>
> All engagements
> are subject to Spicule Terms and Conditions of Business. This email and
its
> contents are intended solely for the individual to whom it is addressed
and
> may contain information that is confidential, privileged or otherwise
> protected from disclosure, distributing or copying. Any views or opinions
> presented in this email are solely those of the author and do not
> necessarily represent those of Spicule Limited. The company accepts no
> liability for any damage caused by any virus transmitted by this email.
If
> you have received this message in error, please notify us immediately by
> reply email before deleting it from your system. Service of legal notice
> cannot be effected on Spicule Limited by email.
>

-- 

Spicule Limited is registered in England & Wales. Company Number: 
09954122. Registered office: First Floor, Telecom House, 125-135 Preston 
Road, Brighton, England, BN1 6AF. VAT No. 251478891.

All engagements 
are subject to Spicule Terms and Conditions of Business. This email and its 
contents are intended solely for the individual to whom it is addressed and 
may contain information that is confidential, privileged or otherwise 
protected from disclosure, distributing or copying. Any views or opinions 
presented in this email are solely those of the author and do not 
necessarily represent those of Spicule Limited. The company accepts no 
liability for any damage caused by any virus transmitted by this email. If 
you have received this message in error, please notify us immediately by 
reply email before deleting it from your system. Service of legal notice 
cannot be effected on Spicule Limited by email.

Re: Running Python Wordcount issues

Posted by Tom Barber <to...@spicule.co.uk>.

That error went away for me on 2.16 from Git.

Tom


On 17 September 2019 at 00:47:27, Robert Bradshaw (robertwb@google.com)
wrote:

What errors did you get with the release-2.16.0 branch?

On Mon, Sep 16, 2019 at 4:43 PM Benjamin Tan
<be...@gmail.com> wrote:
>
> I can confirm. I used the master branch to get this to work. I tried the
"release-2.16.0" branch but that didn't work either.
>
> On 2019/09/16 23:30:54, Tom Barber <to...@spicule.co.uk> wrote:
> > Hello folks,
> >
> > Trying to get started running the python word count example against the
> > portable runner using the beam 2.15 download.
> >
> > In on terminal I have:
> >
> > gradle :runners:spark:job-server:runShadow
> >
> > And another:
> >
> > gradle portableWordCount -PjobEndpoint=localhost:8099
> > -PenvironmentType=LOOPBACK
> >
> >
> > But when it starts it says:
> >
> > Traceback (most recent call last):
> > File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
> > "__main__", fname, loader, pkg_name)
> > File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
> > exec code in run_globals
> > File
> >
"/home/ubuntu/beam-release-2.15.0/sdks/python/apache_beam/examples/wordcount.py",

> > line 135, in <module>
> > run()
> > File
> >
"/home/ubuntu/beam-release-2.15.0/sdks/python/apache_beam/examples/wordcount.py",

> > line 115, in run
> > result.wait_until_finish()
> > File
> >
"/home/ubuntu/beam-release-2.15.0/sdks/python/apache_beam/runners/portability/portable_runner.py",

> > line 446, in wait_until_finish
> > self._job_id, self._state, self._last_error_message()))
> > RuntimeError: Pipeline
> > BeamApp-ubuntu-0916232552-dd0af6d7_90675fc0-c39c-4ca7-93fc-c7c7e980f546
> > failed in state FAILED: java.lang.ClassCastException:
> > org.apache.beam.sdk.coders.LengthPrefixCoder cannot be cast to
> > org.apache.beam.sdk.coders.KvCoder
> >
> > And I have no clue where to start looking to fix that as I’d assume the
> > demos should just run?
> >
> > Tom
> >
> > --
> >
> >
> > Spicule Limited is registered in England & Wales. Company Number:
> > 09954122. Registered office: First Floor, Telecom House, 125-135
Preston
> > Road, Brighton, England, BN1 6AF. VAT No. 251478891.
> >
> >
> >
> >
> > All engagements
> > are subject to Spicule Terms and Conditions of Business. This email and
its
> > contents are intended solely for the individual to whom it is addressed
and
> > may contain information that is confidential, privileged or otherwise
> > protected from disclosure, distributing or copying. Any views or
opinions
> > presented in this email are solely those of the author and do not
> > necessarily represent those of Spicule Limited. The company accepts no
> > liability for any damage caused by any virus transmitted by this email.
If
> > you have received this message in error, please notify us immediately
by
> > reply email before deleting it from your system. Service of legal
notice
> > cannot be effected on Spicule Limited by email.
> >

-- 


Spicule Limited is registered in England & Wales. Company Number: 
09954122. Registered office: First Floor, Telecom House, 125-135 Preston 
Road, Brighton, England, BN1 6AF. VAT No. 251478891.




All engagements 
are subject to Spicule Terms and Conditions of Business. This email and its 
contents are intended solely for the individual to whom it is addressed and 
may contain information that is confidential, privileged or otherwise 
protected from disclosure, distributing or copying. Any views or opinions 
presented in this email are solely those of the author and do not 
necessarily represent those of Spicule Limited. The company accepts no 
liability for any damage caused by any virus transmitted by this email. If 
you have received this message in error, please notify us immediately by 
reply email before deleting it from your system. Service of legal notice 
cannot be effected on Spicule Limited by email.

Re: Running Python Wordcount issues

Posted by Robert Bradshaw <ro...@google.com>.

What errors did you get with the release-2.16.0 branch?

On Mon, Sep 16, 2019 at 4:43 PM Benjamin Tan
<be...@gmail.com> wrote:
>
> I can confirm. I used the master branch to get this to work. I tried the "release-2.16.0" branch but that didn't work either.
>
> On 2019/09/16 23:30:54, Tom Barber <to...@spicule.co.uk> wrote:
> > Hello folks,
> >
> > Trying to get started running the python word count example against the
> > portable runner using the beam 2.15 download.
> >
> > In on terminal I have:
> >
> > gradle :runners:spark:job-server:runShadow
> >
> > And another:
> >
> > gradle portableWordCount -PjobEndpoint=localhost:8099
> > -PenvironmentType=LOOPBACK
> >
> >
> > But when it starts it says:
> >
> > Traceback (most recent call last):
> >   File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
> >     "__main__", fname, loader, pkg_name)
> >   File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
> >     exec code in run_globals
> >   File
> > "/home/ubuntu/beam-release-2.15.0/sdks/python/apache_beam/examples/wordcount.py",
> > line 135, in <module>
> >     run()
> >   File
> > "/home/ubuntu/beam-release-2.15.0/sdks/python/apache_beam/examples/wordcount.py",
> > line 115, in run
> >     result.wait_until_finish()
> >   File
> > "/home/ubuntu/beam-release-2.15.0/sdks/python/apache_beam/runners/portability/portable_runner.py",
> > line 446, in wait_until_finish
> >     self._job_id, self._state, self._last_error_message()))
> > RuntimeError: Pipeline
> > BeamApp-ubuntu-0916232552-dd0af6d7_90675fc0-c39c-4ca7-93fc-c7c7e980f546
> > failed in state FAILED: java.lang.ClassCastException:
> > org.apache.beam.sdk.coders.LengthPrefixCoder cannot be cast to
> > org.apache.beam.sdk.coders.KvCoder
> >
> > And I have no clue where to start looking to fix that as I’d assume the
> > demos should just run?
> >
> > Tom
> >
> > --
> >
> >
> > Spicule Limited is registered in England & Wales. Company Number:
> > 09954122. Registered office: First Floor, Telecom House, 125-135 Preston
> > Road, Brighton, England, BN1 6AF. VAT No. 251478891.
> >
> >
> >
> >
> > All engagements
> > are subject to Spicule Terms and Conditions of Business. This email and its
> > contents are intended solely for the individual to whom it is addressed and
> > may contain information that is confidential, privileged or otherwise
> > protected from disclosure, distributing or copying. Any views or opinions
> > presented in this email are solely those of the author and do not
> > necessarily represent those of Spicule Limited. The company accepts no
> > liability for any damage caused by any virus transmitted by this email. If
> > you have received this message in error, please notify us immediately by
> > reply email before deleting it from your system. Service of legal notice
> > cannot be effected on Spicule Limited by email.
> >

Re: Running Python Wordcount issues

Posted by Benjamin Tan <be...@gmail.com>.

I can confirm. I used the master branch to get this to work. I tried the "release-2.16.0" branch but that didn't work either.

On 2019/09/16 23:30:54, Tom Barber <to...@spicule.co.uk> wrote: 
> Hello folks,
> 
> Trying to get started running the python word count example against the
> portable runner using the beam 2.15 download.
> 
> In on terminal I have:
> 
> gradle :runners:spark:job-server:runShadow
> 
> And another:
> 
> gradle portableWordCount -PjobEndpoint=localhost:8099
> -PenvironmentType=LOOPBACK
> 
> 
> But when it starts it says:
> 
> Traceback (most recent call last):
>   File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
>     "__main__", fname, loader, pkg_name)
>   File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
>     exec code in run_globals
>   File
> "/home/ubuntu/beam-release-2.15.0/sdks/python/apache_beam/examples/wordcount.py",
> line 135, in <module>
>     run()
>   File
> "/home/ubuntu/beam-release-2.15.0/sdks/python/apache_beam/examples/wordcount.py",
> line 115, in run
>     result.wait_until_finish()
>   File
> "/home/ubuntu/beam-release-2.15.0/sdks/python/apache_beam/runners/portability/portable_runner.py",
> line 446, in wait_until_finish
>     self._job_id, self._state, self._last_error_message()))
> RuntimeError: Pipeline
> BeamApp-ubuntu-0916232552-dd0af6d7_90675fc0-c39c-4ca7-93fc-c7c7e980f546
> failed in state FAILED: java.lang.ClassCastException:
> org.apache.beam.sdk.coders.LengthPrefixCoder cannot be cast to
> org.apache.beam.sdk.coders.KvCoder
> 
> And I have no clue where to start looking to fix that as I’d assume the
> demos should just run?
> 
> Tom
> 
> -- 
> 
> 
> Spicule Limited is registered in England & Wales. Company Number: 
> 09954122. Registered office: First Floor, Telecom House, 125-135 Preston 
> Road, Brighton, England, BN1 6AF. VAT No. 251478891.
> 
> 
> 
> 
> All engagements 
> are subject to Spicule Terms and Conditions of Business. This email and its 
> contents are intended solely for the individual to whom it is addressed and 
> may contain information that is confidential, privileged or otherwise 
> protected from disclosure, distributing or copying. Any views or opinions 
> presented in this email are solely those of the author and do not 
> necessarily represent those of Spicule Limited. The company accepts no 
> liability for any damage caused by any virus transmitted by this email. If 
> you have received this message in error, please notify us immediately by 
> reply email before deleting it from your system. Service of legal notice 
> cannot be effected on Spicule Limited by email.
>

Re: Running Python Wordcount issues

Posted by Kyle Weaver <kc...@google.com>.

This is a known issue with Beam 2.15. Using master or the 2.16 branch
should fix the problem.

Kyle Weaver | Software Engineer | github.com/ibzib | kcweaver@google.com


On Mon, Sep 16, 2019 at 4:31 PM Tom Barber <to...@spicule.co.uk> wrote:

> Hello folks,
>
> Trying to get started running the python word count example against the
> portable runner using the beam 2.15 download.
>
> In on terminal I have:
>
> gradle :runners:spark:job-server:runShadow
>
> And another:
>
> gradle portableWordCount -PjobEndpoint=localhost:8099
> -PenvironmentType=LOOPBACK
>
>
> But when it starts it says:
>
> Traceback (most recent call last):
>   File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
>     "__main__", fname, loader, pkg_name)
>   File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
>     exec code in run_globals
>   File
> "/home/ubuntu/beam-release-2.15.0/sdks/python/apache_beam/examples/wordcount.py",
> line 135, in <module>
>     run()
>   File
> "/home/ubuntu/beam-release-2.15.0/sdks/python/apache_beam/examples/wordcount.py",
> line 115, in run
>     result.wait_until_finish()
>   File
> "/home/ubuntu/beam-release-2.15.0/sdks/python/apache_beam/runners/portability/portable_runner.py",
> line 446, in wait_until_finish
>     self._job_id, self._state, self._last_error_message()))
> RuntimeError: Pipeline
> BeamApp-ubuntu-0916232552-dd0af6d7_90675fc0-c39c-4ca7-93fc-c7c7e980f546
> failed in state FAILED: java.lang.ClassCastException:
> org.apache.beam.sdk.coders.LengthPrefixCoder cannot be cast to
> org.apache.beam.sdk.coders.KvCoder
>
> And I have no clue where to start looking to fix that as I’d assume the
> demos should just run?
>
> Tom
>
> Spicule Limited is registered in England & Wales. Company Number:
> 09954122. Registered office: First Floor, Telecom House, 125-135 Preston
> Road, Brighton, England, BN1 6AF. VAT No. 251478891.
>
>
> All engagements are subject to Spicule Terms and Conditions of Business.
> This email and its contents are intended solely for the individual to whom
> it is addressed and may contain information that is confidential,
> privileged or otherwise protected from disclosure, distributing or copying.
> Any views or opinions presented in this email are solely those of the
> author and do not necessarily represent those of Spicule Limited. The
> company accepts no liability for any damage caused by any virus transmitted
> by this email. If you have received this message in error, please notify us
> immediately by reply email before deleting it from your system. Service of
> legal notice cannot be effected on Spicule Limited by email.
>

Re: Running Python Wordcount issues

Posted by Tom Barber <to...@spicule.co.uk>.

Ah I found https://issues.apache.org/jira/browse/BEAM-7864 which I guess is
related. I’ll try a Git based version.

On 17 September 2019 at 00:30:54, Tom Barber (tom@spicule.co.uk) wrote:

Hello folks,

Trying to get started running the python word count example against the
portable runner using the beam 2.15 download.

In on terminal I have:

gradle :runners:spark:job-server:runShadow

And another:

gradle portableWordCount -PjobEndpoint=localhost:8099
-PenvironmentType=LOOPBACK

But when it starts it says:

And I have no clue where to start looking to fix that as I’d assume the
demos should just run?

Tom

Spicule Limited is registered in England & Wales. Company Number:
09954122. Registered office: First Floor, Telecom House, 125-135 Preston
Road, Brighton, England, BN1 6AF. VAT No. 251478891.