You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by boci <bo...@gmail.com> on 2014/07/19 23:39:27 UTC
Out of any idea
Hi guys!
I run out of ideas... I created a spark streaming job (kafka -> spark ->
ES).
If I start my app local machine (inside the editor, but connect to the real
kafka and ES) the application work correctly.
If I start it in my docker container (same kafka and ES, local mode
(local[4]) like inside my editor) the application connect to kafka, receive
the message but after that nothing happened (I put config/log4j.properties
to debug mode and I see BlockGenerator receive the data bu after that
nothing happened with that.
(first step I simply run a map to print the received data with log4j)
I hope somebody can help... :(
b0c1
----------------------------------------------------------------------------------------------------------------------------------
Skype: boci13, Hangout: boci.boci@gmail.com
Re: Out of any idea
Posted by boci <bo...@gmail.com>.
Hi i created a demo input.
https://gist.github.com/b0c1/e3721af839feec433b56#file-gistfile1-txt-L10
As you see in line 10 the json received (json/string nevermind)
After that everything is ok, except the processing not started...
Any idea?
Please help guys... I doesn't have any idea what I miss... dependency?
wrong spark version?
No error in the log just the processing is not started...
----------------------------------------------------------------------------------------------------------------------------------
Skype: boci13, Hangout: boci.boci@gmail.com
On Sun, Jul 20, 2014 at 4:58 AM, Krishna Sankar <ks...@gmail.com> wrote:
> Probably you have - if not, try a very simple app in the docker container
> and make sure it works. Sometimes resource contention/allocation can get in
> the way. This happened to me in the YARN container.
> Also try single worker thread.
> Cheers
> <k/>
>
>
> On Sat, Jul 19, 2014 at 2:39 PM, boci <bo...@gmail.com> wrote:
>
>> Hi guys!
>>
>> I run out of ideas... I created a spark streaming job (kafka -> spark ->
>> ES).
>> If I start my app local machine (inside the editor, but connect to the
>> real kafka and ES) the application work correctly.
>> If I start it in my docker container (same kafka and ES, local mode
>> (local[4]) like inside my editor) the application connect to kafka, receive
>> the message but after that nothing happened (I put config/log4j.properties
>> to debug mode and I see BlockGenerator receive the data bu after that
>> nothing happened with that.
>> (first step I simply run a map to print the received data with log4j)
>>
>> I hope somebody can help... :(
>>
>> b0c1
>>
>> ----------------------------------------------------------------------------------------------------------------------------------
>> Skype: boci13, Hangout: boci.boci@gmail.com
>>
>
>
Re: Out of any idea
Posted by Krishna Sankar <ks...@gmail.com>.
Probably you have - if not, try a very simple app in the docker container
and make sure it works. Sometimes resource contention/allocation can get in
the way. This happened to me in the YARN container.
Also try single worker thread.
Cheers
<k/>
On Sat, Jul 19, 2014 at 2:39 PM, boci <bo...@gmail.com> wrote:
> Hi guys!
>
> I run out of ideas... I created a spark streaming job (kafka -> spark ->
> ES).
> If I start my app local machine (inside the editor, but connect to the
> real kafka and ES) the application work correctly.
> If I start it in my docker container (same kafka and ES, local mode
> (local[4]) like inside my editor) the application connect to kafka, receive
> the message but after that nothing happened (I put config/log4j.properties
> to debug mode and I see BlockGenerator receive the data bu after that
> nothing happened with that.
> (first step I simply run a map to print the received data with log4j)
>
> I hope somebody can help... :(
>
> b0c1
>
> ----------------------------------------------------------------------------------------------------------------------------------
> Skype: boci13, Hangout: boci.boci@gmail.com
>
Re: Out of any idea
Posted by Tathagata Das <ta...@gmail.com>.
Could you collect debug level logs and send us. Without logs its hard to
speculate anything. :)
TD
On Sat, Jul 19, 2014 at 2:39 PM, boci <bo...@gmail.com> wrote:
> Hi guys!
>
> I run out of ideas... I created a spark streaming job (kafka -> spark ->
> ES).
> If I start my app local machine (inside the editor, but connect to the
> real kafka and ES) the application work correctly.
> If I start it in my docker container (same kafka and ES, local mode
> (local[4]) like inside my editor) the application connect to kafka, receive
> the message but after that nothing happened (I put config/log4j.properties
> to debug mode and I see BlockGenerator receive the data bu after that
> nothing happened with that.
> (first step I simply run a map to print the received data with log4j)
>
> I hope somebody can help... :(
>
> b0c1
>
> ----------------------------------------------------------------------------------------------------------------------------------
> Skype: boci13, Hangout: boci.boci@gmail.com
>