You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Роман VVvKamper <vv...@deeplay.io> on 2021/11/08 14:52:58 UTC
Getting mini-cluster logs when debugging pyflink from IDE
Hello,
I'm trying to debug flink and pyflink job from IDE using mini cluster (local mode).
When i doing it in the java flink, everything works like a charm - i can see flink mini-cluster logs in the console.
But when i run pyflink job in local mode (through the IDE of by simply calling python src/pipeline.py) i can see only logs from python code in the console but not a mini-cluster logs.
Is there a way to enable mini-cluster logs from the python local mode? Maybe i can contribute some code for it if you help me where to start.
Re: Getting mini-cluster logs when debugging pyflink from IDE
Posted by Dian Fu <di...@gmail.com>.
The logging directory could be configured via one the following way:
- environment variable: FLINK_LOG_DIR
- configuration 'env.log.dir' in
$PYTHON_INSTALLATION_DIR/site-packages/pyflink/conf/flink-conf.yaml
You could refer to [1] for more details.
PS: This is only available for 1.13.3+ and 1.14.0+.
Regards,
Dian
[1]
https://github.com/apache/flink/blob/master/flink-python/pyflink/pyflink_gateway_server.py#L124
On Tue, Nov 9, 2021 at 12:52 PM Роман VVvKamper <vv...@deeplay.io>
wrote:
> Thanks for help, I’ll look into it :)
>
> Two more questions:
>
> Is there a way to configure this path? For example to write logs to a file
> in the working dir?
>
> And is there a way to redirect logst from file to stdout?
>
> On 9 Nov 2021, at 09:00, Dian Fu <di...@gmail.com> wrote:
>
> Hi,
>
> The logs should appear in the log file of the TaskManger and you could
> find it under directory $PYTHON_INSTALLATION_DIR/site-packages/pyflink/log/
>
> Regards,
> Dian
>
> On Mon, Nov 8, 2021 at 10:53 PM Роман VVvKamper <vv...@deeplay.io>
> wrote:
>
>> Hello,
>>
>> I'm trying to debug flink and pyflink job from IDE using mini cluster
>> (local mode).
>>
>> When i doing it in the java flink, everything works like a charm - i can
>> see flink mini-cluster logs in the console.
>>
>> But when i run pyflink job in local mode (through the IDE of by simply
>> calling python src/pipeline.py) i can see only logs from python code in the
>> console but not a mini-cluster logs.
>>
>> Is there a way to enable mini-cluster logs from the python local mode?
>> Maybe i can contribute some code for it if you help me where to start.
>
>
>
Re: Getting mini-cluster logs when debugging pyflink from IDE
Posted by Роман VVvKamper <vv...@deeplay.io>.
Thanks for help, I’ll look into it :)
Two more questions:
Is there a way to configure this path? For example to write logs to a file in the working dir?
And is there a way to redirect logst from file to stdout?
On 9 Nov 2021, at 09:00, Dian Fu <di...@gmail.com>> wrote:
Hi,
The logs should appear in the log file of the TaskManger and you could find it under directory $PYTHON_INSTALLATION_DIR/site-packages/pyflink/log/
Regards,
Dian
On Mon, Nov 8, 2021 at 10:53 PM Роман VVvKamper <vv...@deeplay.io>> wrote:
Hello,
I'm trying to debug flink and pyflink job from IDE using mini cluster (local mode).
When i doing it in the java flink, everything works like a charm - i can see flink mini-cluster logs in the console.
But when i run pyflink job in local mode (through the IDE of by simply calling python src/pipeline.py) i can see only logs from python code in the console but not a mini-cluster logs.
Is there a way to enable mini-cluster logs from the python local mode? Maybe i can contribute some code for it if you help me where to start.
Re: Getting mini-cluster logs when debugging pyflink from IDE
Posted by Dian Fu <di...@gmail.com>.
Hi,
The logs should appear in the log file of the TaskManger and you could find
it under directory $PYTHON_INSTALLATION_DIR/site-packages/pyflink/log/
Regards,
Dian
On Mon, Nov 8, 2021 at 10:53 PM Роман VVvKamper <vv...@deeplay.io>
wrote:
> Hello,
>
> I'm trying to debug flink and pyflink job from IDE using mini cluster
> (local mode).
>
> When i doing it in the java flink, everything works like a charm - i can
> see flink mini-cluster logs in the console.
>
> But when i run pyflink job in local mode (through the IDE of by simply
> calling python src/pipeline.py) i can see only logs from python code in the
> console but not a mini-cluster logs.
>
> Is there a way to enable mini-cluster logs from the python local mode?
> Maybe i can contribute some code for it if you help me where to start.