You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Avijit Saha <av...@gmail.com> on 2020/07/07 21:16:54 UTC
Not able to see WordCount output in docker /tmp/...
While trying to run the Beam WordCount example on Flink runner using Job
Manager and Taskmanager docker images using the input pom.xml file, I can
see the task manager is creating the following files but no output inside
the counts-0000 file as expected - Any pointer what might be wrong here?
Thanks
bash-4.4$ ls -lrt /tmp
total 4
drwxr-xr-x 1 flink flink 2 Jul 7 21:15 hsperfdata_flink
-rw-r--r-- 1 flink flink 1179 Jul 7 21:15
jaas-3210126497662256938.conf
drwxr-xr-x 1 flink flink 0 Jul 7 21:15
flink-io-5922e9cc-91ee-4ae3-bee3-90006db36aaa
drwxr-xr-x 1 flink flink 0 Jul 7 21:15
blobStore-1f336cf6-8885-4aff-bd36-ddf38f11c0a0
drwxr-xr-x 1 flink flink 0 Jul 7 21:15
blobStore-13b09c7f-e371-430f-90aa-802aec50735e
drwxr-xr-x 1 flink flink 0 Jul 7 21:15
flink-dist-cache-6330c1d8-3ab8-4baa-812b-485d6d850c1d
-rw-r--r-- 1 flink flink 0 Jul 7 21:15
counts-00000-of-00001
drwxr-xr-x 1 flink flink 0 Jul 7 21:15
flink-netty-shuffle-69803f1d-c641-41b0-a090-7dc542ed98f4
drwxr-xr-x 1 flink flink 0 Jul 7 21:15 localState
Re: Not able to see WordCount output in docker /tmp/...
Posted by Avijit Saha <av...@gmail.com>.
Ok - it works in docker when I pick up a file from inside docker system -
e.g - for the following when both the input and output files are
inside docker:
inputFile=/opt/flink/conf/flink-conf.yaml --output=/tmp/counts!
It does not work when using a the file from the host system volume mapped
to docker as in - any pointer?
volumes:
input_data:
name: input_volume
services:
job-cluster:
image: ${FLINK_DOCKER_IMAGE_NAME:-flink-job}
ports:
- "8081:8081"
volumes:
- input_data:~/input/pom.xml:ro
command: job-cluster --job-classname ${FLINK_JOB}
-Djobmanager.rpc.address=job-cluster
-Dparallelism.default=${DEFAULT_PARALLELISM:-1} ${SAVEPOINT_OPTIONS}
${FLINK_JOB_ARGUMENTS}
taskmanager:
image: ${FLINK_DOCKER_IMAGE_NAME:-flink-job}
volumes:
- input_data:/home/asaha/input/pom.xml:ro
command: task-manager -Djobmanager.rpc.address=job-cluster
scale: ${DEFAULT_PARALLELISM:-1}
On Tue, Jul 7, 2020 at 4:27 PM Robert Bradshaw <ro...@google.com> wrote:
> Does it work when you write to a distributed filesystem? (One issue
> with Docker is that the manager and each of their workers have their
> own local filesystem.)
>
> On Tue, Jul 7, 2020 at 2:17 PM Avijit Saha <av...@gmail.com> wrote:
> >
> > While trying to run the Beam WordCount example on Flink runner using
> Job Manager and Taskmanager docker images using the input pom.xml file, I
> can see the task manager is creating the following files but no output
> inside the counts-0000 file as expected - Any pointer what might be wrong
> here?
> >
> > Thanks
> >
> > bash-4.4$ ls -lrt /tmp
> > total 4
> > drwxr-xr-x 1 flink flink 2 Jul 7 21:15 hsperfdata_flink
> > -rw-r--r-- 1 flink flink 1179 Jul 7 21:15
> jaas-3210126497662256938.conf
> > drwxr-xr-x 1 flink flink 0 Jul 7 21:15
> flink-io-5922e9cc-91ee-4ae3-bee3-90006db36aaa
> > drwxr-xr-x 1 flink flink 0 Jul 7 21:15
> blobStore-1f336cf6-8885-4aff-bd36-ddf38f11c0a0
> > drwxr-xr-x 1 flink flink 0 Jul 7 21:15
> blobStore-13b09c7f-e371-430f-90aa-802aec50735e
> > drwxr-xr-x 1 flink flink 0 Jul 7 21:15
> flink-dist-cache-6330c1d8-3ab8-4baa-812b-485d6d850c1d
> > -rw-r--r-- 1 flink flink 0 Jul 7 21:15
> counts-00000-of-00001
> > drwxr-xr-x 1 flink flink 0 Jul 7 21:15
> flink-netty-shuffle-69803f1d-c641-41b0-a090-7dc542ed98f4
> > drwxr-xr-x 1 flink flink 0 Jul 7 21:15 localState
>
Re: Not able to see WordCount output in docker /tmp/...
Posted by Robert Bradshaw <ro...@google.com>.
Does it work when you write to a distributed filesystem? (One issue
with Docker is that the manager and each of their workers have their
own local filesystem.)
On Tue, Jul 7, 2020 at 2:17 PM Avijit Saha <av...@gmail.com> wrote:
>
> While trying to run the Beam WordCount example on Flink runner using Job Manager and Taskmanager docker images using the input pom.xml file, I can see the task manager is creating the following files but no output inside the counts-0000 file as expected - Any pointer what might be wrong here?
>
> Thanks
>
> bash-4.4$ ls -lrt /tmp
> total 4
> drwxr-xr-x 1 flink flink 2 Jul 7 21:15 hsperfdata_flink
> -rw-r--r-- 1 flink flink 1179 Jul 7 21:15 jaas-3210126497662256938.conf
> drwxr-xr-x 1 flink flink 0 Jul 7 21:15 flink-io-5922e9cc-91ee-4ae3-bee3-90006db36aaa
> drwxr-xr-x 1 flink flink 0 Jul 7 21:15 blobStore-1f336cf6-8885-4aff-bd36-ddf38f11c0a0
> drwxr-xr-x 1 flink flink 0 Jul 7 21:15 blobStore-13b09c7f-e371-430f-90aa-802aec50735e
> drwxr-xr-x 1 flink flink 0 Jul 7 21:15 flink-dist-cache-6330c1d8-3ab8-4baa-812b-485d6d850c1d
> -rw-r--r-- 1 flink flink 0 Jul 7 21:15 counts-00000-of-00001
> drwxr-xr-x 1 flink flink 0 Jul 7 21:15 flink-netty-shuffle-69803f1d-c641-41b0-a090-7dc542ed98f4
> drwxr-xr-x 1 flink flink 0 Jul 7 21:15 localState