You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Unais T <tp...@gmail.com> on 2018/11/22 12:33:19 UTC
not able to launch dataflow job - permission issues with service account
I am trying to run a simple data flow job in google cloud - its running
perfectly in local - but when I tried to launch it I'm getting the
following error - I tried debug a lot
can someone help on this
INFO:root:Created job with id: [2018-11-22_02_57_07-12079060901530487381]
INFO:root:To access the Dataflow monitoring console, please navigate
to https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-11-22_02_57_07-12079060901530487381?project=kkfas-main-account-221808
INFO:root:Job 2018-11-22_02_57_07-12079060901530487381 is in state
JOB_STATE_PENDING
INFO:root:2018-11-22T10:57:08.052Z: JOB_MESSAGE_DETAILED: Autoscaling
is enabled for job 2018-11-22_02_57_07-12079060901530487381. The
number of workers will be between 1 and 1000.
INFO:root:2018-11-22T10:57:08.072Z: JOB_MESSAGE_DETAILED: Autoscaling
was automatically enabled for job
2018-11-22_02_57_07-12079060901530487381.
INFO:root:2018-11-22T10:57:40.405Z: JOB_MESSAGE_ERROR: Workflow
failed. Causes: There was a problem refreshing your credentials.
Please check:
1. Dataflow API is enabled for your project.
2. There is a robot service account for your project:
service-[project
number]@dataflow-service-producer-prod.iam.gserviceaccount.com should
have access to your project. If this account does not appear in the
permissions tab for your project, contact Dataflow support.
INFO:root:Job 2018-11-22_02_57_07-12079060901530487381 is in state
JOB_STATE_FAILED
Traceback (most recent call last):
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py",
line 162, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py",
line 72, in _run_code
exec code in run_globals
File "/Users/u/Projects//digital/test.py", line 49, in <module>
run()
File "/Users/u/Projects//dataflow//digital/test.py", line 44, in run
return p.run().wait_until_finish()
File "/Users/u/VirtualEnv/dataflow/lib/python2.7/site-packages/apache_beam/runners/dataflow/dataflow_runner.py",
line 1122, in wait_until_finish
(self.state, getattr(self._runner, 'last_error_msg', None)), self)
apache_beam.runners.dataflow.dataflow_runner.DataflowRuntimeException:
Dataflow pipeline failed. State: FAILED, Error:
Workflow failed. Causes: There was a problem refreshing your
credentials. Please check:
1. Dataflow API is enabled for your project.
2. There is a robot service account for your project:
service-[project
number]@dataflow-service-producer-prod.iam.gserviceaccount.com should
have access to your project. If this account does not appear in the
permissions tab for your project, contact Dataflow support.
Re: not able to launch dataflow job - permission issues with service account
Posted by Unais T <tp...@gmail.com>.
My problem I generated a service account - it has all permission and I
downloaded the credentials.json for that - But when I'm trying to launch
the job - I think it still taking the default account.
Is some configuration I have to do take the service account I created?
On Thu, Nov 22, 2018 at 10:08 PM Ankur Chauhan <an...@malloc64.com> wrote:
> As the message says, enable the api from cloud console of gcloud cli
> utility. (Refer dataflow docs)
>
> If that’s enabled, ensure the robot account has the permissions it needs -
> dataflow api, gcs and anything else you are trying to access.
>
> On Thu, Nov 22, 2018 at 04:33 Unais T <tp...@gmail.com> wrote:
>
>> I am trying to run a simple data flow job in google cloud - its running
>> perfectly in local - but when I tried to launch it I'm getting the
>> following error - I tried debug a lot
>>
>> can someone help on this
>>
>> INFO:root:Created job with id: [2018-11-22_02_57_07-12079060901530487381]
>> INFO:root:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-11-22_02_57_07-12079060901530487381?project=kkfas-main-account-221808
>> INFO:root:Job 2018-11-22_02_57_07-12079060901530487381 is in state JOB_STATE_PENDING
>> INFO:root:2018-11-22T10:57:08.052Z: JOB_MESSAGE_DETAILED: Autoscaling is enabled for job 2018-11-22_02_57_07-12079060901530487381. The number of workers will be between 1 and 1000.
>> INFO:root:2018-11-22T10:57:08.072Z: JOB_MESSAGE_DETAILED: Autoscaling was automatically enabled for job 2018-11-22_02_57_07-12079060901530487381.
>> INFO:root:2018-11-22T10:57:40.405Z: JOB_MESSAGE_ERROR: Workflow failed. Causes: There was a problem refreshing your credentials. Please check:
>> 1. Dataflow API is enabled for your project.
>> 2. There is a robot service account for your project:
>> service-[project number]@dataflow-service-producer-prod.iam.gserviceaccount.com should have access to your project. If this account does not appear in the permissions tab for your project, contact Dataflow support.
>> INFO:root:Job 2018-11-22_02_57_07-12079060901530487381 is in state JOB_STATE_FAILED
>> Traceback (most recent call last):
>> File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 162, in _run_module_as_main
>> "__main__", fname, loader, pkg_name)
>> File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 72, in _run_code
>> exec code in run_globals
>> File "/Users/u/Projects//digital/test.py", line 49, in <module>
>> run()
>> File "/Users/u/Projects//dataflow//digital/test.py", line 44, in run
>> return p.run().wait_until_finish()
>> File "/Users/u/VirtualEnv/dataflow/lib/python2.7/site-packages/apache_beam/runners/dataflow/dataflow_runner.py", line 1122, in wait_until_finish
>> (self.state, getattr(self._runner, 'last_error_msg', None)), self)
>> apache_beam.runners.dataflow.dataflow_runner.DataflowRuntimeException: Dataflow pipeline failed. State: FAILED, Error:
>> Workflow failed. Causes: There was a problem refreshing your credentials. Please check:
>> 1. Dataflow API is enabled for your project.
>> 2. There is a robot service account for your project:
>> service-[project number]@dataflow-service-producer-prod.iam.gserviceaccount.com should have access to your project. If this account does not appear in the permissions tab for your project, contact Dataflow support.
>>
>>
Re: not able to launch dataflow job - permission issues with service account
Posted by Ankur Chauhan <an...@malloc64.com>.
As the message says, enable the api from cloud console of gcloud cli
utility. (Refer dataflow docs)
If that’s enabled, ensure the robot account has the permissions it needs -
dataflow api, gcs and anything else you are trying to access.
On Thu, Nov 22, 2018 at 04:33 Unais T <tp...@gmail.com> wrote:
> I am trying to run a simple data flow job in google cloud - its running
> perfectly in local - but when I tried to launch it I'm getting the
> following error - I tried debug a lot
>
> can someone help on this
>
> INFO:root:Created job with id: [2018-11-22_02_57_07-12079060901530487381]
> INFO:root:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-11-22_02_57_07-12079060901530487381?project=kkfas-main-account-221808
> INFO:root:Job 2018-11-22_02_57_07-12079060901530487381 is in state JOB_STATE_PENDING
> INFO:root:2018-11-22T10:57:08.052Z: JOB_MESSAGE_DETAILED: Autoscaling is enabled for job 2018-11-22_02_57_07-12079060901530487381. The number of workers will be between 1 and 1000.
> INFO:root:2018-11-22T10:57:08.072Z: JOB_MESSAGE_DETAILED: Autoscaling was automatically enabled for job 2018-11-22_02_57_07-12079060901530487381.
> INFO:root:2018-11-22T10:57:40.405Z: JOB_MESSAGE_ERROR: Workflow failed. Causes: There was a problem refreshing your credentials. Please check:
> 1. Dataflow API is enabled for your project.
> 2. There is a robot service account for your project:
> service-[project number]@dataflow-service-producer-prod.iam.gserviceaccount.com should have access to your project. If this account does not appear in the permissions tab for your project, contact Dataflow support.
> INFO:root:Job 2018-11-22_02_57_07-12079060901530487381 is in state JOB_STATE_FAILED
> Traceback (most recent call last):
> File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 162, in _run_module_as_main
> "__main__", fname, loader, pkg_name)
> File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 72, in _run_code
> exec code in run_globals
> File "/Users/u/Projects//digital/test.py", line 49, in <module>
> run()
> File "/Users/u/Projects//dataflow//digital/test.py", line 44, in run
> return p.run().wait_until_finish()
> File "/Users/u/VirtualEnv/dataflow/lib/python2.7/site-packages/apache_beam/runners/dataflow/dataflow_runner.py", line 1122, in wait_until_finish
> (self.state, getattr(self._runner, 'last_error_msg', None)), self)
> apache_beam.runners.dataflow.dataflow_runner.DataflowRuntimeException: Dataflow pipeline failed. State: FAILED, Error:
> Workflow failed. Causes: There was a problem refreshing your credentials. Please check:
> 1. Dataflow API is enabled for your project.
> 2. There is a robot service account for your project:
> service-[project number]@dataflow-service-producer-prod.iam.gserviceaccount.com should have access to your project. If this account does not appear in the permissions tab for your project, contact Dataflow support.
>
>