You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "shivannakarthik (via GitHub)" <gi...@apache.org> on 2023/05/04 12:32:59 UTC
[GitHub] [airflow] shivannakarthik opened a new issue, #31058: ShieldedInstanceConfig not being accepted by Dataproc operators ClusterGenerator
shivannakarthik opened a new issue, #31058:
URL: https://github.com/apache/airflow/issues/31058
### Apache Airflow version
Other Airflow 2 version (please specify below)
### What happened
While creating dataproc cluster using Dataproc Cluster Create Operator, I am unable to to toggle ShieldedInstanceConfig related properties. It works when I create using gcloud command or direct Dataproc APIs though.
This has been discussed here -> https://github.com/apache/airflow/issues/28023
I believe the operator must support this property. Could this be a potential PR?
### What you think should happen instead
_No response_
### How to reproduce
While passing the cluster configs, pass the gce_cluster_config as follows to create the cluster using Dataproc Cluster Create Operator.
"gce_cluster_config": {
"metadata": {
"enable-pepperdata": "false",
"ha_flag": "no"
},
"shielded_instance_config": {
"enable_secure_boot": 1,
"enable_vtpm": 1,
"enable_integrity_monitoring": 1
}
}
### Operating System
debian11
### Versions of Apache Airflow Providers
_No response_
### Deployment
Astronomer
### Deployment details
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] shivannakarthik commented on issue #31058: ShieldedInstanceConfig not being accepted by Dataproc operators ClusterGenerator
Posted by "shivannakarthik (via GitHub)" <gi...@apache.org>.
shivannakarthik commented on issue #31058:
URL: https://github.com/apache/airflow/issues/31058#issuecomment-1534992307
It does not address the issue, but provides a workaround. Ideally I should be able to create the cluster using the airlfow datapro cluster create operator passing the shielded_instance_config properties
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] ShieldedInstanceConfig not being accepted by Dataproc operators ClusterGenerator [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal closed issue #31058: ShieldedInstanceConfig not being accepted by Dataproc operators ClusterGenerator
URL: https://github.com/apache/airflow/issues/31058
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #31058: ShieldedInstanceConfig not being accepted by Dataproc operators ClusterGenerator
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #31058:
URL: https://github.com/apache/airflow/issues/31058#issuecomment-1534698584
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] ShieldedInstanceConfig not being accepted by Dataproc operators ClusterGenerator [airflow]
Posted by "sai3563 (via GitHub)" <gi...@apache.org>.
sai3563 commented on issue #31058:
URL: https://github.com/apache/airflow/issues/31058#issuecomment-2080009300
@eladkal @shivannakarthik It seems no modifications in the `DataprocCreateClusterOperator` are needed. While testing with the latest dev branch i.e. `2.10.0.dev0`, I was able to get `shielded_instance_config` working using the operator. I tried it with both True and False values for the parameters and confirmed it to be working from GCP Console.
Here is the full code I used
```
from datetime import datetime, timedelta
from airflow import DAG
from airflow.providers.google.cloud.operators.dataproc import DataprocCreateClusterOperator
# Constants
PROJECT_ID = 'my-project'
REGION = 'asia-south1'
CLUSTER_NAME = 'testing-cluster'
ZONE = 'asia-south1-a'
# Default DAG arguments
default_args = {
'start_date': datetime.now() - timedelta(days=1)
}
# Define the DAG
dag = DAG(
'dataproc_create_cluster_dag',
default_args=default_args,
description='A DAG to create a Dataproc cluster',
schedule=None,
catchup=False
)
# Dataproc cluster configuration
cluster_config = {
"gce_cluster_config": {
"zone_uri": f"https://www.googleapis.com/compute/v1/projects/{PROJECT_ID}/zones/{ZONE}",
"metadata": {
"enable-pepperdata": "false",
"ha_flag": "no"
},
"shielded_instance_config": {
"enable_secure_boot": True,
"enable_vtpm": True,
"enable_integrity_monitoring": True
}
}
}
# Create Dataproc cluster task
create_cluster = DataprocCreateClusterOperator(
task_id='create_dataproc_cluster',
project_id=PROJECT_ID,
cluster_name=CLUSTER_NAME,
cluster_config=cluster_config,
region=REGION,
dag=dag
)
# Define task sequence
create_cluster
```
@eladkal I believe this issue can be closed for the moment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] rohan472000 commented on issue #31058: ShieldedInstanceConfig not being accepted by Dataproc operators ClusterGenerator
Posted by "rohan472000 (via GitHub)" <gi...@apache.org>.
rohan472000 commented on issue #31058:
URL: https://github.com/apache/airflow/issues/31058#issuecomment-1534987545
If my suggestion worked for you then kindly close this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] shivannakarthik commented on issue #31058: ShieldedInstanceConfig not being accepted by Dataproc operators ClusterGenerator
Posted by "shivannakarthik (via GitHub)" <gi...@apache.org>.
shivannakarthik commented on issue #31058:
URL: https://github.com/apache/airflow/issues/31058#issuecomment-1534980576
Thanks Rohan. Yes, I am able to create it using gcloud and Dataproc API. Will raise a PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] rohan472000 commented on issue #31058: ShieldedInstanceConfig not being accepted by Dataproc operators ClusterGenerator
Posted by "rohan472000 (via GitHub)" <gi...@apache.org>.
rohan472000 commented on issue #31058:
URL: https://github.com/apache/airflow/issues/31058#issuecomment-1534858099
You could use the `gcloud` command or direct `Dataproc` APIs to create the cluster with the `ShieldedInstanceConfig` properties until this feature is added to the Airflow operator.
Here is how you can create a cluster with `ShieldedInstanceConfig` using the `gcloud` command:
gcloud dataproc clusters create my-cluster \
--region=us-central1 \
--subnet=default \
--zone=us-central1-a \
--master-machine-type=n1-standard-2 \
--worker-machine-type=n1-standard-2 \
--num-workers=2 \
--image-version=1.5-debian10 \
--project=my-project \
--shielded-instance-config enable-secure-boot enable-vtpm enable-integrity-monitoring
And here is how you can create a cluster with `ShieldedInstanceConfig` using the `Dataproc` APIs:
from google.cloud import dataproc_v1 as dataproc
from google.protobuf import json_format
project_id = 'my-project'
region = 'us-central1'
cluster_name = 'my-cluster'
client = dataproc.ClusterControllerClient()
cluster = {
'project_id': project_id,
'cluster_name': cluster_name,
'config': {
'gce_cluster_config': {
'metadata': {
'enable-pepperdata': 'false',
'ha_flag': 'no'
},
'shielded_instance_config': {
'enable_secure_boot': True,
'enable_vtpm': True,
'enable_integrity_monitoring': True
}
}
}
}
op = client.create_cluster(
request={'project_id': project_id, 'region': region, 'cluster': cluster}
)
print(json_format.MessageToDict(op))
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] ShieldedInstanceConfig not being accepted by Dataproc operators ClusterGenerator [airflow]
Posted by "sai3563 (via GitHub)" <gi...@apache.org>.
sai3563 commented on issue #31058:
URL: https://github.com/apache/airflow/issues/31058#issuecomment-2076657554
@eladkal Please assign this to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] shivannakarthik commented on issue #31058: ShieldedInstanceConfig not being accepted by Dataproc operators ClusterGenerator
Posted by "shivannakarthik (via GitHub)" <gi...@apache.org>.
shivannakarthik commented on issue #31058:
URL: https://github.com/apache/airflow/issues/31058#issuecomment-1535012920
Working on it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org