You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/07/16 11:57:28 UTC
[GitHub] [airflow] mik-laj opened a new issue #9855: Import connections/variables from a file
mik-laj opened a new issue #9855:
URL: https://github.com/apache/airflow/issues/9855
Hello,
We have a new local file system secret backend that allows us to load connections and variables from files. We support 3 file formats: YAML, JSON, ENV.
https://airflow.readthedocs.io/en/latest/howto/use-alternative-secrets-backend.html#local-filesystem-secrets-backend
https://github.com/apache/airflow/blob/master/airflow/secrets/local_filesystem.py
I think we can use the same code to create new commands for the CLI that allow us o import connections or variables.
Best regards,
Kamil
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] caioalmeida97 commented on issue #9855: Import connections/variables from a file
Posted by GitBox <gi...@apache.org>.
caioalmeida97 commented on issue #9855:
URL: https://github.com/apache/airflow/issues/9855#issuecomment-786125920
@mik-laj is there any updates on this feature? I tried to run `airflow connections import $JSON_FILE` but it didn't work :(
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] utkarsharma2 edited a comment on issue #9855: Import connections/variables from a file
Posted by GitBox <gi...@apache.org>.
utkarsharma2 edited a comment on issue #9855:
URL: https://github.com/apache/airflow/issues/9855#issuecomment-661879437
Hey @mik-laj ,
There is already a feature to import variables - `airflow variables import file`, however, this command only accepts the JSON file. Should we augment, existing command?
Also, I'll add --overwrite-existing flag.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] utkarsharma2 edited a comment on issue #9855: Import connections/variables from a file
Posted by GitBox <gi...@apache.org>.
utkarsharma2 edited a comment on issue #9855:
URL: https://github.com/apache/airflow/issues/9855#issuecomment-661879437
Hey @mik-laj ,
There is already a feature to import variables - `airflow variables import file`, however, this command only accepts the JSON file. Should we augment, existing command?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj edited a comment on issue #9855: Import connections/variables from a file
Posted by GitBox <gi...@apache.org>.
mik-laj edited a comment on issue #9855:
URL: https://github.com/apache/airflow/issues/9855#issuecomment-662062832
I thought about the extra option for a longer time and I don't think this will cover all cases.
I think, we need `conflict-disposition` It should specifies the action that occurs if the variables/connection already exists.
We should support the following values are supported:
`OVERWRITE`: If the table variable/connection already exists, we should overwrites the data.
`IGNORE`: If the table variable/connection already exists, we should skip this connection/variable.
`RESTRICT`: If the variable/connection already exists, an exception is raised.
The default value should be RESTRICT. Each command invocation should be atomic and only occurs if all changes is able to complete successfully (one transaction).
Example:
```
airflow connection import /files/variables.json --conflict-disposition=overwrite
airflow connection import /files/variables.json --conflict-disposition=ignore
airflow connection import /files/variables.json --conflict-disposition=restrict
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj edited a comment on issue #9855: Import connections/variables from a file
Posted by GitBox <gi...@apache.org>.
mik-laj edited a comment on issue #9855:
URL: https://github.com/apache/airflow/issues/9855#issuecomment-662062832
I thought about the extra option for a longer time and I don't think this will cover all cases.
I think, we need `conflict-disposition` It should specifies the action that occurs if the variables/connection already exists.
We should support the following values are supported:
`OVERWRITE`: If the table variable/connection already exists, we should overwrites the data.
`IGNORE`: If the table variable/connection already exists, we should skip this connection/variable.
`RESTRICT`: If the variable/connection already exists, an exception is raised.
The default value should be RESTRICT. Each command invocation should be atomic and only occurs if all changes is able to complete successfully (one transaction).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] utkarsharma2 commented on issue #9855: Import connections/variables from a file
Posted by GitBox <gi...@apache.org>.
utkarsharma2 commented on issue #9855:
URL: https://github.com/apache/airflow/issues/9855#issuecomment-661879437
Hey @mik-laj,
There is already a feature to import variables - `airflow variables import file`, however, this command only accepts the JSON file. Should we augment, existing command?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] natanweinberger commented on issue #9855: Import connections/variables from a file
Posted by GitBox <gi...@apache.org>.
natanweinberger commented on issue #9855:
URL: https://github.com/apache/airflow/issues/9855#issuecomment-812883007
I started a first crack at this issue a few days ago, apologies for not posting sooner. I'll open a PR for it shortly. @chutcheson, would be great to get your feedback since you're interested in this issue as well!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #9855: Import connections/variables from a file
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #9855:
URL: https://github.com/apache/airflow/issues/9855#issuecomment-662067652
We should handle reading values from a standard input. `airflow connection import -` should read value from stdin instead file.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #9855: Import connections/variables from a file
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #9855:
URL: https://github.com/apache/airflow/issues/9855#issuecomment-812613908
That sounds correct to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj edited a comment on issue #9855: Import connections/variables from a file
Posted by GitBox <gi...@apache.org>.
mik-laj edited a comment on issue #9855:
URL: https://github.com/apache/airflow/issues/9855#issuecomment-662062832
I thought about the extra option for a longer time and I don't think this will cover all cases.
I think, we need `conflict-disposition` It should specifies the action that occurs if the variables/connection already exists.
We should support the following values are supported:
`OVERWRITE`: If the table variable/connection already exists, we should overwrites the data.
`IGNORE`: If the table variable/connection already exists, we should skip this connection/variable.
`RESTRICT`: If the variable/connection already exists, we should raise an exception.
The default value should be RESTRICT. Each command invocation should be atomic and only occurs if all changes is able to complete successfully (one transaction).
Example:
```
airflow connection import /files/variables.json --conflict-disposition=overwrite
airflow connection import /files/variables.json --conflict-disposition=ignore
airflow connection import /files/variables.json --conflict-disposition=restrict
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #9855: Import connections/variables from a file
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #9855:
URL: https://github.com/apache/airflow/issues/9855#issuecomment-786162914
Here is PR:https://github.com/apache/airflow/pull/9907 Feel free to continue it
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dimberman closed issue #9855: Import connections/variables from a file
Posted by GitBox <gi...@apache.org>.
dimberman closed issue #9855:
URL: https://github.com/apache/airflow/issues/9855
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] utkarsharma2 commented on issue #9855: Import connections/variables from a file
Posted by GitBox <gi...@apache.org>.
utkarsharma2 commented on issue #9855:
URL: https://github.com/apache/airflow/issues/9855#issuecomment-660540546
Hey mik-laj,
What should happen to existing connections? should we override them?/let them be untouched?/ask for user input?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj edited a comment on issue #9855: Import connections/variables from a file
Posted by GitBox <gi...@apache.org>.
mik-laj edited a comment on issue #9855:
URL: https://github.com/apache/airflow/issues/9855#issuecomment-662062832
I thought about the extra option for a longer time and I don't think this will cover all cases.
I think, we need `conflict-disposition` It should specifies the action that occurs if the variables/connection already exists.
We should support the following values are supported:
`OVERWRITE`: If the table variable/connection already exists, we should overwrites the data.
`IGNORE`: If the table variable/connection already exists, we should skip this connection/variable.
`RESTRICT`: If the variable/connection already exists, an exception is raised.
The default value should be RESTRICT. Each command invocation should be atomic and only occurs if all changes is able to complete successfully (one transaction).
Example:
```
airflow connection import /files/variables.json --merge-disposition=overwrite
airflow connection import /files/variables.json --merge-disposition=ignore
airflow connection import /files/variables.json --merge-disposition=restrict
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] chutcheson commented on issue #9855: Import connections/variables from a file
Posted by GitBox <gi...@apache.org>.
chutcheson commented on issue #9855:
URL: https://github.com/apache/airflow/issues/9855#issuecomment-812222602
I'm looking for a good first issue to work on.
Is this a correct account of the feature request?
> When using the airflow command line interface tool, I want to be able to import connections to databases or other services using a file that specifies the details associated with the connection.
Is there anything else that I should understand with respect to the request?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] utkarsharma2 removed a comment on issue #9855: Import connections/variables from a file
Posted by GitBox <gi...@apache.org>.
utkarsharma2 removed a comment on issue #9855:
URL: https://github.com/apache/airflow/issues/9855#issuecomment-660540546
Hey mik-laj,
What should happen to existing connections? should we override them?/let them be untouched?/ask for user input?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj edited a comment on issue #9855: Import connections/variables from a file
Posted by GitBox <gi...@apache.org>.
mik-laj edited a comment on issue #9855:
URL: https://github.com/apache/airflow/issues/9855#issuecomment-662062832
I thought about the extra option for a longer time and I don't think this will cover all cases.
I thiink, we need `merge-dispossition` It should specifies the action that occurs if the variables/connection already exists.
We should support the following values are supported:
`OVERWRITE`: If the table variable/connection already exists, we should overwrites the data.
`IGNORE`: If the table variable/connection already exists, we should skip this connection/variable.
`RESTRICT`: If the variable/connection already exists, an exception is raised.
The default value should be RESTRICT. Each command invocation should be atomic and only occurs if all changes is able to complete successfully (one transaction).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #9855: Import connections/variables from a file
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #9855:
URL: https://github.com/apache/airflow/issues/9855#issuecomment-662062832
I thought about the extra option for a longer time and I don't think this will cover all cases.
I thiink, we need `merge-dispossition` It should specifies the action that occurs if the variables/connection already exists.
We should support the following values are supported:
`OVERWRITE`: If the table variable/connection exists, we should overwrites the data.
`IGNORE`: If the table variable/connection exists, we should skip this connection/variable.
`RESTRICT`: If the variable/connection already exists, an exception is raised.
The default value should be RESTRICT. Each command invocation should be atomic and only occurs if all changes is able to complete successfully (one transaction).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #9855: Import connections/variables from a file
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #9855:
URL: https://github.com/apache/airflow/issues/9855#issuecomment-660559678
>Hey mik-laj,
>
> What should happen to existing connections? should we override them?/let them be untouched?/ask for user input?
@utkarsharma2 By default, the operation should be canceled, but with an additional option it should be possible to change this behavior ``--overwrite-existing``.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #9855: Import connections/variables from a file
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #9855:
URL: https://github.com/apache/airflow/issues/9855#issuecomment-812851014
@chutcheson Yes. We just want to import connections from a file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org