You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Kaxil Naik (JIRA)" <ji...@apache.org> on 2019/06/21 17:09:00 UTC
[jira] [Resolved] (AIRFLOW-2141) Cannot create airflow variables
when there is a list of dictionary as a value
[ https://issues.apache.org/jira/browse/AIRFLOW-2141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kaxil Naik resolved AIRFLOW-2141.
---------------------------------
Resolution: Fixed
Fix Version/s: 1.10.4
Resolved by https://github.com/apache/airflow/pull/4991
> Cannot create airflow variables when there is a list of dictionary as a value
> -----------------------------------------------------------------------------
>
> Key: AIRFLOW-2141
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2141
> Project: Apache Airflow
> Issue Type: Bug
> Components: aws
> Affects Versions: 1.8.0
> Reporter: Soundar
> Priority: Major
> Labels: beginner, newbie
> Fix For: 1.10.4
>
> Attachments: airflow_cli.png, airflow_cli2_crop.png
>
>
> I'm trying to create Airflow variables using a json file. I am trying to import airflow variables using UI(webserver) when I upload the json file I get this error "Missing file or syntax error" and when I try to upload using airflow cli not all the variables gets uploaded properly. The catch is that I have a list of dictionary in my json file, say
> ex:
> {
> "demo_archivedir": "/home/ubuntu/folders/archive",
> "demo_filepattern": [
> { "id": "reference", "pattern": "Sample Data.xlsx" }
> ,
> { "id": "sale", "pattern": "Sales.xlsx" }
> ],
> "demo_sourcepath": "/home/ubuntu/folders/input",
> "demo_workdir": "/home/ubuntu/folders/working"
> }
> I've attached two images
> img1. Using airflow variables cli command I was able to create partial variables from my json file(airflow_cli.png)img2. After inserting logs in the "airflow/bin/cli.py" file, I got this error. (airflow_cli2_crop.png)
> The thing is I gave this value through the Admin UI one by one and it worked. Then I exported those same variable using "airflow variables" cli command and tried importing them, still it failed and the above mentioned error still occurs.
> Note:
> I am using Python 3.5 with Airflow version 1.8
> The stack trace is as follows
> .compute-1.amazonaws.com:22] out: 0 of 4 variables successfully updated.
> .compute-1.amazonaws.com:22] out: Traceback (most recent call last):
> .compute-1.amazonaws.com:22] out: File "/home/ubuntu/Env/bin/airflow", line 28, in <module>
> .compute-1.amazonaws.com:22] out: args.func(args)
> .compute-1.amazonaws.com:22] out: File "/home/ubuntu/Env/lib/python3.5/site-packages/airflow/bin/cli.py", line 242, in variables
> .compute-1.amazonaws.com:22] out: import_helper(imp)
> .compute-1.amazonaws.com:22] out: File "/home/ubuntu/Env/lib/python3.5/site-packages/airflow/bin/cli.py", line 273, in import_helper
> .compute-1.amazonaws.com:22] out: Variable.set(k, v)
> .compute-1.amazonaws.com:22] out: File "/home/ubuntu/Env/lib/python3.5/site-packages/airflow/utils/db.py", line 53, in wrapper
> .compute-1.amazonaws.com:22] out: result = func(*args, **kwargs)
> .compute-1.amazonaws.com:22] out: File "/home/ubuntu/Env/lib/python3.5/site-packages/airflow/models.py", line 3615, in set
> .compute-1.amazonaws.com:22] out: session.add(Variable(key=key, val=stored_value))
> .compute-1.amazonaws.com:22] out: File "<string>", line 4, in __init__
> .compute-1.amazonaws.com:22] out: File "/home/ubuntu/Env/lib/python3.5/site-packages/sqlalchemy/orm/state.py", line 417, in _initialize_instance
> .compute-1.amazonaws.com:22] out: manager.dispatch.init_failure(self, args, kwargs)
> .compute-1.amazonaws.com:22] out: File "/home/ubuntu/Env/lib/python3.5/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__
> .compute-1.amazonaws.com:22] out: compat.reraise(exc_type, exc_value, exc_tb)
> .compute-1.amazonaws.com:22] out: File "/home/ubuntu/Env/lib/python3.5/site-packages/sqlalchemy/util/compat.py", line 187, in reraise
> .compute-1.amazonaws.com:22] out: raise value
> .compute-1.amazonaws.com:22] out: File "/home/ubuntu/Env/lib/python3.5/site-packages/sqlalchemy/orm/state.py", line 414, in _initialize_instance
> .compute-1.amazonaws.com:22] out: return manager.original_init(*mixed[1:], **kwargs)
> .compute-1.amazonaws.com:22] out: File "/home/ubuntu/Env/lib/python3.5/site-packages/sqlalchemy/ext/declarative/base.py", line 700, in _declarative_constructor
> .compute-1.amazonaws.com:22] out: setattr(self, k, kwargs[k])
> compute-1.amazonaws.com:22] out: File "<string>", line 1, in __set__
> .compute-1.amazonaws.com:22] out: File "/home/ubuntu/Env/lib/python3.5/site-packages/airflow/models.py", line 3550, in set_val
> .compute-1.amazonaws.com:22] out: self._val = FERNET.encrypt(bytes(value, 'utf-8')).decode()
> .compute-1.amazonaws.com:22] out: TypeError: encoding without a string argument
> .compute-1.amazonaws.com:22] out:
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)