You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/10/28 14:09:44 UTC
[GitHub] [airflow] kaxil opened a new issue #11920: Add how-to Guide for Sqlite operators
kaxil opened a new issue #11920:
URL: https://github.com/apache/airflow/issues/11920
**Description**
A guide that describes how to use all the operators for Sqlite (https://github.com/apache/airflow/tree/master/airflow/providers/sqlite/operators) would be useful.
Other guides are available:
https://airflow.readthedocs.io/en/latest/howto/operator/index.html
Source code for those guides are at:
https://github.com/apache/airflow/tree/master/docs/howto/operator
Are you wondering how to start contributing to this project? Start by reading our [contributor guide](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst)
Best regards,
Kaxil
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #11920: Add how-to Guide for Sqlite operators
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #11920:
URL: https://github.com/apache/airflow/issues/11920#issuecomment-736028298
@WilliamTun Use the following:
```python
from airflow.providers.sqlite.operators.sqlite import SqliteOperator
```
Check https://github.com/apache/airflow/blob/dee304b222d355b03794aa063f39e3ee13997730/airflow/operators/sqlite_operator.py#L23
https://github.com/apache/airflow/blob/dee304b222d355b03794aa063f39e3ee13997730/airflow/providers/sqlite/operators/sqlite.py#L25
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil closed issue #11920: Add how-to Guide for Sqlite operators
Posted by GitBox <gi...@apache.org>.
kaxil closed issue #11920:
URL: https://github.com/apache/airflow/issues/11920
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] WilliamTun commented on issue #11920: Add how-to Guide for Sqlite operators
Posted by GitBox <gi...@apache.org>.
WilliamTun commented on issue #11920:
URL: https://github.com/apache/airflow/issues/11920#issuecomment-727633873
Hey @kaxil
I have created/edited the following files:
_airflow/providers/sqlite/example_dags/__init__.py
airflow/providers/sqlite/example_dags/example_sqlite.py
docs/howto/operator/index.rst
docs/howto/operator/sqlite.rst_
However, I have two problems:
1. When I try to git push my changes, I am returned the following message:
ERROR: Permission to apache/airflow.git denied to WilliamTun.
fatal: Could not read from remote repository.
Could you please tell me where I can get the relevant ssh key to push to this repo?
2. In the sqlite.rst file, I was unsure what the input parameters were. Could you tell me where I can get the info? I assumed it was in airflow/providers/sqlite/hooks/sqlite.py
However, by looking at the code:
`class SqliteHook(DbApiHook):
"""Interact with SQLite."""
conn_name_attr = 'sqlite_conn_id'
default_conn_name = 'sqlite_default'
def get_conn(self) -> sqlite3.dbapi2.Connection:
"""Returns a sqlite connection object"""
conn_id = getattr(self, self.conn_name_attr)
airflow_conn = self.get_connection(conn_id)
conn = sqlite3.connect(airflow_conn.host)
return conn`
I could only assume the parameters needed to connect to sqlite was:
* - Parameter
- Input
* - conn_id: string
- connection ID
Please do correct me if I am wrong. Thank you.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] WilliamTun commented on issue #11920: Add how-to Guide for Sqlite operators
Posted by GitBox <gi...@apache.org>.
WilliamTun commented on issue #11920:
URL: https://github.com/apache/airflow/issues/11920#issuecomment-735254331
Hey @kaxil
I have successfully set up breeze.
I have created this folder:
airflow/providers/sqlite/example_dags
and using other examples as a template, I have created two files in this folder:
1) __init__.py
2) example_sqlite.py
I noticed other paralogous files for file 2 in other folders, eg providers/mysql/example_dags/example_mysql.py
has this import:
from airflow.operators.mysql_operator import MySqlOperator
HOWEVER,
`airflow.operators.mysql_operator is now deprecated. `
simialrly,
`airflow.operators.sqlite_operator is also deprecated. `
Thus, I have made the appropriate changes to call this line instead:
`from airflow.providers.sqlite.operators import SqliteOperator`
UNFORTUNATELY, this line is not working.
My IDE says: "cannot find reference to SqliteOperator" in __init__.py
FUTHERMORE, when I run pre-commit run.
I get:
```
pylint........................................................................................................................Failed
- hook id: pylint
- exit code: 2
************* Module airflow.providers.sqlite.example_dags.example_sqlite
airflow/providers/sqlite/example_dags/example_sqlite.py:29:0: E0611: No name 'SqliteOperator' in module 'airflow.providers.sqlite.operators' (no-name-in-module)
```
I am confused because I can see in the repo there is clearly an __init__.py and sqlite.py file in airflow/providers/sqlite/operators
and when we look in sqlite.py, there is clearly a class:
class SqliteOperator(BaseOperator)
So why can I not import this class? And subsequently, why is pylint failing when I run pre-commit tests?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #11920: Add how-to Guide for Sqlite operators
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #11920:
URL: https://github.com/apache/airflow/issues/11920#issuecomment-753215845
Closed by https://github.com/apache/airflow/pull/13196
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #11920: Add how-to Guide for Sqlite operators
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #11920:
URL: https://github.com/apache/airflow/issues/11920#issuecomment-718202658
Assigned this ticket to you, and please feel free to ask questions, don't worry at all :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #11920: Add how-to Guide for Sqlite operators
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #11920:
URL: https://github.com/apache/airflow/issues/11920#issuecomment-728869648
> Hey @kaxil
>
> I have created/edited the following files:
>
> _airflow/providers/sqlite/example_dags/**init**.py airflow/providers/sqlite/example_dags/example_sqlite.py docs/howto/operator/index.rst docs/howto/operator/sqlite.rst_
>
> However, I have two problems:
>
> 1. When I try to git push my changes, I am returned the following message:
> ERROR: Permission to apache/airflow.git denied to WilliamTun.
> fatal: Could not read from remote repository.
> Could you please tell me where I can get the relevant ssh key to push to this repo?
> 2. In the sqlite.rst file, I was unsure what the input parameters were. Could you tell me where I can get the info? I assumed it was in airflow/providers/sqlite/hooks/sqlite.py
> However, by looking at the code:
>
> `class SqliteHook(DbApiHook):
> """Interact with SQLite."""
>
> ```
> conn_name_attr = 'sqlite_conn_id'
> default_conn_name = 'sqlite_default'
>
> def get_conn(self) -> sqlite3.dbapi2.Connection:
> """Returns a sqlite connection object"""
> conn_id = getattr(self, self.conn_name_attr)
> airflow_conn = self.get_connection(conn_id)
> conn = sqlite3.connect(airflow_conn.host)
> return conn`
> ```
>
> I could only assume the parameters needed to connect to sqlite was:
>
> * * Parameter
> * Input
> * * conn_id: string
> * connection ID
>
> Please do correct me if I am wrong. Thank you.
1) You won't have write access to https://github.com/apache/airflow, so you will have to create a fork - https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#step-1-fork-the-apache-airflow-repo
2) Same as MySQL: https://airflow.readthedocs.io/en/latest/howto/operator/mysql.html
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] WilliamTun commented on issue #11920: Add how-to Guide for Sqlite operators
Posted by GitBox <gi...@apache.org>.
WilliamTun commented on issue #11920:
URL: https://github.com/apache/airflow/issues/11920#issuecomment-718198360
Hey Kaxil.
I'd like to have a go at this task please.
Please do note that this is the first time I've ever tried to contribute to an open source project so you'll have to excuse me if I end up asking what may appear to be very basic questions in the coming days.
Will
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org