You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Bolke de Bruin <bd...@gmail.com> on 2018/07/01 10:02:34 UTC
Re: Apache Airflow 1.10.0b3
Separating the tests is where the effort lies. So having that as a consequence of splitting the packages would be nice. It has come up a couple of times but it was not picked up unfortunately.
B.
> On 28 Jun 2018, at 08:32, Maxime Beauchemin <ma...@gmail.com> wrote:
>
> It would be so nice to have a fast test suite. Having to wait for Travis
> for up to an hour makes many workflows (like working on a release) super
> painful.
>
> I spoke with folks at Astronomer recently about moving all operators and
> hooks to another Python package that airflow would import. This would allow
> for independent test suites and to have a more regular release cadence on
> hooks and operators. What do you think?
>
> Max
>
> On Wed, Jun 27, 2018 at 11:18 PM Bolke de Bruin <bd...@gmail.com> wrote:
>
>> Arghhh. The downside of doing this late at night and wanting to go to
>> bed... :-). Will make a new one
>>
>> Sent from my iPhone
>>
>>> On 28 Jun 2018, at 00:07, Chris Fei <cf...@gmail.com> wrote:
>>>
>>> Great, thank you! I just took this for a quick spin and it looks like
>>> there's DB migration task missing. The task you committed just recently,
>>> 9635ae0956e7_index_faskfail.py, has a down_revision of 856955da8476
>>> which can't be found when running airflow initdb (seehttps://
>> github.com/apache/incubator-airflow/tree/v1-10-test/airflow/migrations/versions
>> ).
>>> Chris
>>>
>>>
>>>> On Wed, Jun 27, 2018, at 5:09 PM, Bolke de Bruin wrote:
>>>> Hi All,
>>>>
>>>> I have created a sdist package that is available at:
>>>>
>>>>
>> http://people.apache.org/~bolke/apache-airflow-1.10.0b3+incubating.tar.gz
>>>> <
>> http://people.apache.org/~bolke/apache-airflow-1.10.0b3+incubating.tar.gz>>
>>
>>>> In order to distinguish it from an actual (apache) release it is:
>>>>
>>>> 1. Marked as beta (python package managers do not install beta
>>>> versions by default - PEP 440)> 2. It is not signed
>>>> 3. It is not at an official apache distribution location
>>>>
>>>> You can also put something like this in a requirements.txt file:
>>>>
>>>> git+
>>>>
>> https://github.com/apache/incubator-airflow@v1-10-test#egg=apache-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,redis,slack,s3
>>>> <
>> https://github.com/apache/incubator-airflow@v1-10-test#egg=apache-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,redis,slack,s3
>>>
>>>> ]> <https://github.com/rodrigc/incubator-airflow@master#egg=apache-
>>>> airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,redis,slack,s-
>>>> 3]
>>>> <
>> https://github.com/rodrigc/incubator-airflow@master#egg=apache-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,redis,slack,s3][1]
>>>
>>>>>>
>>>> and then "pip install -r requirements.txt”.
>>>>
>>>> I hope that after this beta we can go to RC and start voting on 1.10.>
>>>> Cheers
>>>> Bolke
>>>
>>>
>>> Links:
>>>
>>> 1.
>> https://github.com/rodrigc/incubator-airflow@master#egg=apache-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,redis,slack,s3]%20%3Chttps://github.com/rodrigc/incubator-airflow@master#egg=apache-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,redis,slack,s3]
>> <https://github.com/rodrigc/incubator-airflow@master#egg=apache-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,redis,slack,s3]%20%3Chttps://github.com/rodrigc/incubator-airflow@master%23egg=apache-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,redis,slack,s3]>
>>
Re: Splitting hooks/operators
Posted by Ash Berlin-Taylor <as...@firemirror.com>.
I have a few questions about the split: How do we ensure that changes to core doesn't break hooks/operators unintentionally? Will we release hooks at the same time as Airflow? Will there some hooks/ops in core still? Will apache-airflow (the python package) depend upon apache-airflow-operators? The otherway around?
The bigger question to me is how much will splitting the hooks help test times? Looking at one of Travis jobs just now and the slow tests were https://travis-ci.org/apache/incubator-airflow/jobs/402914949#L6076: <https://travis-ci.org/apache/incubator-airflow/jobs/402914949#L6076:>
<>[success] 3.48% tests.transplant_class.<locals>.C.test_scheduler_start_date: 50.6752s
<>[success] 2.38% tests.transplant_class.<locals>.C.test_scheduler_multiprocessing: 34.5760s
<>[success] 2.38% tests.transplant_class.<locals>.C.test_new_import_error_replaces_old: 34.5627s
<>[success] 2.12% tests.transplant_class.<locals>.C.test_backfill: 30.8402s
<>[success] 1.39% tests.executors.test_celery_executor.CeleryExecutorTest.test_celery_integration: 20.1523s
<>[success] 1.32% tests.transplant_class.<locals>.C.test_schedule_dag_no_end_date_up_to_today_only: 19.2137s
<>[success] 1.06% tests.transplant_class.<locals>.C.test_impersonation_subdag: 15.4438s
<>[success] 1.04% tests.sensors.test_timeout_sensor.SensorTimeoutTest.test_timeout: 15.0731s
<>[success] 1.01% tests.transplant_class.<locals>.C.test_subdag_deadlock: 14.7244s
<>[success] 0.97% tests.www_rbac.test_views.TestPoolModelView.test_create_pool_with_same_name: 14.0634s
<>[success] 0.96% tests.www_rbac.test_views.TestLogView.test_get_file_task_log: 14.0030s
<>[success] 0.95% tests.www_rbac.test_views.TestVariableModelView.test_can_handle_error_on_decrypt: 13.7853s
<>[success] 0.93% tests.www_rbac.test_views.TestVariableModelView.test_import_variables: 13.4654s
<>[success] 0.92% tests.www_rbac.test_views.TestVersionView.test_version: 13.3112s
<>[success] 0.91% tests.www_rbac.test_views.TestVariableModelView.test_xss_prevention: 13.1712s
<>[success] 0.90% tests.transplant_class.<locals>.C.test_impersonation_custom: 13.1109s
<>[success] 0.89% tests.www_rbac.test_views.TestPoolModelView.test_create_pool_with_empty_name: 13.0072s
<>[success] 0.89% tests.transplant_class.<locals>.C.test_run_ignores_all_dependencies: 13.0002s
<>[success] 0.89% tests.www_rbac.test_views.TestLogView.test_get_logs_with_metadata: 12.9184s
<>[success] 0.88% tests.www_rbac.test_views.TestGanttView.test_dt_nr_dr_form_default_parameters: 12.7953s
<>[success] 0.87% tests.www_rbac.test_views.TestLogView.test_get_logs_with_null_metadata: 12.7197s
<>[success] 0.87% tests.transplant_class.<locals>.C.test_cli_backfill_depends_on_past: 12.6946s
<>[success] 0.87% tests.transplant_class.<locals>.C.test_backfill_max_limit_check: 12.6605s
<>[success] 0.86% tests.transplant_class.<locals>.C.test_backfill_max_limit_check_complete_loop: 12.4513s
The slowest operator test only appears as the 100th slowest test!
I'm not against splitting tests if we have answers to the questions :)
-ash
> On 9 Jul 2018, at 21:56, Taylor Edmiston <tedmiston@gmail.com <ma...@gmail.com>> wrote:
>
> We discussed the splitting out of hooks & operators from core internally at
> Astronomer again today. I should be able to allocate some time to
> splitting out hooks & operators for the purpose of speeding up the CI. A
> faster CI on core would be hugely beneficial for all contributors.
>
> I created a Jira issue for this at https://issues.apache.org/ <https://issues.apache.org/>
> jira/browse/AIRFLOW-2732.
>
> If anyone has pointers for this work, I'm all ears. I've done a lot with
> Python unit tests in the past but this part of the Airflow codebase is new
> to me.
>
> Taylor
>
> *Taylor Edmiston*
> Blog <https://blog.tedmiston.com/ <https://blog.tedmiston.com/>> | CV
> <https://stackoverflow.com/cv/taylor <https://stackoverflow.com/cv/taylor>> | LinkedIn
> <https://www.linkedin.com/in/tedmiston/ <https://www.linkedin.com/in/tedmiston/>> | AngelList
> <https://angel.co/taylor <https://angel.co/taylor>> | Stack Overflow
> <https://stackoverflow.com/users/149428/taylor-edmiston <https://stackoverflow.com/users/149428/taylor-edmiston>>
>
>
> On Sun, Jul 1, 2018 at 6:02 AM, Bolke de Bruin <bdbruin@gmail.com <ma...@gmail.com>> wrote:
>
>> Separating the tests is where the effort lies. So having that as a
>> consequence of splitting the packages would be nice. It has come up a
>> couple of times but it was not picked up unfortunately.
>>
>> B.
>>
>>> On 28 Jun 2018, at 08:32, Maxime Beauchemin <maximebeauchemin@gmail.com <ma...@gmail.com>>
>> wrote:
>>>
>>> It would be so nice to have a fast test suite. Having to wait for Travis
>>> for up to an hour makes many workflows (like working on a release) super
>>> painful.
>>>
>>> I spoke with folks at Astronomer recently about moving all operators and
>>> hooks to another Python package that airflow would import. This would
>> allow
>>> for independent test suites and to have a more regular release cadence on
>>> hooks and operators. What do you think?
>>>
>>> Max
>>>
>>> On Wed, Jun 27, 2018 at 11:18 PM Bolke de Bruin <bdbruin@gmail.com <ma...@gmail.com>>
>> wrote:
>>>
>>>> Arghhh. The downside of doing this late at night and wanting to go to
>>>> bed... :-). Will make a new one
>>>>
>>>> Sent from my iPhone
>>>>
>>>>> On 28 Jun 2018, at 00:07, Chris Fei <cfei18@gmail.com <ma...@gmail.com>> wrote:
>>>>>
>>>>> Great, thank you! I just took this for a quick spin and it looks like
>>>>> there's DB migration task missing. The task you committed just
>> recently,
>>>>> 9635ae0956e7_index_faskfail.py, has a down_revision of 856955da8476
>>>>> which can't be found when running airflow initdb (seehttps://
>>>> github.com/apache/incubator-airflow/tree/v1-10-test/airflow/ <http://github.com/apache/incubator-airflow/tree/v1-10-test/airflow/>
>> migrations/versions
>>>> ).
>>>>> Chris
>>>>>
>>>>>
>>>>>> On Wed, Jun 27, 2018, at 5:09 PM, Bolke de Bruin wrote:
>>>>>> Hi All,
>>>>>>
>>>>>> I have created a sdist package that is available at:
>>>>>>
>>>>>>
>>>> http://people.apache.org/~bolke/apache-airflow-1.10.0b3+incu <http://people.apache.org/~bolke/apache-airflow-1.10.0b3+incu>
>> bating.tar.gz
>>>>>> <
>>>> http://people.apache.org/~bolke/apache-airflow-1.10.0b3+incu <http://people.apache.org/~bolke/apache-airflow-1.10.0b3+incu>
>> bating.tar.gz>>
>>>>
>>>>>> In order to distinguish it from an actual (apache) release it is:
>>>>>>
>>>>>> 1. Marked as beta (python package managers do not install beta
>>>>>> versions by default - PEP 440)> 2. It is not signed
>>>>>> 3. It is not at an official apache distribution location
>>>>>>
>>>>>> You can also put something like this in a requirements.txt file:
>>>>>>
>>>>>> git+
>>>>>>
>>>> https://github.com/apache/incubator-airflow@v1-10-test#egg= <https://github.com/apache/incubator-airflow@v1-10-test#egg=>
>> apache-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,
>> postgres,redis,slack,s3
>>>>>> <
>>>> https://github.com/apache/incubator-airflow@v1-10-test#egg= <https://github.com/apache/incubator-airflow@v1-10-test#egg=>
>> apache-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,
>> postgres,redis,slack,s3
>>>>>
>>>>>> ]> <https://github.com/rodrigc/incubator-airflow@master#egg=apache- <https://github.com/rodrigc/incubator-airflow@master#egg=apache->
>>>>>> airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,redi
>> s,slack,s-
>>>>>> 3]
>>>>>> <
>>>> https://github.com/rodrigc/incubator-airflow@master#egg=apac <https://github.com/rodrigc/incubator-airflow@master#egg=apac>
>> he-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,
>> redis,slack,s3][1]
>>>>>
>>>>>>>>
>>>>>> and then "pip install -r requirements.txt”.
>>>>>>
>>>>>> I hope that after this beta we can go to RC and start voting on 1.10.>
>>>>>> Cheers
>>>>>> Bolke
>>>>>
>>>>>
>>>>> Links:
>>>>>
>>>>> 1.
>>>> https://github.com/rodrigc/incubator-airflow@master#egg=apac <https://github.com/rodrigc/incubator-airflow@master#egg=apac>
>> he-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,
>> redis,slack,s3]%20%3Chttps://github.com/rodrigc/incubator- <http://github.com/rodrigc/incubator->
>> airflow@master#egg=apache-airflow[celery,crypto,emr,
>> hive,hdfs,ldap,mysql,postgres,redis,slack,s3]
>> <https://github.com/rodrigc/incubator-airflow@master#egg=apache-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,redis,slack,s3]%20%3Chttps://github.com/rodrigc/incubator-airflow@master%23egg=apache-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,redis,slack,s3] <https://github.com/rodrigc/incubator-airflow@master#egg=apache-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,redis,slack,s3]%20%3Chttps://github.com/rodrigc/incubator-airflow@master%23egg=apache-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,redis,slack,s3]>>
>>>> <https://github.com/rodrigc/incubator-airflow@master#egg=apa <https://github.com/rodrigc/incubator-airflow@master#egg=apa>
>> che-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,
>> redis,slack,s3]%20%3Chttps://github.com/rodrigc/incubator- <http://github.com/rodrigc/incubator->
>> airflow@master%23egg=apache-airflow[celery,crypto,emr,
>> hive,hdfs,ldap,mysql,postgres,redis,slack,s3]>
>>>>
>>
>>
Re: Apache Airflow 1.10.0b3
Posted by "Driesprong, Fokko" <fo...@driesprong.frl>.
Ah, I picked the wrong tar.gz, Thanks B
Cheers, Fokko
2018-07-13 15:07 GMT+02:00 Bolke de Bruin <bd...@gmail.com>:
> Please use the Rc Fokko. This has been fixed in the RC.
>
> B.
>
> Verstuurd vanaf mijn iPad
>
> > Op 13 jul. 2018 om 15:01 heeft Driesprong, Fokko <fo...@driesprong.frl>
> het volgende geschreven:
> >
> > Hi all,
> >
> > I just tried to install a local Airflow instance within Docker, using
> Python 3.5, but I got an error on initdb. Anyone else experiencing this? It
> has to do with the schema migrations.
> >
> > Cheers,
> > Fokko
> >
> > 2018-07-09 22:56 GMT+02:00 Taylor Edmiston <te...@gmail.com>:
> >> We discussed the splitting out of hooks & operators from core
> internally at
> >> Astronomer again today. I should be able to allocate some time to
> >> splitting out hooks & operators for the purpose of speeding up the CI.
> A
> >> faster CI on core would be hugely beneficial for all contributors.
> >>
> >> I created a Jira issue for this at https://issues.apache.org/
> >> jira/browse/AIRFLOW-2732.
> >>
> >> If anyone has pointers for this work, I'm all ears. I've done a lot
> with
> >> Python unit tests in the past but this part of the Airflow codebase is
> new
> >> to me.
> >>
> >> Taylor
> >>
> >> *Taylor Edmiston*
> >> Blog <https://blog.tedmiston.com/> | CV
> >> <https://stackoverflow.com/cv/taylor> | LinkedIn
> >> <https://www.linkedin.com/in/tedmiston/> | AngelList
> >> <https://angel.co/taylor> | Stack Overflow
> >> <https://stackoverflow.com/users/149428/taylor-edmiston>
> >>
> >>
> >> On Sun, Jul 1, 2018 at 6:02 AM, Bolke de Bruin <bd...@gmail.com>
> wrote:
> >>
> >> > Separating the tests is where the effort lies. So having that as a
> >> > consequence of splitting the packages would be nice. It has come up a
> >> > couple of times but it was not picked up unfortunately.
> >> >
> >> > B.
> >> >
> >> > > On 28 Jun 2018, at 08:32, Maxime Beauchemin <
> maximebeauchemin@gmail.com>
> >> > wrote:
> >> > >
> >> > > It would be so nice to have a fast test suite. Having to wait for
> Travis
> >> > > for up to an hour makes many workflows (like working on a release)
> super
> >> > > painful.
> >> > >
> >> > > I spoke with folks at Astronomer recently about moving all
> operators and
> >> > > hooks to another Python package that airflow would import. This
> would
> >> > allow
> >> > > for independent test suites and to have a more regular release
> cadence on
> >> > > hooks and operators. What do you think?
> >> > >
> >> > > Max
> >> > >
> >> > > On Wed, Jun 27, 2018 at 11:18 PM Bolke de Bruin <bd...@gmail.com>
> >> > wrote:
> >> > >
> >> > >> Arghhh. The downside of doing this late at night and wanting to go
> to
> >> > >> bed... :-). Will make a new one
> >> > >>
> >> > >> Sent from my iPhone
> >> > >>
> >> > >>> On 28 Jun 2018, at 00:07, Chris Fei <cf...@gmail.com> wrote:
> >> > >>>
> >> > >>> Great, thank you! I just took this for a quick spin and it looks
> like
> >> > >>> there's DB migration task missing. The task you committed just
> >> > recently,
> >> > >>> 9635ae0956e7_index_faskfail.py, has a down_revision of
> 856955da8476
> >> > >>> which can't be found when running airflow initdb (seehttps://
> >> > >> github.com/apache/incubator-airflow/tree/v1-10-test/airflow/
> >> > migrations/versions
> >> > >> ).
> >> > >>> Chris
> >> > >>>
> >> > >>>
> >> > >>>> On Wed, Jun 27, 2018, at 5:09 PM, Bolke de Bruin wrote:
> >> > >>>> Hi All,
> >> > >>>>
> >> > >>>> I have created a sdist package that is available at:
> >> > >>>>
> >> > >>>>
> >> > >> http://people.apache.org/~bolke/apache-airflow-1.10.0b3+incu
> >> > bating.tar.gz
> >> > >>>> <
> >> > >> http://people.apache.org/~bolke/apache-airflow-1.10.0b3+incu
> >> > bating.tar.gz>>
> >> > >>
> >> > >>>> In order to distinguish it from an actual (apache) release it is:
> >> > >>>>
> >> > >>>> 1. Marked as beta (python package managers do not install beta
> >> > >>>> versions by default - PEP 440)> 2. It is not signed
> >> > >>>> 3. It is not at an official apache distribution location
> >> > >>>>
> >> > >>>> You can also put something like this in a requirements.txt file:
> >> > >>>>
> >> > >>>> git+
> >> > >>>>
> >> > >> https://github.com/apache/incubator-airflow@v1-10-test#egg=
> >> > apache-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,
> >> > postgres,redis,slack,s3
> >> > >>>> <
> >> > >> https://github.com/apache/incubator-airflow@v1-10-test#egg=
> >> > apache-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,
> >> > postgres,redis,slack,s3
> >> > >>>
> >> > >>>> ]> <https://github.com/rodrigc/incubator-airflow@master#egg=
> apache-
> >> > >>>> airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,redi
> >> > s,slack,s-
> >> > >>>> 3]
> >> > >>>> <
> >> > >> https://github.com/rodrigc/incubator-airflow@master#egg=apac
> >> > he-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,
> >> > redis,slack,s3][1]
> >> > >>>
> >> > >>>>>>
> >> > >>>> and then "pip install -r requirements.txt”.
> >> > >>>>
> >> > >>>> I hope that after this beta we can go to RC and start voting on
> 1.10.>
> >> > >>>> Cheers
> >> > >>>> Bolke
> >> > >>>
> >> > >>>
> >> > >>> Links:
> >> > >>>
> >> > >>> 1.
> >> > >> https://github.com/rodrigc/incubator-airflow@master#egg=apac
> >> > he-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,
> >> > redis,slack,s3]%20%3Chttps://github.com/rodrigc/incubator-
> >> > airflow@master#egg=apache-airflow[celery,crypto,emr,
> >> > hive,hdfs,ldap,mysql,postgres,redis,slack,s3]
> >> > <https://github.com/rodrigc/incubator-airflow@master#egg=
> apache-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,
> postgres,redis,slack,s3]%20%3Chttps://github.com/rodrigc/
> incubator-airflow@master%23egg=apache-airflow[celery,
> crypto,emr,hive,hdfs,ldap,mysql,postgres,redis,slack,s3]>
> >> > >> <https://github.com/rodrigc/incubator-airflow@master#egg=apa
> >> > che-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,
> >> > redis,slack,s3]%20%3Chttps://github.com/rodrigc/incubator-
> >> > airflow@master%23egg=apache-airflow[celery,crypto,emr,
> >> > hive,hdfs,ldap,mysql,postgres,redis,slack,s3]>
> >> > >>
> >> >
> >> >
> >
> > <airvo.txt>
>
Re: Apache Airflow 1.10.0b3
Posted by Bolke de Bruin <bd...@gmail.com>.
Please use the Rc Fokko. This has been fixed in the RC.
B.
Verstuurd vanaf mijn iPad
> Op 13 jul. 2018 om 15:01 heeft Driesprong, Fokko <fo...@driesprong.frl> het volgende geschreven:
>
> Hi all,
>
> I just tried to install a local Airflow instance within Docker, using Python 3.5, but I got an error on initdb. Anyone else experiencing this? It has to do with the schema migrations.
>
> Cheers,
> Fokko
>
> 2018-07-09 22:56 GMT+02:00 Taylor Edmiston <te...@gmail.com>:
>> We discussed the splitting out of hooks & operators from core internally at
>> Astronomer again today. I should be able to allocate some time to
>> splitting out hooks & operators for the purpose of speeding up the CI. A
>> faster CI on core would be hugely beneficial for all contributors.
>>
>> I created a Jira issue for this at https://issues.apache.org/
>> jira/browse/AIRFLOW-2732.
>>
>> If anyone has pointers for this work, I'm all ears. I've done a lot with
>> Python unit tests in the past but this part of the Airflow codebase is new
>> to me.
>>
>> Taylor
>>
>> *Taylor Edmiston*
>> Blog <https://blog.tedmiston.com/> | CV
>> <https://stackoverflow.com/cv/taylor> | LinkedIn
>> <https://www.linkedin.com/in/tedmiston/> | AngelList
>> <https://angel.co/taylor> | Stack Overflow
>> <https://stackoverflow.com/users/149428/taylor-edmiston>
>>
>>
>> On Sun, Jul 1, 2018 at 6:02 AM, Bolke de Bruin <bd...@gmail.com> wrote:
>>
>> > Separating the tests is where the effort lies. So having that as a
>> > consequence of splitting the packages would be nice. It has come up a
>> > couple of times but it was not picked up unfortunately.
>> >
>> > B.
>> >
>> > > On 28 Jun 2018, at 08:32, Maxime Beauchemin <ma...@gmail.com>
>> > wrote:
>> > >
>> > > It would be so nice to have a fast test suite. Having to wait for Travis
>> > > for up to an hour makes many workflows (like working on a release) super
>> > > painful.
>> > >
>> > > I spoke with folks at Astronomer recently about moving all operators and
>> > > hooks to another Python package that airflow would import. This would
>> > allow
>> > > for independent test suites and to have a more regular release cadence on
>> > > hooks and operators. What do you think?
>> > >
>> > > Max
>> > >
>> > > On Wed, Jun 27, 2018 at 11:18 PM Bolke de Bruin <bd...@gmail.com>
>> > wrote:
>> > >
>> > >> Arghhh. The downside of doing this late at night and wanting to go to
>> > >> bed... :-). Will make a new one
>> > >>
>> > >> Sent from my iPhone
>> > >>
>> > >>> On 28 Jun 2018, at 00:07, Chris Fei <cf...@gmail.com> wrote:
>> > >>>
>> > >>> Great, thank you! I just took this for a quick spin and it looks like
>> > >>> there's DB migration task missing. The task you committed just
>> > recently,
>> > >>> 9635ae0956e7_index_faskfail.py, has a down_revision of 856955da8476
>> > >>> which can't be found when running airflow initdb (seehttps://
>> > >> github.com/apache/incubator-airflow/tree/v1-10-test/airflow/
>> > migrations/versions
>> > >> ).
>> > >>> Chris
>> > >>>
>> > >>>
>> > >>>> On Wed, Jun 27, 2018, at 5:09 PM, Bolke de Bruin wrote:
>> > >>>> Hi All,
>> > >>>>
>> > >>>> I have created a sdist package that is available at:
>> > >>>>
>> > >>>>
>> > >> http://people.apache.org/~bolke/apache-airflow-1.10.0b3+incu
>> > bating.tar.gz
>> > >>>> <
>> > >> http://people.apache.org/~bolke/apache-airflow-1.10.0b3+incu
>> > bating.tar.gz>>
>> > >>
>> > >>>> In order to distinguish it from an actual (apache) release it is:
>> > >>>>
>> > >>>> 1. Marked as beta (python package managers do not install beta
>> > >>>> versions by default - PEP 440)> 2. It is not signed
>> > >>>> 3. It is not at an official apache distribution location
>> > >>>>
>> > >>>> You can also put something like this in a requirements.txt file:
>> > >>>>
>> > >>>> git+
>> > >>>>
>> > >> https://github.com/apache/incubator-airflow@v1-10-test#egg=
>> > apache-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,
>> > postgres,redis,slack,s3
>> > >>>> <
>> > >> https://github.com/apache/incubator-airflow@v1-10-test#egg=
>> > apache-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,
>> > postgres,redis,slack,s3
>> > >>>
>> > >>>> ]> <https://github.com/rodrigc/incubator-airflow@master#egg=apache-
>> > >>>> airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,redi
>> > s,slack,s-
>> > >>>> 3]
>> > >>>> <
>> > >> https://github.com/rodrigc/incubator-airflow@master#egg=apac
>> > he-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,
>> > redis,slack,s3][1]
>> > >>>
>> > >>>>>>
>> > >>>> and then "pip install -r requirements.txt”.
>> > >>>>
>> > >>>> I hope that after this beta we can go to RC and start voting on 1.10.>
>> > >>>> Cheers
>> > >>>> Bolke
>> > >>>
>> > >>>
>> > >>> Links:
>> > >>>
>> > >>> 1.
>> > >> https://github.com/rodrigc/incubator-airflow@master#egg=apac
>> > he-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,
>> > redis,slack,s3]%20%3Chttps://github.com/rodrigc/incubator-
>> > airflow@master#egg=apache-airflow[celery,crypto,emr,
>> > hive,hdfs,ldap,mysql,postgres,redis,slack,s3]
>> > <https://github.com/rodrigc/incubator-airflow@master#egg=apache-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,redis,slack,s3]%20%3Chttps://github.com/rodrigc/incubator-airflow@master%23egg=apache-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,redis,slack,s3]>
>> > >> <https://github.com/rodrigc/incubator-airflow@master#egg=apa
>> > che-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,
>> > redis,slack,s3]%20%3Chttps://github.com/rodrigc/incubator-
>> > airflow@master%23egg=apache-airflow[celery,crypto,emr,
>> > hive,hdfs,ldap,mysql,postgres,redis,slack,s3]>
>> > >>
>> >
>> >
>
> <airvo.txt>
Re: Apache Airflow 1.10.0b3
Posted by "Driesprong, Fokko" <fo...@driesprong.frl>.
Hi all,
I just tried to install a local Airflow instance within Docker, using
Python 3.5, but I got an error on initdb. Anyone else experiencing this? It
has to do with the schema migrations.
Cheers,
Fokko
2018-07-09 22:56 GMT+02:00 Taylor Edmiston <te...@gmail.com>:
> We discussed the splitting out of hooks & operators from core internally at
> Astronomer again today. I should be able to allocate some time to
> splitting out hooks & operators for the purpose of speeding up the CI. A
> faster CI on core would be hugely beneficial for all contributors.
>
> I created a Jira issue for this at https://issues.apache.org/
> jira/browse/AIRFLOW-2732.
>
> If anyone has pointers for this work, I'm all ears. I've done a lot with
> Python unit tests in the past but this part of the Airflow codebase is new
> to me.
>
> Taylor
>
> *Taylor Edmiston*
> Blog <https://blog.tedmiston.com/> | CV
> <https://stackoverflow.com/cv/taylor> | LinkedIn
> <https://www.linkedin.com/in/tedmiston/> | AngelList
> <https://angel.co/taylor> | Stack Overflow
> <https://stackoverflow.com/users/149428/taylor-edmiston>
>
>
> On Sun, Jul 1, 2018 at 6:02 AM, Bolke de Bruin <bd...@gmail.com> wrote:
>
> > Separating the tests is where the effort lies. So having that as a
> > consequence of splitting the packages would be nice. It has come up a
> > couple of times but it was not picked up unfortunately.
> >
> > B.
> >
> > > On 28 Jun 2018, at 08:32, Maxime Beauchemin <
> maximebeauchemin@gmail.com>
> > wrote:
> > >
> > > It would be so nice to have a fast test suite. Having to wait for
> Travis
> > > for up to an hour makes many workflows (like working on a release)
> super
> > > painful.
> > >
> > > I spoke with folks at Astronomer recently about moving all operators
> and
> > > hooks to another Python package that airflow would import. This would
> > allow
> > > for independent test suites and to have a more regular release cadence
> on
> > > hooks and operators. What do you think?
> > >
> > > Max
> > >
> > > On Wed, Jun 27, 2018 at 11:18 PM Bolke de Bruin <bd...@gmail.com>
> > wrote:
> > >
> > >> Arghhh. The downside of doing this late at night and wanting to go to
> > >> bed... :-). Will make a new one
> > >>
> > >> Sent from my iPhone
> > >>
> > >>> On 28 Jun 2018, at 00:07, Chris Fei <cf...@gmail.com> wrote:
> > >>>
> > >>> Great, thank you! I just took this for a quick spin and it looks like
> > >>> there's DB migration task missing. The task you committed just
> > recently,
> > >>> 9635ae0956e7_index_faskfail.py, has a down_revision of 856955da8476
> > >>> which can't be found when running airflow initdb (seehttps://
> > >> github.com/apache/incubator-airflow/tree/v1-10-test/airflow/
> > migrations/versions
> > >> ).
> > >>> Chris
> > >>>
> > >>>
> > >>>> On Wed, Jun 27, 2018, at 5:09 PM, Bolke de Bruin wrote:
> > >>>> Hi All,
> > >>>>
> > >>>> I have created a sdist package that is available at:
> > >>>>
> > >>>>
> > >> http://people.apache.org/~bolke/apache-airflow-1.10.0b3+incu
> > bating.tar.gz
> > >>>> <
> > >> http://people.apache.org/~bolke/apache-airflow-1.10.0b3+incu
> > bating.tar.gz>>
> > >>
> > >>>> In order to distinguish it from an actual (apache) release it is:
> > >>>>
> > >>>> 1. Marked as beta (python package managers do not install beta
> > >>>> versions by default - PEP 440)> 2. It is not signed
> > >>>> 3. It is not at an official apache distribution location
> > >>>>
> > >>>> You can also put something like this in a requirements.txt file:
> > >>>>
> > >>>> git+
> > >>>>
> > >> https://github.com/apache/incubator-airflow@v1-10-test#egg=
> > apache-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,
> > postgres,redis,slack,s3
> > >>>> <
> > >> https://github.com/apache/incubator-airflow@v1-10-test#egg=
> > apache-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,
> > postgres,redis,slack,s3
> > >>>
> > >>>> ]> <https://github.com/rodrigc/incubator-airflow@master#egg=apache-
> > >>>> airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,redi
> > s,slack,s-
> > >>>> 3]
> > >>>> <
> > >> https://github.com/rodrigc/incubator-airflow@master#egg=apac
> > he-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,
> > redis,slack,s3][1]
> > >>>
> > >>>>>>
> > >>>> and then "pip install -r requirements.txt”.
> > >>>>
> > >>>> I hope that after this beta we can go to RC and start voting on
> 1.10.>
> > >>>> Cheers
> > >>>> Bolke
> > >>>
> > >>>
> > >>> Links:
> > >>>
> > >>> 1.
> > >> https://github.com/rodrigc/incubator-airflow@master#egg=apac
> > he-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,
> > redis,slack,s3]%20%3Chttps://github.com/rodrigc/incubator-
> > airflow@master#egg=apache-airflow[celery,crypto,emr,
> > hive,hdfs,ldap,mysql,postgres,redis,slack,s3]
> > <https://github.com/rodrigc/incubator-airflow@master#egg=
> apache-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,
> postgres,redis,slack,s3]%20%3Chttps://github.com/rodrigc/
> incubator-airflow@master%23egg=apache-airflow[celery,
> crypto,emr,hive,hdfs,ldap,mysql,postgres,redis,slack,s3]>
> > >> <https://github.com/rodrigc/incubator-airflow@master#egg=apa
> > che-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,
> > redis,slack,s3]%20%3Chttps://github.com/rodrigc/incubator-
> > airflow@master%23egg=apache-airflow[celery,crypto,emr,
> > hive,hdfs,ldap,mysql,postgres,redis,slack,s3]>
> > >>
> >
> >
>
Re: Apache Airflow 1.10.0b3
Posted by Taylor Edmiston <te...@gmail.com>.
We discussed the splitting out of hooks & operators from core internally at
Astronomer again today. I should be able to allocate some time to
splitting out hooks & operators for the purpose of speeding up the CI. A
faster CI on core would be hugely beneficial for all contributors.
I created a Jira issue for this at https://issues.apache.org/
jira/browse/AIRFLOW-2732.
If anyone has pointers for this work, I'm all ears. I've done a lot with
Python unit tests in the past but this part of the Airflow codebase is new
to me.
Taylor
*Taylor Edmiston*
Blog <https://blog.tedmiston.com/> | CV
<https://stackoverflow.com/cv/taylor> | LinkedIn
<https://www.linkedin.com/in/tedmiston/> | AngelList
<https://angel.co/taylor> | Stack Overflow
<https://stackoverflow.com/users/149428/taylor-edmiston>
On Sun, Jul 1, 2018 at 6:02 AM, Bolke de Bruin <bd...@gmail.com> wrote:
> Separating the tests is where the effort lies. So having that as a
> consequence of splitting the packages would be nice. It has come up a
> couple of times but it was not picked up unfortunately.
>
> B.
>
> > On 28 Jun 2018, at 08:32, Maxime Beauchemin <ma...@gmail.com>
> wrote:
> >
> > It would be so nice to have a fast test suite. Having to wait for Travis
> > for up to an hour makes many workflows (like working on a release) super
> > painful.
> >
> > I spoke with folks at Astronomer recently about moving all operators and
> > hooks to another Python package that airflow would import. This would
> allow
> > for independent test suites and to have a more regular release cadence on
> > hooks and operators. What do you think?
> >
> > Max
> >
> > On Wed, Jun 27, 2018 at 11:18 PM Bolke de Bruin <bd...@gmail.com>
> wrote:
> >
> >> Arghhh. The downside of doing this late at night and wanting to go to
> >> bed... :-). Will make a new one
> >>
> >> Sent from my iPhone
> >>
> >>> On 28 Jun 2018, at 00:07, Chris Fei <cf...@gmail.com> wrote:
> >>>
> >>> Great, thank you! I just took this for a quick spin and it looks like
> >>> there's DB migration task missing. The task you committed just
> recently,
> >>> 9635ae0956e7_index_faskfail.py, has a down_revision of 856955da8476
> >>> which can't be found when running airflow initdb (seehttps://
> >> github.com/apache/incubator-airflow/tree/v1-10-test/airflow/
> migrations/versions
> >> ).
> >>> Chris
> >>>
> >>>
> >>>> On Wed, Jun 27, 2018, at 5:09 PM, Bolke de Bruin wrote:
> >>>> Hi All,
> >>>>
> >>>> I have created a sdist package that is available at:
> >>>>
> >>>>
> >> http://people.apache.org/~bolke/apache-airflow-1.10.0b3+incu
> bating.tar.gz
> >>>> <
> >> http://people.apache.org/~bolke/apache-airflow-1.10.0b3+incu
> bating.tar.gz>>
> >>
> >>>> In order to distinguish it from an actual (apache) release it is:
> >>>>
> >>>> 1. Marked as beta (python package managers do not install beta
> >>>> versions by default - PEP 440)> 2. It is not signed
> >>>> 3. It is not at an official apache distribution location
> >>>>
> >>>> You can also put something like this in a requirements.txt file:
> >>>>
> >>>> git+
> >>>>
> >> https://github.com/apache/incubator-airflow@v1-10-test#egg=
> apache-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,
> postgres,redis,slack,s3
> >>>> <
> >> https://github.com/apache/incubator-airflow@v1-10-test#egg=
> apache-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,
> postgres,redis,slack,s3
> >>>
> >>>> ]> <https://github.com/rodrigc/incubator-airflow@master#egg=apache-
> >>>> airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,redi
> s,slack,s-
> >>>> 3]
> >>>> <
> >> https://github.com/rodrigc/incubator-airflow@master#egg=apac
> he-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,
> redis,slack,s3][1]
> >>>
> >>>>>>
> >>>> and then "pip install -r requirements.txt”.
> >>>>
> >>>> I hope that after this beta we can go to RC and start voting on 1.10.>
> >>>> Cheers
> >>>> Bolke
> >>>
> >>>
> >>> Links:
> >>>
> >>> 1.
> >> https://github.com/rodrigc/incubator-airflow@master#egg=apac
> he-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,
> redis,slack,s3]%20%3Chttps://github.com/rodrigc/incubator-
> airflow@master#egg=apache-airflow[celery,crypto,emr,
> hive,hdfs,ldap,mysql,postgres,redis,slack,s3]
> <https://github.com/rodrigc/incubator-airflow@master#egg=apache-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,redis,slack,s3]%20%3Chttps://github.com/rodrigc/incubator-airflow@master%23egg=apache-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,redis,slack,s3]>
> >> <https://github.com/rodrigc/incubator-airflow@master#egg=apa
> che-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,
> redis,slack,s3]%20%3Chttps://github.com/rodrigc/incubator-
> airflow@master%23egg=apache-airflow[celery,crypto,emr,
> hive,hdfs,ldap,mysql,postgres,redis,slack,s3]>
> >>
>
>