You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/05/20 22:08:28 UTC

[GitHub] [airflow] ashb opened a new pull request #8938: Hive/Hadoop minicluster needs JDK8 and JAVA_HOME to work

ashb opened a new pull request #8938:
URL: https://github.com/apache/airflow/pull/8938


   Debian Buster only ships with a JDK11, and Hive/Hadoop fails in odd,
   hard to debug ways (complains about metastore not being initalized,
   possibly related to the class loader issues.)
   
   Until we rip Hive out from the CI (replacing it with Hadoop in a seprate
   integration, only on for some builds) we'll have to stick with JRE8
   
   Our previous approach of installing openjdk-8 from Sid/Unstable started
   failing as Debian Sid has a new (and conflicting) version of GCC/libc.
   The adoptopenjdk package archive is designed for Buster so should be
   more resilient
   
   ---
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x] Description above provides context of the change
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Target Github ISSUE in description if exists
   - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information.
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk merged pull request #8938: Hive/Hadoop minicluster needs JDK8 and JAVA_HOME to work

Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #8938:
URL: https://github.com/apache/airflow/pull/8938


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk edited a comment on pull request #8938: Hive/Hadoop minicluster needs JDK8 and JAVA_HOME to work

Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #8938:
URL: https://github.com/apache/airflow/pull/8938#issuecomment-631888486


   Thanks @ashb ! 
   
   I merged it now - even though one of the tests failed (but It looked like a transient error). Once the master build for this one succeeds, the case in the github registry will get updated. 
   
   Again Python buster image change with some dependency update made us suffer. I am rebasing
    and adding my python caching change shortly to avoid this in the future. Looking at it now.
   
   And yes - ripping out hive and minicluster is quite high on the priority list. There are event those issues grouped under #8783 #8784 and #8785 to do that ...


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #8938: Hive/Hadoop minicluster needs JDK8 and JAVA_HOME to work

Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #8938:
URL: https://github.com/apache/airflow/pull/8938#issuecomment-631888486


   Thanks @ashb ! 
   
   I merged it now - even though one of the tests failed (but It looked like a transient error). Once the master build for this one succeeds, the case in the github registry will get updated. 
   
   Again Python buster image change with some dependency update mad us suffer. I am reading and adding my python caching change shortly to avoid this in the future. Looking at it now.
   
   And yes - ripping out hive and minicluster is quite high on the priority list. There are event those issues grouped under #8783 #8784 and #8785 to do that ...


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk edited a comment on pull request #8938: Hive/Hadoop minicluster needs JDK8 and JAVA_HOME to work

Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #8938:
URL: https://github.com/apache/airflow/pull/8938#issuecomment-631888486


   Thanks @ashb ! 
   
   I merged it now - even though one of the tests failed (but It looked like a transient error). Once the master build for this one succeeds, the case in the github registry will get updated. 
   
   Again Python buster image change with some dependency update made us suffer. I am reading and adding my python caching change shortly to avoid this in the future. Looking at it now.
   
   And yes - ripping out hive and minicluster is quite high on the priority list. There are event those issues grouped under #8783 #8784 and #8785 to do that ...


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org