You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "potiuk (via GitHub)" <gi...@apache.org> on 2023/02/12 23:26:55 UTC
[GitHub] [airflow] potiuk opened a new pull request, #29500: Move libjars parameter in Sqoop Hook to Hook parameter
potiuk opened a new pull request, #29500:
URL: https://github.com/apache/airflow/pull/29500
The libjars parameter is not needed to be configured in configuration, it should be implemented in Hook (and Operator transitively).
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #29500: Move libjars parameter in Sqoop Hook to Hook parameter
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #29500:
URL: https://github.com/apache/airflow/pull/29500#issuecomment-1433066103
> Looks fine to me. Up to you if you feel `files` and `archives` parameters should get the same treatment.
Yeah. Just a minimum set of changes for this one - we can always separate them outside.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] josh-fell commented on a diff in pull request #29500: Move libjars parameter in Sqoop Hook to Hook parameter
Posted by "josh-fell (via GitHub)" <gi...@apache.org>.
josh-fell commented on code in PR #29500:
URL: https://github.com/apache/airflow/pull/29500#discussion_r1105194435
##########
airflow/providers/apache/sqoop/hooks/sqoop.py:
##########
@@ -61,14 +61,15 @@ def __init__(
hcatalog_database: str | None = None,
hcatalog_table: str | None = None,
properties: dict[str, Any] | None = None,
+ libjars: str | None = None,
Review Comment:
Maybe this idea to extend to `files` and `archives` too seeing as they are ["not typically used with Sqoop, but they are included as part of Hadoop’s internal argument-parsing system."](https://sqoop.apache.org/docs/1.4.2/SqoopUserGuide.html#_using_generic_and_specific_arguments)?
##########
airflow/providers/apache/sqoop/hooks/sqoop.py:
##########
@@ -46,6 +45,7 @@ class SqoopHook(BaseHook):
:param verbose: Set sqoop to verbose.
:param num_mappers: Number of map tasks to import in parallel.
:param properties: Properties to set via the -D argument
+ :param libjars: Optional Comma separated jar files to include in the classpath.
Review Comment:
```suggestion
:param libjars: Optional comma separated jar files to include in the classpath.
```
Nit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on pull request #29500: Move libjars parameter in Sqoop Hook to Hook parameter
Posted by "dstandish (via GitHub)" <gi...@apache.org>.
dstandish commented on PR #29500:
URL: https://github.com/apache/airflow/pull/29500#issuecomment-1432488524
I don't understand why this change. What's wrong with configuring in connection?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #29500: Move libjars parameter in Sqoop Hook to Hook parameter
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #29500:
URL: https://github.com/apache/airflow/pull/29500#issuecomment-1433064289
> I don't understand why this change. What's wrong with configuring in connection? Like it's fine to add as hook param but, why not allow continued configuration via airflow connection?
Just for consistency with all the other similar approaches, any system level configuration like that is better to be controlled by the operator's parameter rather than configuration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #29500: Move libjars parameter in Sqoop Hook to Hook parameter
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #29500:
URL: https://github.com/apache/airflow/pull/29500
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org