You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@camel.apache.org by "ank19 (via GitHub)" <gi...@apache.org> on 2023/03/16 18:33:46 UTC
[GitHub] [camel] ank19 opened a new pull request, #9567: Azure Data Lake - CAMEL-19152 additional file client header field + CAMEL-18636 additional auth options
ank19 opened a new pull request, #9567:
URL: https://github.com/apache/camel/pull/9567
# Description
- see CAMEL-19152 for a detailed description
- patched validateConfiguration() method in the component to accept string based configuration of auth parameters in addition to Java objects
- Added SAS token-based authentication as well as the usage of the default identity if enabled (disabled by default, prioritised if enabled)
# Target
- [x] I checked that the commit is targeting the correct branch (note that Camel 3 uses `camel-3.x`, whereas Camel 4 uses the `main` branch)
# Tracking
- [x] If this is a large change, bug fix, or code improvement, I checked there is a [JIRA issue](https://issues.apache.org/jira/browse/CAMEL) filed for the change (usually before you start working on it).
<!--
# *Note*: trivial changes like, typos, minor documentation fixes and other small items do not require a JIRA issue. In this case your pull request should address just this issue, without pulling in other changes.
-->
# Apache Camel coding standards and style
- [x] I checked that each commit in the pull request has a meaningful subject line and body.
<!--
If you're unsure, you can format the pull request title like `[CAMEL-XXX] Fixes bug in camel-file component`, where you replace `CAMEL-XXX` with the appropriate JIRA issue.
-->
- [x] I formatted the code using `mvn -Pformat,fastinstall install && mvn -Psourcecheck`
<!--
You can run the aforementioned command in your module so that the build auto-formats your code and the source check verifies that is complies with our coding style. This will also be verified as part of the checks and your PR may be rejected if the checkstyle does not pass.
You can learn more about the contribution guidelines at https://github.com/apache/camel/blob/main/CONTRIBUTING.md
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel] oscerd commented on pull request #9567: Azure Data Lake - CAMEL-19152 additional file client header field + CAMEL-18636 additional auth options
Posted by "oscerd (via GitHub)" <gi...@apache.org>.
oscerd commented on PR #9567:
URL: https://github.com/apache/camel/pull/9567#issuecomment-1473426536
@ank19 this has conflicts because I completely removed the validateConfiguration. To me it's better to notice errors than having to deal with validation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel] ank19 commented on pull request #9567: Azure Data Lake - CAMEL-19152 additional file client header field + CAMEL-18636 additional auth options
Posted by "ank19 (via GitHub)" <gi...@apache.org>.
ank19 commented on PR #9567:
URL: https://github.com/apache/camel/pull/9567#issuecomment-1473862724
@oscerd - You're welcome. Concerning the rebase: I removed the validation method, rebased the commit and run format and the source check; I even tried to run the same check style command as in the action above, but it still failing. Do you have an idea why that is the case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel] oscerd commented on pull request #9567: Azure Data Lake - CAMEL-19152 additional file client header field + CAMEL-18636 additional auth options
Posted by "oscerd (via GitHub)" <gi...@apache.org>.
oscerd commented on PR #9567:
URL: https://github.com/apache/camel/pull/9567#issuecomment-1473405748
Can you please rebase? @ank19
Can you please also share how are you setting up your data lake instance? Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel] ank19 commented on pull request #9567: Azure Data Lake - CAMEL-19152 additional file client header field + CAMEL-18636 additional auth options
Posted by "ank19 (via GitHub)" <gi...@apache.org>.
ank19 commented on PR #9567:
URL: https://github.com/apache/camel/pull/9567#issuecomment-1475709547
@oscerd - Hm, can you give me a hint how to detect the conflict? I get no indication that there is one...
<img width="945" alt="image" src="https://user-images.githubusercontent.com/10431940/226267569-f35b9eb2-9bc5-4d3f-8550-3e509f0376d8.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel] oscerd commented on pull request #9567: Azure Data Lake - CAMEL-19152 additional file client header field + CAMEL-18636 additional auth options
Posted by "oscerd (via GitHub)" <gi...@apache.org>.
oscerd commented on PR #9567:
URL: https://github.com/apache/camel/pull/9567#issuecomment-1475712282
Can you also backport to main? Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel] oscerd merged pull request #9567: Azure Data Lake - CAMEL-19152 additional file client header field + CAMEL-18636 additional auth options
Posted by "oscerd (via GitHub)" <gi...@apache.org>.
oscerd merged PR #9567:
URL: https://github.com/apache/camel/pull/9567
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel] oscerd commented on pull request #9567: Azure Data Lake - CAMEL-19152 additional file client header field + CAMEL-18636 additional auth options
Posted by "oscerd (via GitHub)" <gi...@apache.org>.
oscerd commented on PR #9567:
URL: https://github.com/apache/camel/pull/9567#issuecomment-1473875431
Not your fault:
```
[ERROR] Failed to execute goal on project camel-dhis2: Could not resolve dependencies for project org.apache.camel:camel-dhis2:jar:3.21.0-SNAPSHOT: Could not find artifact org.apache.camel:camel-dhis2-api:jar:3.21.0-SNAPSHOT in apache.snapshots (https://repository.apache.org/snapshots/) -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal on project camel-dhis2: Could not resolve dependencies for project org.apache.camel:camel-dhis2:jar:3.21.0-SNAPSHOT: Could not find artifact org.apache.camel:camel-dhis2-api:jar:3.21.0-SNAPSHOT in apache.snapshots (https://repository.apache.org/snapshots/)
at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.getDependencies(LifecycleDependencyResolver.java:269)
at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.resolveProjectDependencies(LifecycleDependencyResolver.java:147)
at org.apache.maven.lifecycle.internal.MojoExecutor.ensureDependenciesAreResolved(MojoExecutor.java:401)
at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute(MojoExecutor.java:347)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:215)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:171)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:163)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:117)
at org.mvndaemon.mvnd.builder.SmartBuilderImpl.buildProject(SmartBuilderImpl.java:178)
at org.mvndaemon.mvnd.builder.SmartBuilderImpl$ProjectBuildTask.run(SmartBuilderImpl.java:198)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.apache.maven.project.DependencyResolutionException: Could not resolve dependencies for project org.apache.camel:camel-dhis2:jar:3.21.0-SNAPSHOT: Could not find artifact org.apache.camel:camel-dhis2-api:jar:3.21.0-SNAPSHOT in apache.snapshots (https://repository.apache.org/snapshots/)
at org.apache.maven.project.DefaultProjectDependenciesResolver.resolve(DefaultProjectDependenciesResolver.java:214)
at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.getDependencies(LifecycleDependencyResolver.java:243)
... 14 common frames omitted
Caused by: org.eclipse.aether.resolution.DependencyResolutionException: Could not find artifact org.apache.camel:camel-dhis2-api:jar:3.21.0-SNAPSHOT in apache.snapshots (https://repository.apache.org/snapshots/)
at org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveDependencies(DefaultRepositorySystem.java:374)
at org.apache.maven.project.DefaultProjectDependenciesResolver.resolve(DefaultProjectDependenciesResolver.java:207)
... 15 common frames omitted
Caused by: org.eclipse.aether.resolution.ArtifactResolutionException: Could not find artifact org.apache.camel:camel-dhis2-api:jar:3.21.0-SNAPSHOT in apache.snapshots (https://repository.apache.org/snapshots/)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:431)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifacts(DefaultArtifactResolver.java:235)
at org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveDependencies(DefaultRepositorySystem.java:357)
... 16 common frames omitted
Caused by: org.eclipse.aether.transfer.ArtifactNotFoundException: Could not find artifact org.apache.camel:camel-dhis2-api:jar:3.21.0-SNAPSHOT in apache.snapshots (https://repository.apache.org/snapshots/)
at org.eclipse.aether.connector.basic.ArtifactTransportListener.transferFailed(ArtifactTransportListener.java:48)
at org.eclipse.aether.connector.basic.BasicRepositoryConnector$TaskRunner.run(BasicRepositoryConnector.java:369)
at org.eclipse.aether.util.concurrency.RunnableErrorForwarder.lambda$wrap$0(RunnableErrorForwarder.java:73)
at org.eclipse.aether.connector.basic.BasicRepositoryConnector$DirectExecutor.execute(BasicRepositoryConnector.java:627)
at org.eclipse.aether.connector.basic.BasicRepositoryConnector.get(BasicRepositoryConnector.java:262)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.performDownloads(DefaultArtifactResolver.java:520)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:408)
... 18 common frames omitted
```
We need to set up a deploy action to always have the 3.21.0 in sync
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel] oscerd commented on pull request #9567: Azure Data Lake - CAMEL-19152 additional file client header field + CAMEL-18636 additional auth options
Posted by "oscerd (via GitHub)" <gi...@apache.org>.
oscerd commented on PR #9567:
URL: https://github.com/apache/camel/pull/9567#issuecomment-1473908677
BTW we still have conflicts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel] oscerd commented on pull request #9567: Azure Data Lake - CAMEL-19152 additional file client header field + CAMEL-18636 additional auth options
Posted by "oscerd (via GitHub)" <gi...@apache.org>.
oscerd commented on PR #9567:
URL: https://github.com/apache/camel/pull/9567#issuecomment-1475711039
My fault, i didn't reload the PR page. It's good to go now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel] github-actions[bot] commented on pull request #9567: Azure Data Lake - CAMEL-19152 additional file client header field + CAMEL-18636 additional auth options
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #9567:
URL: https://github.com/apache/camel/pull/9567#issuecomment-1472537191
:star2: Thank you for your contribution to the Apache Camel project! :star2:
:warning: Please note that the changes on this PR may be **tested automatically**.
If necessary Apache Camel Committers may access logs and test results in the job summaries!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel] oscerd commented on pull request #9567: Azure Data Lake - CAMEL-19152 additional file client header field + CAMEL-18636 additional auth options
Posted by "oscerd (via GitHub)" <gi...@apache.org>.
oscerd commented on PR #9567:
URL: https://github.com/apache/camel/pull/9567#issuecomment-1473866226
I think it's something not related to your PR. Once the build will end I'll access the log and let you know.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel] github-actions[bot] commented on pull request #9567: Azure Data Lake - CAMEL-19152 additional file client header field + CAMEL-18636 additional auth options
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #9567:
URL: https://github.com/apache/camel/pull/9567#issuecomment-1473873032
### Components tested:
| Total | Tested | Failed :x: | Passed :white_check_mark: |
| --- | --- | --- | --- |
| 1 | 1 | 0 | 1 |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel] oscerd commented on pull request #9567: Azure Data Lake - CAMEL-19152 additional file client header field + CAMEL-18636 additional auth options
Posted by "oscerd (via GitHub)" <gi...@apache.org>.
oscerd commented on PR #9567:
URL: https://github.com/apache/camel/pull/9567#issuecomment-1473661148
Thanks a lot for the point 3, because I was probably missing some steps. Thanks a lot! I'll try to that to create a Kamelets for the component. Thanks again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel] ank19 commented on a diff in pull request #9567: Azure Data Lake - CAMEL-19152 additional file client header field + CAMEL-18636 additional auth options
Posted by "ank19 (via GitHub)" <gi...@apache.org>.
ank19 commented on code in PR #9567:
URL: https://github.com/apache/camel/pull/9567#discussion_r1139905888
##########
components/camel-azure/camel-azure-storage-datalake/src/main/java/org/apache/camel/component/azure/storage/datalake/DataLakeConfiguration.java:
##########
@@ -93,6 +94,12 @@ public class DataLakeConfiguration implements Cloneable {
private String umask;
@UriParam(description = "set open options for creating file")
private Set<OpenOption> openOptions;
+ @UriParam(description = "SAS token signature")
Review Comment:
Sure, added it...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel] ank19 commented on pull request #9567: Azure Data Lake - CAMEL-19152 additional file client header field + CAMEL-18636 additional auth options
Posted by "ank19 (via GitHub)" <gi...@apache.org>.
ank19 commented on PR #9567:
URL: https://github.com/apache/camel/pull/9567#issuecomment-1475762100
@oscerd - Sure, just created a PR: https://github.com/apache/camel/pull/9577
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel] davsclaus commented on a diff in pull request #9567: Azure Data Lake - CAMEL-19152 additional file client header field + CAMEL-18636 additional auth options
Posted by "davsclaus (via GitHub)" <gi...@apache.org>.
davsclaus commented on code in PR #9567:
URL: https://github.com/apache/camel/pull/9567#discussion_r1139270513
##########
components/camel-azure/camel-azure-storage-datalake/src/main/java/org/apache/camel/component/azure/storage/datalake/DataLakeConfiguration.java:
##########
@@ -93,6 +94,12 @@ public class DataLakeConfiguration implements Cloneable {
private String umask;
@UriParam(description = "set open options for creating file")
private Set<OpenOption> openOptions;
+ @UriParam(description = "SAS token signature")
Review Comment:
Can you add label = "security" for these new options so we know they are related to security stuff
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel] ank19 commented on pull request #9567: Azure Data Lake - CAMEL-19152 additional file client header field + CAMEL-18636 additional auth options
Posted by "ank19 (via GitHub)" <gi...@apache.org>.
ank19 commented on PR #9567:
URL: https://github.com/apache/camel/pull/9567#issuecomment-1473658034
@oscerd - Sure, for me it's fine to drop the validation method, I'm just not sure what is "common" or "usual" in the project. I'm removing it and updating the PR accordingly.
Concerning the Data Lake setup, what I did is basically creating a standard Azure storage account, then, on the overview page, setting Hierarchical namespace to "enabled", which starts that sort of a migration process to Data Lake Gen2. Then I tried three authentication methods for creation of a new file and appending data to that file, specifying an account name and:
1. Within "Security + networking / Shared access signature": Generate a new SAS with the appropriate permissions and constraints set, then using the "Signature" to set the "sasSignature" as configuration parameter on the component (the thing starting with "?...").
2. Within "Security + networking / Access keys": Using one of the two provided tokens to set "accountKey".
3. Creating an app registration on Azure AD, and then under "Certificates & secrets", creating a new client secret, using the "Value" from that page as "clientSecret", and from the overview page the "Application (client) id" as "clientId" as well as "Directory (tenant) id" as "tenantId". Afterwards, I configured on the storage account, in "Access Control (IAM)", that the app registration just created gets the role "Contributor" for instance. Afterwards I used the Microsoft Storage Explorer to "Propagate Access Control Lists..." on a newly created container, so that the ACLs get applied.
I created a Zulip account - If you like you can try to contact me there, too. However, I might be online from time to time only.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org