You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2020/02/28 21:08:17 UTC
[GitHub] [druid] mcbrewster opened a new pull request #9437: Add Azure
ingestion flow to web console
mcbrewster opened a new pull request #9437: Add Azure ingestion flow to web console
URL: https://github.com/apache/druid/pull/9437
<img width="1162" alt="Screen Shot 2020-02-28 at 1 03 39 PM" src="https://user-images.githubusercontent.com/37322608/75587377-2f302780-5a2b-11ea-8eb1-c16f6a52a7d5.png">
<img width="1159" alt="Screen Shot 2020-02-28 at 1 03 30 PM" src="https://user-images.githubusercontent.com/37322608/75587415-45d67e80-5a2b-11ea-9410-fc2fa1266033.png">
Adds the ability to ingest data from azure via the web console. There is now a tile that allows the user to access the Azure ingestion flow if druid-azure-extensions is in the load list. The default input type will be azure. The user can then put a can then input data using A URI or a Prefix. the resulting IO config changes will be :
```
"ioConfig": {
"type": "index_parallel",
"inputSource": {
"type": "azure",
"prefixes": [
"azure://containerName"
]
}
},
```
or
```
"ioConfig": {
"type": "index_parallel",
"inputSource": {
"type": "azure",
"prefixes": [
"azure://container-name/path-name.json"
]
}
},
```
The user can configure the rest of the spec using the normal web console flow.
Additionally
azure is now available as an input type in the input menu for all flows
<img width="342" alt="Screen Shot 2020-02-28 at 1 05 59 PM" src="https://user-images.githubusercontent.com/37322608/75587357-20e20b80-5a2b-11ea-92b6-cbec6ecccee5.png">
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] vogievetsky merged pull request #9437: Add Azure ingestion
flow to web console
Posted by GitBox <gi...@apache.org>.
vogievetsky merged pull request #9437: Add Azure ingestion flow to web console
URL: https://github.com/apache/druid/pull/9437
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] mcbrewster commented on issue #9437: Add Azure ingestion
flow to web console
Posted by GitBox <gi...@apache.org>.
mcbrewster commented on issue #9437: Add Azure ingestion flow to web console
URL: https://github.com/apache/druid/pull/9437#issuecomment-592852655
I just wasn’t sure if zachs stuff has been merged
Get Outlook for iOS<https://aka.ms/o0ukef>
________________________________
From: Vadim Ogievetsky <no...@github.com>
Sent: Friday, February 28, 2020 7:47:24 PM
To: apache/druid <dr...@noreply.github.com>
Cc: Maggie Brewster <mc...@edu.uwaterloo.ca>; Author <au...@noreply.github.com>
Subject: Re: [apache/druid] Add Azure ingestion flow to web console (#9437)
Please do not merge until support for azure has been merged
Can you indicate what this is blocked on?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub<https://github.com/apache/druid/pull/9437?email_source=notifications&email_token=AI4X64HHWD4XZGVFMMEC4FTRFHLEZA5CNFSM4K55KZIKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENLC3DQ#issuecomment-592850318>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AI4X64DNID2UOBXZEVIUARDRFHLEZANCNFSM4K55KZIA>.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] vogievetsky commented on issue #9437: Add Azure ingestion
flow to web console
Posted by GitBox <gi...@apache.org>.
vogievetsky commented on issue #9437: Add Azure ingestion flow to web console
URL: https://github.com/apache/druid/pull/9437#issuecomment-594115402
Looks great thank you 👍
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] vogievetsky commented on a change in pull request #9437:
Add Azure ingestion flow to web console
Posted by GitBox <gi...@apache.org>.
vogievetsky commented on a change in pull request #9437: Add Azure ingestion flow to web console
URL: https://github.com/apache/druid/pull/9437#discussion_r385998507
##########
File path: web-console/src/utils/ingestion-spec.tsx
##########
@@ -2615,6 +2687,10 @@ export function upgradeSpec(spec: any): any {
deepSet(spec, 'ioConfig.firehose.type', 's3');
break;
+ case 'static-azure':
+ deepSet(spec, 'ioConfig.firehose.type', 'azure');
+ break;
Review comment:
There is no need for this upgrade spec clause, there was never a `'static-azure'`. This is back compat code but Azure ingestion did not used to exist. You can just remove this change.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] vogievetsky commented on a change in pull request #9437:
Add Azure ingestion flow to web console
Posted by GitBox <gi...@apache.org>.
vogievetsky commented on a change in pull request #9437: Add Azure ingestion flow to web console
URL: https://github.com/apache/druid/pull/9437#discussion_r385998538
##########
File path: web-console/src/utils/ingestion-spec.tsx
##########
@@ -2653,6 +2729,10 @@ export function downgradeSpec(spec: any): any {
deepSet(spec, 'ioConfig.firehose.type', 'static-s3');
break;
+ case 'azure':
+ deepSet(spec, 'ioConfig.firehose.type', 'static-azure');
+ break;
Review comment:
Same story as above.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] vogievetsky commented on issue #9437: Add Azure ingestion
flow to web console
Posted by GitBox <gi...@apache.org>.
vogievetsky commented on issue #9437: Add Azure ingestion flow to web console
URL: https://github.com/apache/druid/pull/9437#issuecomment-592850318
> Please do not merge until support for azure has been merged
Can you indicate what this is blocked on?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] vogievetsky commented on issue #9437: Add Azure ingestion
flow to web console
Posted by GitBox <gi...@apache.org>.
vogievetsky commented on issue #9437: Add Azure ingestion flow to web console
URL: https://github.com/apache/druid/pull/9437#issuecomment-592853427
I think you are referring to https://github.com/apache/druid/pull/9394 and it is merged!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org