You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "gianm (via GitHub)" <gi...@apache.org> on 2023/07/27 18:50:59 UTC
[PR] Save a metadata call when reading files from CloudObjectInputSource. (druid)
gianm opened a new pull request, #14677:
URL: https://github.com/apache/druid/pull/14677
The call to createSplits(inputFormat, null) in formattableReader would use the default split hint spec, MaxSizeSplitHintSpec, which makes getObjectMetadata calls in order to compute its splits. This isn't necessary; we're just trying to unpack the files inside the input source.
To fix this, use FilePerSplitHintSpec to extract files without any funny business.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Save a metadata call when reading files from CloudObjectInputSource. (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on PR #14677:
URL: https://github.com/apache/druid/pull/14677#issuecomment-1655001380
Does this mean that for cloud input sources, if we don't specify a split hint spec explicitly, each file would correspond to a single split? Couldn't that potentially cause a large skew between the amount of work each batch ingestion task does?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Save a metadata call when reading files from CloudObjectInputSource. (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on PR #14677:
URL: https://github.com/apache/druid/pull/14677#issuecomment-1656119092
> Ah, no, because the code here runs on worker tasks after the splits have already been made by the main task. (Or on the Overlord to do sampling, where there is no parallelism anyway.) It is using createSplits simply as a way of getting the list of objects out of prefixes, objects, or uris.
Ah, I see. Thanks for the clarification!
> You can especially tell it doesn't matter because the splits are flatMap together anyway.
Yes, fair point.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Save a metadata call when reading files from CloudObjectInputSource. (druid)
Posted by "gianm (via GitHub)" <gi...@apache.org>.
gianm commented on PR #14677:
URL: https://github.com/apache/druid/pull/14677#issuecomment-1656047741
> Does this mean that for cloud input sources, if we don't specify a split hint spec explicitly, each file would correspond to a single split? Couldn't that potentially cause a large skew between the amount of work each batch ingestion task does if the files differ a lot in size?
Ah, no, because the code here runs on worker tasks after the splits have already been made by the main task. (Or on the Overlord to do sampling, where there is no parallelism anyway.) It is using `createSplits` simply as a way of getting the list of objects out of `prefixes`, `objects`, or `uris`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Save a metadata call when reading files from CloudObjectInputSource. (druid)
Posted by "gianm (via GitHub)" <gi...@apache.org>.
gianm merged PR #14677:
URL: https://github.com/apache/druid/pull/14677
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org