You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "Jibing-Li (via GitHub)" <gi...@apache.org> on 2023/03/12 15:04:31 UTC
[GitHub] [doris] Jibing-Li opened a new pull request, #17706: [Improvement](multi catalog)Hive splitter. Get HDFS/S3 splits by using FileSystem api
Jibing-Li opened a new pull request, #17706:
URL: https://github.com/apache/doris/pull/17706
Use FileSystem API to get splits for file in HDFS/S3 instead of calling `InputFormat.getSplits`.
The splits is based on blocks in HDFS/S3.
## Checklist(Required)
* [ ] Does it affect the original behavior
* [ ] Has unit tests been added
* [ ] Has document been added or modified
* [ ] Does it need to update dependencies
* [ ] Is this PR support rollback (If NO, please explain WHY)
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] Jibing-Li commented on pull request #17706: [Improvement](multi catalog)Hive splitter. Get HDFS/S3 splits by using FileSystem api
Posted by "Jibing-Li (via GitHub)" <gi...@apache.org>.
Jibing-Li commented on PR #17706:
URL: https://github.com/apache/doris/pull/17706#issuecomment-1467761387
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morningman merged pull request #17706: [Improvement](multi catalog)Hive splitter. Get HDFS/S3 splits by using FileSystem api
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman merged PR #17706:
URL: https://github.com/apache/doris/pull/17706
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morningman commented on a diff in pull request #17706: [Improvement](multi catalog)Hive splitter. Get HDFS/S3 splits by using FileSystem api
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on code in PR #17706:
URL: https://github.com/apache/doris/pull/17706#discussion_r1134233541
##########
fe/fe-core/src/main/java/org/apache/doris/planner/external/HiveSplitter.java:
##########
@@ -152,4 +161,31 @@ public int getTotalPartitionNum() {
public int getReadPartitionNum() {
return readPartitionNum;
}
+
+ // Get splits by using FileSystem API, the splits are blocks in HDFS or S3 like storage system.
+ public static InputSplit[] getHiveSplits(Path path, InputFormat<?, ?> inputFormat,
+ JobConf jobConf) throws IOException {
+ FileSystem fs = path.getFileSystem(jobConf);
+ boolean splittable = HiveUtil.isSplittable(inputFormat, fs, path);
+ List<InputSplit> splits = Lists.newArrayList();
+ RemoteIterator<LocatedFileStatus> locatedFileStatusRemoteIterator = fs.listFiles(path, true);
+ if (!locatedFileStatusRemoteIterator.hasNext()) {
+ LOG.debug("File status for path {} is empty.", path);
+ return new InputSplit[0];
+ }
+ if (!splittable) {
+ LOG.debug("Path {} is not splittable.", path);
+ LocatedFileStatus status = locatedFileStatusRemoteIterator.next();
+ splits.add(new FileSplit(status.getPath(), 0, status.getLen(), status.getBlockLocations()[0].getHosts()));
+ return splits.toArray(new InputSplit[1]);
+ }
+ while (locatedFileStatusRemoteIterator.hasNext()) {
+ LocatedFileStatus status = locatedFileStatusRemoteIterator.next();
+ for (BlockLocation block : status.getBlockLocations()) {
+ splits.add(new FileSplit(status.getPath(), block.getOffset(), block.getLength(), block.getHosts()));
Review Comment:
This need to be cached
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morningman commented on a diff in pull request #17706: [Improvement](multi catalog)Hive splitter. Get HDFS/S3 splits by using FileSystem api
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on code in PR #17706:
URL: https://github.com/apache/doris/pull/17706#discussion_r1134229250
##########
fe/fe-common/src/main/java/org/apache/doris/common/Config.java:
##########
@@ -2057,7 +2057,14 @@ public class Config extends ConfigBase {
@ConfField(mutable = false, masterOnly = false)
public static String mysql_load_server_secure_path = "";
+
@ConfField(mutable = false, masterOnly = false)
public static int mysql_load_thread_pool = 4;
+
+ /**
+ * Temp config. Flag to set use InputFormat.getSplits or use the code in doris.
+ */
+ @ConfField(mutable = true)
+ public static boolean enable_self_splitter = false;
Review Comment:
It should be a property of catalog, not a config
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org