You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@gobblin.apache.org by "Urmi Mustafi (Jira)" <ji...@apache.org> on 2021/10/06 22:42:00 UTC
[jira] [Updated] (GOBBLIN-1559) Support wildcard for input paths
[ https://issues.apache.org/jira/browse/GOBBLIN-1559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Urmi Mustafi updated GOBBLIN-1559:
----------------------------------
Description:
Currently wildcard does not work with GaaS paths due to complications with path finding. See FSDatasetDescriptor.isPathContaining, if the path is a glob it will just return false. The reason for this is that it is not really possible to calculate if one path glob pattern "contains"/ is a subset of another glob pattern.
But for our self serve edges, we usually just accept any path by defining
{code:java}
gobblin.flow.edge.input.dataset.descriptor.0.path=${gobblin.flow.input.dataset.descriptor.path}{code}
We should be able to add a special case in this method, where glob patterns are accepted if the paths match exactly, and that should allow glob patterns as input paths for the self serve edges.
was:Glob patterns should be accepted if the paths match exactly, and that should allow glob patterns as input paths for the self serve edges.
> Support wildcard for input paths
> --------------------------------
>
> Key: GOBBLIN-1559
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1559
> Project: Apache Gobblin
> Issue Type: Improvement
> Components: gobblin-service
> Reporter: Urmi Mustafi
> Assignee: Abhishek Tiwari
> Priority: Major
>
> Currently wildcard does not work with GaaS paths due to complications with path finding. See FSDatasetDescriptor.isPathContaining, if the path is a glob it will just return false. The reason for this is that it is not really possible to calculate if one path glob pattern "contains"/ is a subset of another glob pattern.
> But for our self serve edges, we usually just accept any path by defining
> {code:java}
> gobblin.flow.edge.input.dataset.descriptor.0.path=${gobblin.flow.input.dataset.descriptor.path}{code}
> We should be able to add a special case in this method, where glob patterns are accepted if the paths match exactly, and that should allow glob patterns as input paths for the self serve edges.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)