You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@calcite.apache.org by Gonzalo Ortiz Jaureguizar <go...@gmail.com> on 2024/07/19 13:28:58 UTC
Question about Project.getPartialMapping
Hi there,
Fixing an issue in Apache Pinot I've found something that may be a bug in
Apache Calcite, although probably is a misunderstanding from my side.
In Pinot we are not (yet) using the standard MetadataHandler (which I'm
having issues to understand, but that is not the topic here) to calculate
the distribution of each node. Instead, we add our own distribution in a
trait and process that by ourselves in our own rules. That may not be the
best in terms of DRY, but it is working.
We have found an issue when using projections that are not mapping.
Probably a more expert eye will see in PinotRelDistributionTraitRule#L113
<http://PinotRelDistributionTraitRule.java#L113> that our naive way to
calculate the new distribution fails in that case
because project.getMapping() returns null.
I planned to solve that by using project.getPartialMapping() in the same
way done by Calcite in RelMdDistribution#L165
<https://github.com/apache/calcite/blob/e371b336a8b404ed36955f517196a5e8606455d7/core/src/main/java/org/apache/calcite/rel/metadata/RelMdDistribution.java#L165C17-L165C34>,
but when I tried to do that, I found the following runtime exception:
java.lang.UnsupportedOperationException
at
org.apache.calcite.util.mapping.Mappings$AbstractMapping.getSourceOpt(Mappings.java:952)
at
org.apache.calcite.util.mapping.Mappings$InverseMapping.getTargetOpt(Mappings.java:1841)
at
org.apache.calcite.rel.RelDistributions$RelDistributionImpl.apply(RelDistributions.java:146)
at
org.apache.pinot.calcite.rel.rules.PinotRelDistributionTraitRule.deriveDistribution(PinotRelDistributionTraitRule.java:128)
Which makes me think there is a problem in the way
Project.getPartialMapping is initializing the Mapping. My solution was to
do something like:
Mappings.TargetMapping mapping =
Project.getPartialMapping(input.getRowType().getFieldCount(),
project.getProjects());
Mapping actualMapping =
Mappings.create(MappingType.PARTIAL_FUNCTION,
input.getRowType().getFieldCount(),
project.getRowType().getFieldCount());
for (IntPair intPair : mapping) {
actualMapping.set(intPair.source, intPair.target);
}
return inputRelDistribution.apply(actualMapping);
But I don't think that is the most elegant solution. I think it is fair to
assume that the Mapping returned by Project.getPartialMapping should be
usable to call getTargetOpt.
Is there something I am doing or understanding wrong?
Re: Question about Project.getPartialMapping
Posted by Julian Hyde <jh...@gmail.com>.
See https://issues.apache.org/jira/browse/CALCITE-4166.
> On Jul 19, 2024, at 6:28 AM, Gonzalo Ortiz Jaureguizar <go...@gmail.com> wrote:
>
> Hi there,
>
> Fixing an issue in Apache Pinot I've found something that may be a bug in
> Apache Calcite, although probably is a misunderstanding from my side.
>
> In Pinot we are not (yet) using the standard MetadataHandler (which I'm
> having issues to understand, but that is not the topic here) to calculate
> the distribution of each node. Instead, we add our own distribution in a
> trait and process that by ourselves in our own rules. That may not be the
> best in terms of DRY, but it is working.
>
> We have found an issue when using projections that are not mapping.
> Probably a more expert eye will see in PinotRelDistributionTraitRule#L113
> <http://PinotRelDistributionTraitRule.java#L113> that our naive way to
> calculate the new distribution fails in that case
> because project.getMapping() returns null.
>
> I planned to solve that by using project.getPartialMapping() in the same
> way done by Calcite in RelMdDistribution#L165
> <https://github.com/apache/calcite/blob/e371b336a8b404ed36955f517196a5e8606455d7/core/src/main/java/org/apache/calcite/rel/metadata/RelMdDistribution.java#L165C17-L165C34>,
> but when I tried to do that, I found the following runtime exception:
>
> java.lang.UnsupportedOperationException
> at
> org.apache.calcite.util.mapping.Mappings$AbstractMapping.getSourceOpt(Mappings.java:952)
> at
> org.apache.calcite.util.mapping.Mappings$InverseMapping.getTargetOpt(Mappings.java:1841)
> at
> org.apache.calcite.rel.RelDistributions$RelDistributionImpl.apply(RelDistributions.java:146)
> at
> org.apache.pinot.calcite.rel.rules.PinotRelDistributionTraitRule.deriveDistribution(PinotRelDistributionTraitRule.java:128)
>
> Which makes me think there is a problem in the way
> Project.getPartialMapping is initializing the Mapping. My solution was to
> do something like:
>
> Mappings.TargetMapping mapping =
> Project.getPartialMapping(input.getRowType().getFieldCount(),
> project.getProjects());
> Mapping actualMapping =
> Mappings.create(MappingType.PARTIAL_FUNCTION,
> input.getRowType().getFieldCount(),
> project.getRowType().getFieldCount());
> for (IntPair intPair : mapping) {
> actualMapping.set(intPair.source, intPair.target);
> }
> return inputRelDistribution.apply(actualMapping);
>
> But I don't think that is the most elegant solution. I think it is fair to
> assume that the Mapping returned by Project.getPartialMapping should be
> usable to call getTargetOpt.
>
> Is there something I am doing or understanding wrong?