You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2020/03/20 16:22:59 UTC
[GitHub] [druid] xvrl opened a new pull request #9546: support schemaless
ingestion for transformed dimensions
xvrl opened a new pull request #9546: support schemaless ingestion for transformed dimensions
URL: https://github.com/apache/druid/pull/9546
I might be missing some subtleties or edge cases, but I'm wondering if this is all it takes to support adding schemaless ingestion for dimensions created by transformspecs?
Of course this doesn't address the backwards incompatibility concerns. We can discuss whether we feel it is important to maintain the old behavior or not.
fixes #7952
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] jihoonson commented on issue #9546: support schemaless
ingestion for transformed dimensions
Posted by GitBox <gi...@apache.org>.
jihoonson commented on issue #9546: support schemaless ingestion for transformed dimensions
URL: https://github.com/apache/druid/pull/9546#issuecomment-605139493
The discussion in https://github.com/apache/druid/issues/3599 looks related.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] clintropolis commented on a change in pull request #9546:
support schemaless ingestion for transformed dimensions
Posted by GitBox <gi...@apache.org>.
clintropolis commented on a change in pull request #9546: support schemaless ingestion for transformed dimensions
URL: https://github.com/apache/druid/pull/9546#discussion_r399139991
##########
File path: processing/src/main/java/org/apache/druid/segment/transform/TransformSpec.java
##########
@@ -44,19 +44,23 @@
*/
public class TransformSpec
{
- public static final TransformSpec NONE = new TransformSpec(null, null);
+ public static final TransformSpec NONE = new TransformSpec(null, null, null);
private final DimFilter filter;
private final List<Transform> transforms;
+ private final List<String> addDimensions;
+
@JsonCreator
public TransformSpec(
@JsonProperty("filter") final DimFilter filter,
- @JsonProperty("transforms") final List<Transform> transforms
+ @JsonProperty("transforms") final List<Transform> transforms,
+ @JsonProperty("addDimensions") final List<String> addDimensions
Review comment:
This seems like it could work, without the downside of always including them as dimensions. The alternative I guess would be adding a `isDimension` to the `Transform` interface? I'm not sure which is better, though the way you've done it here is definitely less intrusive in terms of modifications.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] xvrl commented on issue #9546: support schemaless ingestion
for transformed dimensions
Posted by GitBox <gi...@apache.org>.
xvrl commented on issue #9546: support schemaless ingestion for transformed dimensions
URL: https://github.com/apache/druid/pull/9546#issuecomment-603379060
@clintropolis agree there are some quirks, it's a bit unfortunate that dimensionExclusions don't get applied after transforms, otherwise we could use that as a workaround. One option would be to pass in an additional field to the transform spec to indicate which field should be added as dimensions.
The behavior might still be odd since those new dimensions would still be added even if they are not part of the schema when one is explicitly defined. Maybe we can add this as an experimental/undocumented feature for now and see if we find a better solution long-term.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org