You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2020/06/06 01:02:20 UTC

[GitHub] [incubator-pinot] npawar opened a new issue #5509: Derived columns

npawar opened a new issue #5509:
URL: https://github.com/apache/incubator-pinot/issues/5509

Introduce derived columns - columns which are derived from other columns in the Pinot data record.
This is not the same as simply having transform functions. Transform functions are applied to source columns (x) to get pinot columns (y) i.e. `y = f(x)`
Derived columns are derived from pinot columns i.e. `y = f(y)`.

By default, the derived columns will be materialized during ingestion. We also want an option, by which we don't materialize them, instead execute the transform function during query time (An example of where this would be useful - populate a column based on current time)

Derived columns will also be useful in adding columns to existing data, if the column value can be derived from existing data. For instance, if we have column `foo`, and then add `bar` to the schema, if `bar = f(foo)`, we can backfill all of the data, without having to regenerate segment.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [incubator-pinot] npawar commented on issue #5509: Derived columns

Posted by GitBox <gi...@apache.org>.

npawar commented on issue #5509:
URL: https://github.com/apache/incubator-pinot/issues/5509#issuecomment-708719006


   Should the flag "derived" be part of the FieldSpec, or part of the TableConfig -> IngestionConfig -> TransformConfig?
   It makes more sense in the fieldSpec. But we end up having isDerived in the schema, and the actual transform function string in the table


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [incubator-pinot] yupeng9 commented on issue #5509: Derived columns

Posted by GitBox <gi...@apache.org>.

yupeng9 commented on issue #5509:
URL: https://github.com/apache/incubator-pinot/issues/5509#issuecomment-708861168


   > Should the flag "derived" be part of the FieldSpec, or part of the TableConfig -> IngestionConfig -> TransformConfig?
   > 
   > It makes more sense in the fieldSpec. But we end up having isDerived in the schema, and the actual transform function string in the table
   
   I suggest we keep schema logical, away from the actual impl


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [incubator-pinot] yupeng9 commented on issue #5509: Derived columns

Posted by GitBox <gi...@apache.org>.

yupeng9 commented on issue #5509:
URL: https://github.com/apache/incubator-pinot/issues/5509#issuecomment-708860324


   +1 to derived over derived. It's useful for defining common expressions. Also, from impl it's just a topology sort


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [incubator-pinot] npawar commented on issue #5509: Derived columns

Posted by GitBox <gi...@apache.org>.

npawar commented on issue #5509:
URL: https://github.com/apache/incubator-pinot/issues/5509#issuecomment-708706434

**Challenges**
Although this seems exactly like transform functions, there's some differences because of which we cannot handle this solely as regular transform functions.

Say we have columns `a, b, c` in the raw data source.
Say we have columns `a, c, x, y` in the Pinot schema, such that
```
a -> a
c -> c
x -> f (b)
y -> f(a, c)

```
1) The way transform functions are designed right now, arguments to transform functions can only be from `a, b, c`. If we wanted to add `z -> f(x, y)`, this would not be supported.
2) Transform functions are only evaluated during ingestion. In case of derived columns, we want to support adding even after segment creation i.e. add some derived columns to an existing schema, and see the new values in the segments after a reload.

**Changes**
1 is easy to fix, and can be done by simply enhancing the support for transform configs. Some changes needed:
- Remove validations which prevent adding transform functions of the derived kind (i.e. y = f(z) and x = f(y) is blocked in Table config validations right now, can be easily removed).
- In the ExpressionTransformer, we simply identify the derived columns, and evaluate them after the non-derived fields.

Handling 2 will need some more changes. We need to start computing the derived field's transform functions, during segment reload. For this, we could piggyback on the `DefaultColumnHandler`. Similar to `BaseDefaultColumnHandler#updateDefaultColumn`, we can introduce a `BaseDefaultColumnHandler#updateDerivedColumns`, which can be called instead of `updateDefaultColumn` if the column is a derived field.

**Identifying derived fields**
We also need a flag on the FieldSpec called `derived`. This flag will help us distinguish between derived fields and regular fields. Here's an example for why we need this:
You may have `y = f(z) and x = f(y)`. Here x is obviously derived, as it is using `y` as arguments, and `y` is not in source data.
You may also have `y = y and x = f(y)`. Here it is not obvious that x is a derived field or not, because y is both source and destination.
In both of the examples, `x` will be evaluated during segment creation. The deciding factor for whether a field is derived or not, is whether user wants Pinot to generate the values during a segment reload, if the column was not already present.
If user marks `x` as a derived column and reloads segments, all segments missing `x` should evaluate `x = f(y)` using value of `y` already in the segment.
If user does not mark `x` as a derived column, during reload, all segments missing `x` should simply add default value for `x`.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [incubator-pinot] npawar commented on issue #5509: Derived columns

Posted by GitBox <gi...@apache.org>.

npawar commented on issue #5509:
URL: https://github.com/apache/incubator-pinot/issues/5509#issuecomment-708710753


   FYI @kishoreg @Jackie-Jiang 
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [incubator-pinot] npawar commented on issue #5509: Derived columns

Posted by GitBox <gi...@apache.org>.

npawar commented on issue #5509:
URL: https://github.com/apache/incubator-pinot/issues/5509#issuecomment-708715214


   I think we should support derived on top of derived.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [incubator-pinot] Jackie-Jiang commented on issue #5509: Derived columns

Posted by GitBox <gi...@apache.org>.

Jackie-Jiang commented on issue #5509:
URL: https://github.com/apache/incubator-pinot/issues/5509#issuecomment-708713833


   Do we want to support derived field on top of derived field?
   E.g. we have x in the source data, and we want to add y = f(x) and z = f(y)


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org