You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@griffin.apache.org by Vikram Jain <vi...@enquero.com> on 2019/02/06 07:58:00 UTC

Measure creation with DSL Type as "DF-OPS"

Hi,
Can someone please explain the process of creating a measure with DSL type as "DF-OPS". A sample measure.json with explanation of associated fields with df-ops would be highly appreciated. I could not find any resources on cwiki or github that explains the process.

Thanks in advance.
Vikram

RE: Measure creation with DSL Type as "DF-OPS"

Posted by "Lionel, Liu" <bh...@163.com>.
That’s right, Griffin depends on the operations of spark sql, transferring a data frame into another, but for the operations which could not be covered by spark sql, some pre-defined “df-ops” could help on this. 
Users can implement their own “df-ops” for such specific operations.

Thanks
Lionel, Liu

From: Nick Sokolov
Sent: 2019年2月8日 0:47
To: users@griffin.apache.org
Cc: dev@griffin.apache.org
Subject: Re: Measure creation with DSL Type as "DF-OPS"

I did not see any documentation on it, but from source code, it is doing some pre-defined transformation based on "rule" parameter (from_json, clear, accuracy), with in.dataframe.name as input and out.dataframe.name as output.

Transformations themselves are defined in DataFrameOps.scala, and picking transformation is done in DataFrameOpsTransformStep.scala#L36. From context where df-ops is mentioned, it looks like it's mostly useful for reading json from kafka topics or from flat files, or defining empty RDDs in DQ job context.

On Tue, Feb 5, 2019 at 11:58 PM Vikram Jain <vi...@enquero.com> wrote:
Hi,
Can someone please explain the process of creating a measure with DSL type as “DF-OPS”. A sample measure.json with explanation of associated fields with df-ops would be highly appreciated. I could not find any resources on cwiki or github that explains the process.
 
Thanks in advance.
Vikram

RE: Measure creation with DSL Type as "DF-OPS"

Posted by "Lionel, Liu" <bh...@163.com>.
That’s right, Griffin depends on the operations of spark sql, transferring a data frame into another, but for the operations which could not be covered by spark sql, some pre-defined “df-ops” could help on this. 
Users can implement their own “df-ops” for such specific operations.

Thanks
Lionel, Liu

From: Nick Sokolov
Sent: 2019年2月8日 0:47
To: users@griffin.apache.org
Cc: dev@griffin.apache.org
Subject: Re: Measure creation with DSL Type as "DF-OPS"

I did not see any documentation on it, but from source code, it is doing some pre-defined transformation based on "rule" parameter (from_json, clear, accuracy), with in.dataframe.name as input and out.dataframe.name as output.

Transformations themselves are defined in DataFrameOps.scala, and picking transformation is done in DataFrameOpsTransformStep.scala#L36. From context where df-ops is mentioned, it looks like it's mostly useful for reading json from kafka topics or from flat files, or defining empty RDDs in DQ job context.

On Tue, Feb 5, 2019 at 11:58 PM Vikram Jain <vi...@enquero.com> wrote:
Hi,
Can someone please explain the process of creating a measure with DSL type as “DF-OPS”. A sample measure.json with explanation of associated fields with df-ops would be highly appreciated. I could not find any resources on cwiki or github that explains the process.
 
Thanks in advance.
Vikram

Re: Measure creation with DSL Type as "DF-OPS"

Posted by Nick Sokolov <ch...@gmail.com>.
I did not see any documentation on it, but from source code, it is doing
some pre-defined transformation based on "rule" parameter (from_json,
clear, accuracy), with in.dataframe.name as input and out.dataframe.name as
output.

Transformations themselves are defined in DataFrameOps.scala
<https://github.com/apache/griffin/blob/master/measure/src/main/scala/org/apache/griffin/measure/step/transform/DataFrameOps.scala>,
and picking transformation is done in DataFrameOpsTransformStep.scala#L36
<https://github.com/apache/griffin/blob/master/measure/src/main/scala/org/apache/griffin/measure/step/transform/DataFrameOpsTransformStep.scala#L36>.
From context where df-ops is mentioned, it looks like it's mostly useful
for reading json from kafka topics or from flat files, or defining empty
RDDs in DQ job context.

On Tue, Feb 5, 2019 at 11:58 PM Vikram Jain <vi...@enquero.com> wrote:

> Hi,
>
> Can someone please explain the process of creating a measure with DSL type
> as “DF-OPS”. A sample measure.json with explanation of associated fields
> with df-ops would be highly appreciated. I could not find any resources on
> cwiki or github that explains the process.
>
>
>
> Thanks in advance.
>
> Vikram
>

Re: Measure creation with DSL Type as "DF-OPS"

Posted by Nick Sokolov <ch...@gmail.com>.
I did not see any documentation on it, but from source code, it is doing
some pre-defined transformation based on "rule" parameter (from_json,
clear, accuracy), with in.dataframe.name as input and out.dataframe.name as
output.

Transformations themselves are defined in DataFrameOps.scala
<https://github.com/apache/griffin/blob/master/measure/src/main/scala/org/apache/griffin/measure/step/transform/DataFrameOps.scala>,
and picking transformation is done in DataFrameOpsTransformStep.scala#L36
<https://github.com/apache/griffin/blob/master/measure/src/main/scala/org/apache/griffin/measure/step/transform/DataFrameOpsTransformStep.scala#L36>.
From context where df-ops is mentioned, it looks like it's mostly useful
for reading json from kafka topics or from flat files, or defining empty
RDDs in DQ job context.

On Tue, Feb 5, 2019 at 11:58 PM Vikram Jain <vi...@enquero.com> wrote:

> Hi,
>
> Can someone please explain the process of creating a measure with DSL type
> as “DF-OPS”. A sample measure.json with explanation of associated fields
> with df-ops would be highly appreciated. I could not find any resources on
> cwiki or github that explains the process.
>
>
>
> Thanks in advance.
>
> Vikram
>