You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@apex.apache.org by "Herger, Brendan" <Br...@capitalone.com> on 2016/03/23 00:49:51 UTC
Apex DataFrame
Hey all,
I and the Data Innovation Lab team at Capital One (including Ilya Ganelin, Brennon York, Dongming Liang and Rikin Shah) would like to propose the DataFrame for Apache Apex.
We feel that a DataFrame object would solve Apex’s need for a standardized internal data format (schema aware object containing data), and a high level API for common data operations (integrated with the data store).
This standardized data storage and data operations allow for rapid development and iteration on data pipelines, by allowing users to leverage common code components. It also allows for more standardized workflows, built on common components with regularized syntax.
For further details, please see the design document located at:
https://docs.google.com/document/d/13Eidx-Xg5DrLtbVX-UXhpkjt5iwxoZqTUMGhzh48GPw/edit?usp=sharing
Thanks,
Brendan Herger
Data Scientist, Member of Technical Staff
Data Innovation Lab @ Capital One
________________________________________________________
The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.
Re: Apex DataFrame
Posted by "Herger, Brendan" <Br...@capitalone.com>.
David,
You bring up a great point. This hits on short term and long term goals.
Version 1.0
I’ve purposely not included a DAG, because the DAG Planner trivially translates each operation (basically each line in the example) in to one operator. Additionally, every operator has one input port for each DataFrame involved and one output port for each DataFrame involved. As a result, almost every operator has one input and one output (except for joining two data frames, and similar cases).
Version 2.0
Once we have functioning operations, we can also optimize our DAG to reduce redundant work and increase efficiency. In a second iteration, we’d like to evaluate creating a DAG planner which can act as a query optimizer. This would allow users to write the same code, but then the DAG Planner would seek out the most optimized DAG (similar to Flink and SystemML).
Thanks,
Brendan Herger
Data Scientist, Member of Technical Staff
Data Innovation Lab @ Capital One
415.582.7457 (cell)
On 3/23/16, 4:12 PM, "David Yan" <da...@datatorrent.com> wrote:
>Hi Brendan,
>
>Can you draw the result DAG for the DataFrame pseudo-code in the document?
>I'd assume there will be a file input operator, a CSV parser operator, a
>filter operator (for price != null and datetime > 1992/1/24), an enricher
>(for normalized price and price max 30 days), a custom non-DataFrame
>operator, and an output operator to H2O and an output operator to csv, with
>both of the operators connecting to the output port of the custom
>non-DataFrame operator?
>
>Thanks,
>
>David
>
>On Tue, Mar 22, 2016 at 4:49 PM, Herger, Brendan <
>Brendan.Herger@capitalone.com> wrote:
>
>> Hey all,
>>
>> I and the Data Innovation Lab team at Capital One (including Ilya Ganelin,
>> Brennon York, Dongming Liang and Rikin Shah) would like to propose the
>> DataFrame for Apache Apex.
>>
>> We feel that a DataFrame object would solve Apex’s need for a standardized
>> internal data format (schema aware object containing data), and a high
>> level API for common data operations (integrated with the data store).
>>
>> This standardized data storage and data operations allow for rapid
>> development and iteration on data pipelines, by allowing users to leverage
>> common code components. It also allows for more standardized workflows,
>> built on common components with regularized syntax.
>>
>> For further details, please see the design document located at:
>>
>> https://docs.google.com/document/d/13Eidx-Xg5DrLtbVX-UXhpkjt5iwxoZqTUMGhzh48GPw/edit?usp=sharing
>>
>> Thanks,
>> Brendan Herger
>> Data Scientist, Member of Technical Staff
>> Data Innovation Lab @ Capital One
>>
>> ________________________________________________________
>>
>> The information contained in this e-mail is confidential and/or
>> proprietary to Capital One and/or its affiliates and may only be used
>> solely in performance of work or services for Capital One. The information
>> transmitted herewith is intended only for use by the individual or entity
>> to which it is addressed. If the reader of this message is not the intended
>> recipient, you are hereby notified that any review, retransmission,
>> dissemination, distribution, copying or other use of, or taking of any
>> action in reliance upon this information is strictly prohibited. If you
>> have received this communication in error, please contact the sender and
>> delete the material from your computer.
>>
________________________________________________________
The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.
Re: Apex DataFrame
Posted by David Yan <da...@datatorrent.com>.
Hi Brendan,
Can you draw the result DAG for the DataFrame pseudo-code in the document?
I'd assume there will be a file input operator, a CSV parser operator, a
filter operator (for price != null and datetime > 1992/1/24), an enricher
(for normalized price and price max 30 days), a custom non-DataFrame
operator, and an output operator to H2O and an output operator to csv, with
both of the operators connecting to the output port of the custom
non-DataFrame operator?
Thanks,
David
On Tue, Mar 22, 2016 at 4:49 PM, Herger, Brendan <
Brendan.Herger@capitalone.com> wrote:
> Hey all,
>
> I and the Data Innovation Lab team at Capital One (including Ilya Ganelin,
> Brennon York, Dongming Liang and Rikin Shah) would like to propose the
> DataFrame for Apache Apex.
>
> We feel that a DataFrame object would solve Apex’s need for a standardized
> internal data format (schema aware object containing data), and a high
> level API for common data operations (integrated with the data store).
>
> This standardized data storage and data operations allow for rapid
> development and iteration on data pipelines, by allowing users to leverage
> common code components. It also allows for more standardized workflows,
> built on common components with regularized syntax.
>
> For further details, please see the design document located at:
>
> https://docs.google.com/document/d/13Eidx-Xg5DrLtbVX-UXhpkjt5iwxoZqTUMGhzh48GPw/edit?usp=sharing
>
> Thanks,
> Brendan Herger
> Data Scientist, Member of Technical Staff
> Data Innovation Lab @ Capital One
>
> ________________________________________________________
>
> The information contained in this e-mail is confidential and/or
> proprietary to Capital One and/or its affiliates and may only be used
> solely in performance of work or services for Capital One. The information
> transmitted herewith is intended only for use by the individual or entity
> to which it is addressed. If the reader of this message is not the intended
> recipient, you are hereby notified that any review, retransmission,
> dissemination, distribution, copying or other use of, or taking of any
> action in reliance upon this information is strictly prohibited. If you
> have received this communication in error, please contact the sender and
> delete the material from your computer.
>
Re: Apex DataFrame
Posted by Devendra Tagare <de...@datatorrent.com>.
Hi,
You can create a case class.Then map the incoming RDD's to the case class &
convert the map to a data frame.By doing this you would have a Dataframe
with the respective fields and associated datatypes set as per the ETL
rules defined before setting the members of the case class.
Sample below (from the zepplin tutorial)
import org.apache.commons.io.IOUtils
import java.net.URL
import java.nio.charset.Charset
// load bank data
val bankText = sc.parallelize(
IOUtils.toString(
new URL("
https://s3.amazonaws.com/apache-zeppelin/tutorial/bank/bank.csv"),
Charset.forName("utf8")).split("\n"))
case class Bank(age: Integer, job: String, marital: String, education:
String, balance: Integer)
val bank = bankText.map(s => s.split(";")).filter(s => s(0) !=
"\"age\"").map(
s => Bank(s(0).toInt,
s(1).replaceAll("\"", ""),
s(2).replaceAll("\"", ""),
s(3).replaceAll("\"", ""),
s(5).replaceAll("\"", "").toInt
)
).toDF()
bank.registerTempTable("bank")
The objects the above code creates are,
bankText: org.apache.spark.rdd.RDD[String] = ParallelCollectionRDD[32] at
parallelize at <console>:65 defined class Bank bank:
org.apache.spark.sql.DataFrame = [age: int, job: string, marital: string,
education: string, balance: int]
Thanks,
Dev
On Wed, Mar 23, 2016 at 2:36 PM, Ashwin Chandra Putta <
ashwinchandrap@gmail.com> wrote:
> Can you also provide types for fields in the code example? How should a
> fieldname associate itself with a type?
>
> Regards,
> Ashwin.
>
> On Wed, Mar 23, 2016 at 9:32 AM, Amol Kekre <am...@datatorrent.com> wrote:
>
> > Ilya, Brendan,
> > Thanks for driving this feature set. We are looking into it. We should
> have
> > feedback soon.
> >
> > Thks,
> > Amol
> >
> >
> > On Tue, Mar 22, 2016 at 6:27 PM, Ganelin, Ilya <
> > Ilya.Ganelin@capitalone.com>
> > wrote:
> >
> > > Would love the community's input on this idea and thoughts on technical
> > > challenges we may run into when implementing this within Apex.
> > >
> > >
> > >
> > > Sent with Good (www.good.com)
> > > ________________________________
> > > From: Herger, Brendan <Br...@capitalone.com>
> > > Sent: Tuesday, March 22, 2016 7:49:51 PM
> > > To: dev@apex.incubator.apache.org
> > > Subject: Apex DataFrame
> > >
> > > Hey all,
> > >
> > > I and the Data Innovation Lab team at Capital One (including Ilya
> > Ganelin,
> > > Brennon York, Dongming Liang and Rikin Shah) would like to propose the
> > > DataFrame for Apache Apex.
> > >
> > > We feel that a DataFrame object would solve Apex’s need for a
> > standardized
> > > internal data format (schema aware object containing data), and a high
> > > level API for common data operations (integrated with the data store).
> > >
> > > This standardized data storage and data operations allow for rapid
> > > development and iteration on data pipelines, by allowing users to
> > leverage
> > > common code components. It also allows for more standardized workflows,
> > > built on common components with regularized syntax.
> > >
> > > For further details, please see the design document located at:
> > >
> > >
> >
> https://docs.google.com/document/d/13Eidx-Xg5DrLtbVX-UXhpkjt5iwxoZqTUMGhzh48GPw/edit?usp=sharing
> > >
> > > Thanks,
> > > Brendan Herger
> > > Data Scientist, Member of Technical Staff
> > > Data Innovation Lab @ Capital One
> > >
> > > ________________________________________________________
> > >
> > > The information contained in this e-mail is confidential and/or
> > > proprietary to Capital One and/or its affiliates and may only be used
> > > solely in performance of work or services for Capital One. The
> > information
> > > transmitted herewith is intended only for use by the individual or
> entity
> > > to which it is addressed. If the reader of this message is not the
> > intended
> > > recipient, you are hereby notified that any review, retransmission,
> > > dissemination, distribution, copying or other use of, or taking of any
> > > action in reliance upon this information is strictly prohibited. If you
> > > have received this communication in error, please contact the sender
> and
> > > delete the material from your computer.
> > > ________________________________________________________
> > >
> > > The information contained in this e-mail is confidential and/or
> > > proprietary to Capital One and/or its affiliates and may only be used
> > > solely in performance of work or services for Capital One. The
> > information
> > > transmitted herewith is intended only for use by the individual or
> entity
> > > to which it is addressed. If the reader of this message is not the
> > intended
> > > recipient, you are hereby notified that any review, retransmission,
> > > dissemination, distribution, copying or other use of, or taking of any
> > > action in reliance upon this information is strictly prohibited. If you
> > > have received this communication in error, please contact the sender
> and
> > > delete the material from your computer.
> > >
> >
>
>
>
> --
>
> Regards,
> Ashwin.
>
Re: Apex DataFrame
Posted by Ashwin Chandra Putta <as...@gmail.com>.
Can you also provide types for fields in the code example? How should a
fieldname associate itself with a type?
Regards,
Ashwin.
On Wed, Mar 23, 2016 at 9:32 AM, Amol Kekre <am...@datatorrent.com> wrote:
> Ilya, Brendan,
> Thanks for driving this feature set. We are looking into it. We should have
> feedback soon.
>
> Thks,
> Amol
>
>
> On Tue, Mar 22, 2016 at 6:27 PM, Ganelin, Ilya <
> Ilya.Ganelin@capitalone.com>
> wrote:
>
> > Would love the community's input on this idea and thoughts on technical
> > challenges we may run into when implementing this within Apex.
> >
> >
> >
> > Sent with Good (www.good.com)
> > ________________________________
> > From: Herger, Brendan <Br...@capitalone.com>
> > Sent: Tuesday, March 22, 2016 7:49:51 PM
> > To: dev@apex.incubator.apache.org
> > Subject: Apex DataFrame
> >
> > Hey all,
> >
> > I and the Data Innovation Lab team at Capital One (including Ilya
> Ganelin,
> > Brennon York, Dongming Liang and Rikin Shah) would like to propose the
> > DataFrame for Apache Apex.
> >
> > We feel that a DataFrame object would solve Apex’s need for a
> standardized
> > internal data format (schema aware object containing data), and a high
> > level API for common data operations (integrated with the data store).
> >
> > This standardized data storage and data operations allow for rapid
> > development and iteration on data pipelines, by allowing users to
> leverage
> > common code components. It also allows for more standardized workflows,
> > built on common components with regularized syntax.
> >
> > For further details, please see the design document located at:
> >
> >
> https://docs.google.com/document/d/13Eidx-Xg5DrLtbVX-UXhpkjt5iwxoZqTUMGhzh48GPw/edit?usp=sharing
> >
> > Thanks,
> > Brendan Herger
> > Data Scientist, Member of Technical Staff
> > Data Innovation Lab @ Capital One
> >
> > ________________________________________________________
> >
> > The information contained in this e-mail is confidential and/or
> > proprietary to Capital One and/or its affiliates and may only be used
> > solely in performance of work or services for Capital One. The
> information
> > transmitted herewith is intended only for use by the individual or entity
> > to which it is addressed. If the reader of this message is not the
> intended
> > recipient, you are hereby notified that any review, retransmission,
> > dissemination, distribution, copying or other use of, or taking of any
> > action in reliance upon this information is strictly prohibited. If you
> > have received this communication in error, please contact the sender and
> > delete the material from your computer.
> > ________________________________________________________
> >
> > The information contained in this e-mail is confidential and/or
> > proprietary to Capital One and/or its affiliates and may only be used
> > solely in performance of work or services for Capital One. The
> information
> > transmitted herewith is intended only for use by the individual or entity
> > to which it is addressed. If the reader of this message is not the
> intended
> > recipient, you are hereby notified that any review, retransmission,
> > dissemination, distribution, copying or other use of, or taking of any
> > action in reliance upon this information is strictly prohibited. If you
> > have received this communication in error, please contact the sender and
> > delete the material from your computer.
> >
>
--
Regards,
Ashwin.
Re: Apex DataFrame
Posted by Amol Kekre <am...@datatorrent.com>.
Ilya, Brendan,
Thanks for driving this feature set. We are looking into it. We should have
feedback soon.
Thks,
Amol
On Tue, Mar 22, 2016 at 6:27 PM, Ganelin, Ilya <Il...@capitalone.com>
wrote:
> Would love the community's input on this idea and thoughts on technical
> challenges we may run into when implementing this within Apex.
>
>
>
> Sent with Good (www.good.com)
> ________________________________
> From: Herger, Brendan <Br...@capitalone.com>
> Sent: Tuesday, March 22, 2016 7:49:51 PM
> To: dev@apex.incubator.apache.org
> Subject: Apex DataFrame
>
> Hey all,
>
> I and the Data Innovation Lab team at Capital One (including Ilya Ganelin,
> Brennon York, Dongming Liang and Rikin Shah) would like to propose the
> DataFrame for Apache Apex.
>
> We feel that a DataFrame object would solve Apex’s need for a standardized
> internal data format (schema aware object containing data), and a high
> level API for common data operations (integrated with the data store).
>
> This standardized data storage and data operations allow for rapid
> development and iteration on data pipelines, by allowing users to leverage
> common code components. It also allows for more standardized workflows,
> built on common components with regularized syntax.
>
> For further details, please see the design document located at:
>
> https://docs.google.com/document/d/13Eidx-Xg5DrLtbVX-UXhpkjt5iwxoZqTUMGhzh48GPw/edit?usp=sharing
>
> Thanks,
> Brendan Herger
> Data Scientist, Member of Technical Staff
> Data Innovation Lab @ Capital One
>
> ________________________________________________________
>
> The information contained in this e-mail is confidential and/or
> proprietary to Capital One and/or its affiliates and may only be used
> solely in performance of work or services for Capital One. The information
> transmitted herewith is intended only for use by the individual or entity
> to which it is addressed. If the reader of this message is not the intended
> recipient, you are hereby notified that any review, retransmission,
> dissemination, distribution, copying or other use of, or taking of any
> action in reliance upon this information is strictly prohibited. If you
> have received this communication in error, please contact the sender and
> delete the material from your computer.
> ________________________________________________________
>
> The information contained in this e-mail is confidential and/or
> proprietary to Capital One and/or its affiliates and may only be used
> solely in performance of work or services for Capital One. The information
> transmitted herewith is intended only for use by the individual or entity
> to which it is addressed. If the reader of this message is not the intended
> recipient, you are hereby notified that any review, retransmission,
> dissemination, distribution, copying or other use of, or taking of any
> action in reliance upon this information is strictly prohibited. If you
> have received this communication in error, please contact the sender and
> delete the material from your computer.
>
RE: Apex DataFrame
Posted by "Ganelin, Ilya" <Il...@capitalone.com>.
Would love the community's input on this idea and thoughts on technical challenges we may run into when implementing this within Apex.
Sent with Good (www.good.com)
________________________________
From: Herger, Brendan <Br...@capitalone.com>
Sent: Tuesday, March 22, 2016 7:49:51 PM
To: dev@apex.incubator.apache.org
Subject: Apex DataFrame
Hey all,
I and the Data Innovation Lab team at Capital One (including Ilya Ganelin, Brennon York, Dongming Liang and Rikin Shah) would like to propose the DataFrame for Apache Apex.
We feel that a DataFrame object would solve Apex’s need for a standardized internal data format (schema aware object containing data), and a high level API for common data operations (integrated with the data store).
This standardized data storage and data operations allow for rapid development and iteration on data pipelines, by allowing users to leverage common code components. It also allows for more standardized workflows, built on common components with regularized syntax.
For further details, please see the design document located at:
https://docs.google.com/document/d/13Eidx-Xg5DrLtbVX-UXhpkjt5iwxoZqTUMGhzh48GPw/edit?usp=sharing
Thanks,
Brendan Herger
Data Scientist, Member of Technical Staff
Data Innovation Lab @ Capital One
________________________________________________________
The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.
________________________________________________________
The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.