You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Adam Roberts <AR...@uk.ibm.com> on 2016/09/01 14:25:56 UTC
Re: Hi, guys, does anyone use Spark in finance market?
Hi, yes, there's definitely a market for Apache Spark and financial
institutions, I can't provide specific details but to answer your survey:
"yes" and "more than a few GB!"
Here are a couple of examples showing Spark with financial data, full
disclosure that I work for IBM, I'm sure there are lots more examples you
can find too:
https://www.youtube.com/watch?v=VWBNoIwGEjo shows how Spark can be used
with simple sentiment analysis to figure out correlations between real
world events and stock market changes. The Spark specific part is from 3
04 until the 8th minute
https://www.youtube.com/watch?v=sDmWcuO5Rk8 is a similar example where
Spark is again used with sentiment analysis. One could also analyse
financial data to identify trends, I think a lot of the machine learning
APIs will be useful here e.g. logistic regression with many features could
be used to decide whether or not an investment is a good idea based on
training data (so we'd look at real outcomes from previous speculations)
In both cases you can see Spark is a very important component for
performing the calculations with financial data.
I also know that Goldman Sachs mentioned they are interested in Spark, one
talk is at https://www.youtube.com/watch?v=HWwAoTK2YrQ, so this is more
evidence of the financial industries paying attention to big data and
Spark.
Regarding your app: I expected it to be similar to the first example where
the signals you mention are real world events (e.g. the fed lowers
interest rates or companies are rumoured to either be about to float or be
acquired).
At the 4 30 part I think you actually identify previous index values and
extrapolate what they are likely to become using, so in theory your system
would become more accurate over time although would going off indexes
alone be sufficient (if indeed this is what you're doing).
I think you'd want to combine this with real world speculation/news to
figure out *why* the price is likely to change, how much by and in which
direction.
I agree that Apache Spark can be just the right tool for doing the heavy
lifting required for analysis, computation and modelling of big data so
looking forward to future Spark work in this area, and I wonder how we as
Spark developers can make it easier/more powerful for Spark users to do so
From: "Taotao.Li" <ch...@gmail.com>
To: user <us...@spark.apache.org>
Date: 30/08/2016 14:14
Subject: Hi, guys, does anyone use Spark in finance market?
Hi, guys,
I'm a quant engineer in China, and I believe it's very promising when
using Spark in the financial market. But I didn't find cases which combine
spark and finance.
So here I wanna do a small survey:
do you guys use Spark in financial market related project?
if yes, how large data was fed in your spark application?
thanks a lot.
___________________
A little ad, I attended IBM Spark Hackathon, which is here :
http://apachespark.devpost.com/ , and I submitted a small application,
which will be used in my strategies, hope you guys and give me a vote and
some suggestions on how to use spark in financial market, to discover some
trade opportunity.
here is my small app:
http://devpost.com/software/spark-in-finance-quantitative-investing
thanks a lot.
--
___________________
Quant | Engineer | Boy
___________________
blog: http://litaotao.github.io
github: www.github.com/litaotao
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Re: Hi, guys, does anyone use Spark in finance market?
Posted by "Taotao.Li" <ch...@gmail.com>.
Hi, Mich, thanks for your detailed reply, that system seems interesting,
I'll take a deep look on it, great thanks again.
On Fri, Sep 2, 2016 at 6:54 AM, Mich Talebzadeh <mi...@gmail.com>
wrote:
> I have worked for a long time in the Financial sector and Spark plus Spark
> Streaming is used for both Market risk and Credit risk calculations.
>
> I am currently working on a design pattern of mine to use Lambda
> Architecture for Financial risk. Big data techniques and applications are
> providing data-intensive industries such as banks a solution to extract
> latent information from the vast volume of data accumulated from all the
> lines of enterprise business. Given the volume of data and the available
> resources, the banks have now begun to focus on the benefits that big data
> techniques can bring in streamlining current practices, as well as
> designing new solutions. A high level design of mine is attached.
>
>
>
>
> HTH
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
> On 1 September 2016 at 15:25, Adam Roberts <AR...@uk.ibm.com> wrote:
>
>> Hi, yes, there's definitely a market for Apache Spark and financial
>> institutions, I can't provide specific details but to answer your survey:
>> "yes" and "more than a few GB!"
>>
>> Here are a couple of examples showing Spark with financial data, full
>> disclosure that I work for IBM, I'm sure there are lots more examples you
>> can find too:
>>
>> - https://www.youtube.com/watch?v=VWBNoIwGEjo shows how Spark can be
>> used with simple sentiment analysis to figure out correlations between real
>> world events and stock market changes. The Spark specific part is from 3 04
>> until the 8th minute
>> - https://www.youtube.com/watch?v=sDmWcuO5Rk8 is a similar example
>> where Spark is again used with sentiment analysis. One could also analyse
>> financial data to identify trends, I think a lot of the machine learning
>> APIs will be useful here e.g. logistic regression with many features could
>> be used to decide whether or not an investment is a good idea based on
>> training data (so we'd look at real outcomes from previous speculations)
>>
>>
>> In both cases you can see Spark is a very important component for
>> performing the calculations with financial data.
>>
>> I also know that Goldman Sachs mentioned they are interested in Spark,
>> one talk is at https://www.youtube.com/watch?v=HWwAoTK2YrQ, so this is
>> more evidence of the financial industries paying attention to big data and
>> Spark.
>>
>> Regarding your app: I expected it to be similar to the first example
>> where the signals you mention are real world events (e.g. the fed lowers
>> interest rates or companies are rumoured to either be about to float or be
>> acquired).
>>
>> At the 4 30 part I think you actually identify previous index values and
>> extrapolate what they are likely to become using, so in theory your system
>> would become more accurate over time although would going off indexes alone
>> be sufficient (if indeed this is what you're doing).
>>
>> I think you'd want to combine this with real world speculation/news to
>> figure out *why* the price is likely to change, how much by and in which
>> direction.
>>
>> I agree that Apache Spark can be just the right tool for doing the heavy
>> lifting required for analysis, computation and modelling of big data so
>> looking forward to future Spark work in this area, and I wonder how we as
>> Spark developers can make it easier/more powerful for Spark users to do so
>>
>>
>>
>>
>> From: "Taotao.Li" <ch...@gmail.com>
>> To: user <us...@spark.apache.org>
>> Date: 30/08/2016 14:14
>> Subject: Hi, guys, does anyone use Spark in finance market?
>> ------------------------------
>>
>>
>>
>>
>> Hi, guys,
>>
>> I'm a quant engineer in China, and I believe it's very promising
>> when using Spark in the financial market. But I didn't find cases which
>> combine spark and finance.
>>
>> So here I wanna do a small survey:
>>
>> - do you guys use Spark in financial market related project?
>> - if yes, how large data was fed in your spark application?
>>
>>
>> thanks a lot.
>>
>> *___________________***
>> A little ad, I attended IBM Spark Hackathon, which is here :
>> *http://apachespark.devpost.com/* <http://apachespark.devpost.com/> ,
>> and I submitted a small application, which will be used in my strategies,
>> hope you guys and give me a vote and some suggestions on how to use spark
>> in financial market, to discover some trade opportunity.
>>
>> here is my small app:
>> *http://devpost.com/software/spark-in-finance-quantitative-investing*
>> <http://devpost.com/software/spark-in-finance-quantitative-investing>
>>
>> thanks a lot.
>>
>>
>> --
>> *___________________*
>> Quant | Engineer | Boy
>> *___________________*
>> *blog*: *http://litaotao.github.io*
>> <http://litaotao.github.io?utm_source=spark_mail>
>> *github*: *www.github.com/litaotao* <http://www.github.com/litaotao>
>>
>> Unless stated otherwise above:
>> IBM United Kingdom Limited - Registered in England and Wales with number
>> 741598.
>> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>>
>
>
--
*___________________*
Quant | Engineer | Boy
*___________________*
*blog*: http://litaotao.github.io
<http://litaotao.github.io?utm_source=spark_mail>
*github*: www.github.com/litaotao
Re: Hi, guys, does anyone use Spark in finance market?
Posted by Mich Talebzadeh <mi...@gmail.com>.
I have worked for a long time in the Financial sector and Spark plus Spark
Streaming is used for both Market risk and Credit risk calculations.
I am currently working on a design pattern of mine to use Lambda
Architecture for Financial risk. Big data techniques and applications are
providing data-intensive industries such as banks a solution to extract
latent information from the vast volume of data accumulated from all the
lines of enterprise business. Given the volume of data and the available
resources, the banks have now begun to focus on the benefits that big data
techniques can bring in streamlining current practices, as well as
designing new solutions. A high level design of mine is attached.
HTH
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On 1 September 2016 at 15:25, Adam Roberts <AR...@uk.ibm.com> wrote:
> Hi, yes, there's definitely a market for Apache Spark and financial
> institutions, I can't provide specific details but to answer your survey:
> "yes" and "more than a few GB!"
>
> Here are a couple of examples showing Spark with financial data, full
> disclosure that I work for IBM, I'm sure there are lots more examples you
> can find too:
>
> - https://www.youtube.com/watch?v=VWBNoIwGEjo shows how Spark can be
> used with simple sentiment analysis to figure out correlations between real
> world events and stock market changes. The Spark specific part is from 3 04
> until the 8th minute
> - https://www.youtube.com/watch?v=sDmWcuO5Rk8 is a similar example
> where Spark is again used with sentiment analysis. One could also analyse
> financial data to identify trends, I think a lot of the machine learning
> APIs will be useful here e.g. logistic regression with many features could
> be used to decide whether or not an investment is a good idea based on
> training data (so we'd look at real outcomes from previous speculations)
>
>
> In both cases you can see Spark is a very important component for
> performing the calculations with financial data.
>
> I also know that Goldman Sachs mentioned they are interested in Spark, one
> talk is at https://www.youtube.com/watch?v=HWwAoTK2YrQ, so this is more
> evidence of the financial industries paying attention to big data and Spark.
>
> Regarding your app: I expected it to be similar to the first example where
> the signals you mention are real world events (e.g. the fed lowers interest
> rates or companies are rumoured to either be about to float or be
> acquired).
>
> At the 4 30 part I think you actually identify previous index values and
> extrapolate what they are likely to become using, so in theory your system
> would become more accurate over time although would going off indexes alone
> be sufficient (if indeed this is what you're doing).
>
> I think you'd want to combine this with real world speculation/news to
> figure out *why* the price is likely to change, how much by and in which
> direction.
>
> I agree that Apache Spark can be just the right tool for doing the heavy
> lifting required for analysis, computation and modelling of big data so
> looking forward to future Spark work in this area, and I wonder how we as
> Spark developers can make it easier/more powerful for Spark users to do so
>
>
>
>
> From: "Taotao.Li" <ch...@gmail.com>
> To: user <us...@spark.apache.org>
> Date: 30/08/2016 14:14
> Subject: Hi, guys, does anyone use Spark in finance market?
> ------------------------------
>
>
>
>
> Hi, guys,
>
> I'm a quant engineer in China, and I believe it's very promising when
> using Spark in the financial market. But I didn't find cases which combine
> spark and finance.
>
> So here I wanna do a small survey:
>
> - do you guys use Spark in financial market related project?
> - if yes, how large data was fed in your spark application?
>
>
> thanks a lot.
>
> *___________________***
> A little ad, I attended IBM Spark Hackathon, which is here :
> *http://apachespark.devpost.com/* <http://apachespark.devpost.com/> , and
> I submitted a small application, which will be used in my strategies, hope
> you guys and give me a vote and some suggestions on how to use spark in
> financial market, to discover some trade opportunity.
>
> here is my small app:
> *http://devpost.com/software/spark-in-finance-quantitative-investing*
> <http://devpost.com/software/spark-in-finance-quantitative-investing>
>
> thanks a lot.
>
>
> --
> *___________________*
> Quant | Engineer | Boy
> *___________________*
> *blog*: *http://litaotao.github.io*
> <http://litaotao.github.io?utm_source=spark_mail>
> *github*: *www.github.com/litaotao* <http://www.github.com/litaotao>
>
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number
> 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>
Re: Hi, guys, does anyone use Spark in finance market?
Posted by "Taotao.Li" <ch...@gmail.com>.
Hi, Adam, great thanks for your detailed reply, the three videos are
very referential for me. Actually, the App submitted to IBM Spark Contest
is a very small demo, I'll do much more work to enhance that model, and
recently we just started a new project which aims to building a platform
that makes it possible and easy for Chinese financial institutions to
analysis high-frequency market data, which will be ~30G one day.
great thanks again,
On Thu, Sep 1, 2016 at 10:25 PM, Adam Roberts <AR...@uk.ibm.com> wrote:
> Hi, yes, there's definitely a market for Apache Spark and financial
> institutions, I can't provide specific details but to answer your survey:
> "yes" and "more than a few GB!"
>
> Here are a couple of examples showing Spark with financial data, full
> disclosure that I work for IBM, I'm sure there are lots more examples you
> can find too:
>
> - https://www.youtube.com/watch?v=VWBNoIwGEjo shows how Spark can be
> used with simple sentiment analysis to figure out correlations between real
> world events and stock market changes. The Spark specific part is from 3 04
> until the 8th minute
> - https://www.youtube.com/watch?v=sDmWcuO5Rk8 is a similar example
> where Spark is again used with sentiment analysis. One could also analyse
> financial data to identify trends, I think a lot of the machine learning
> APIs will be useful here e.g. logistic regression with many features could
> be used to decide whether or not an investment is a good idea based on
> training data (so we'd look at real outcomes from previous speculations)
>
>
> In both cases you can see Spark is a very important component for
> performing the calculations with financial data.
>
> I also know that Goldman Sachs mentioned they are interested in Spark, one
> talk is at https://www.youtube.com/watch?v=HWwAoTK2YrQ, so this is more
> evidence of the financial industries paying attention to big data and Spark.
>
> Regarding your app: I expected it to be similar to the first example where
> the signals you mention are real world events (e.g. the fed lowers interest
> rates or companies are rumoured to either be about to float or be
> acquired).
>
> At the 4 30 part I think you actually identify previous index values and
> extrapolate what they are likely to become using, so in theory your system
> would become more accurate over time although would going off indexes alone
> be sufficient (if indeed this is what you're doing).
>
> I think you'd want to combine this with real world speculation/news to
> figure out *why* the price is likely to change, how much by and in which
> direction.
>
> I agree that Apache Spark can be just the right tool for doing the heavy
> lifting required for analysis, computation and modelling of big data so
> looking forward to future Spark work in this area, and I wonder how we as
> Spark developers can make it easier/more powerful for Spark users to do so
>
>
>
>
> From: "Taotao.Li" <ch...@gmail.com>
> To: user <us...@spark.apache.org>
> Date: 30/08/2016 14:14
> Subject: Hi, guys, does anyone use Spark in finance market?
> ------------------------------
>
>
>
>
> Hi, guys,
>
> I'm a quant engineer in China, and I believe it's very promising when
> using Spark in the financial market. But I didn't find cases which combine
> spark and finance.
>
> So here I wanna do a small survey:
>
> - do you guys use Spark in financial market related project?
> - if yes, how large data was fed in your spark application?
>
>
> thanks a lot.
>
> *___________________***
> A little ad, I attended IBM Spark Hackathon, which is here :
> *http://apachespark.devpost.com/* <http://apachespark.devpost.com/> , and
> I submitted a small application, which will be used in my strategies, hope
> you guys and give me a vote and some suggestions on how to use spark in
> financial market, to discover some trade opportunity.
>
> here is my small app:
> *http://devpost.com/software/spark-in-finance-quantitative-investing*
> <http://devpost.com/software/spark-in-finance-quantitative-investing>
>
> thanks a lot.
>
>
> --
> *___________________*
> Quant | Engineer | Boy
> *___________________*
> *blog*: *http://litaotao.github.io*
> <http://litaotao.github.io?utm_source=spark_mail>
> *github*: *www.github.com/litaotao* <http://www.github.com/litaotao>
>
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number
> 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>
--
*___________________*
Quant | Engineer | Boy
*___________________*
*blog*: http://litaotao.github.io
<http://litaotao.github.io?utm_source=spark_mail>
*github*: www.github.com/litaotao