You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by ishan-jain <ja...@gmail.com> on 2017/06/12 05:08:51 UTC
Data Analysis and visualization
I am new to BIG Data .Just been working for a month.
I have HDFS data of stock prices. I need to perform data analysis(maybe some
ML) and visualizations(Graphs and charts). For that i need Mapreduce
functions. Which approach should i use?
1. Stream data from IGFS into ignite cache and work on it?
2. Use Hive with Tez and LLap function.(Should i use it with ignite or
independent and directly on HDFS. No info available on the net.)
3. Use presto (Which is the better variant?(Hive or presto))
4. Some other fast way with IGFS if possible.
5. Also which open source tools should i use to accomplish this.
Any help would be appreciated.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Data-Analysis-and-visualization-tp13614.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Data Analysis and visualization
Posted by Ishan Jain <ja...@gmail.com>.
Size would be very large as stock prices would be streamed every hour
On Tue, Jun 13, 2017 at 12:05 PM, Jörn Franke <jo...@gmail.com> wrote:
> What is the size of the data?
> For me it looks more that orc or parquet would be enough.
>
> I do not see here specific in-memory requirements.
>
> On 12. Jun 2017, at 09:59, Ishan Jain <ja...@gmail.com> wrote:
>
> I need to just get the price of a stock which is stored in hdfs with
> timestamp and make a graph with the prices of that stock over time.
>
> On Mon, Jun 12, 2017 at 1:03 PM, Jörn Franke <jo...@gmail.com> wrote:
>
>> First you need the user requirements - without them answering your
>> questions will be difficult
>>
>> > On 12. Jun 2017, at 07:08, ishan-jain <ja...@gmail.com> wrote:
>> >
>> > I am new to BIG Data .Just been working for a month.
>> > I have HDFS data of stock prices. I need to perform data analysis(maybe
>> some
>> > ML) and visualizations(Graphs and charts). For that i need Mapreduce
>> > functions. Which approach should i use?
>> > 1. Stream data from IGFS into ignite cache and work on it?
>> > 2. Use Hive with Tez and LLap function.(Should i use it with ignite or
>> > independent and directly on HDFS. No info available on the net.)
>> > 3. Use presto (Which is the better variant?(Hive or presto))
>> > 4. Some other fast way with IGFS if possible.
>> > 5. Also which open source tools should i use to accomplish this.
>> > Any help would be appreciated.
>> >
>> >
>> >
>> > --
>> > View this message in context: http://apache-ignite-users.705
>> 18.x6.nabble.com/Data-Analysis-and-visualization-tp13614.html
>> > Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>>
>
>
Re: Data Analysis and visualization
Posted by Jörn Franke <jo...@gmail.com>.
What is the size of the data?
For me it looks more that orc or parquet would be enough.
I do not see here specific in-memory requirements.
> On 12. Jun 2017, at 09:59, Ishan Jain <ja...@gmail.com> wrote:
>
> I need to just get the price of a stock which is stored in hdfs with timestamp and make a graph with the prices of that stock over time.
>
>> On Mon, Jun 12, 2017 at 1:03 PM, Jörn Franke <jo...@gmail.com> wrote:
>> First you need the user requirements - without them answering your questions will be difficult
>>
>> > On 12. Jun 2017, at 07:08, ishan-jain <ja...@gmail.com> wrote:
>> >
>> > I am new to BIG Data .Just been working for a month.
>> > I have HDFS data of stock prices. I need to perform data analysis(maybe some
>> > ML) and visualizations(Graphs and charts). For that i need Mapreduce
>> > functions. Which approach should i use?
>> > 1. Stream data from IGFS into ignite cache and work on it?
>> > 2. Use Hive with Tez and LLap function.(Should i use it with ignite or
>> > independent and directly on HDFS. No info available on the net.)
>> > 3. Use presto (Which is the better variant?(Hive or presto))
>> > 4. Some other fast way with IGFS if possible.
>> > 5. Also which open source tools should i use to accomplish this.
>> > Any help would be appreciated.
>> >
>> >
>> >
>> > --
>> > View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Data-Analysis-and-visualization-tp13614.html
>> > Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
Re: Data Analysis and visualization
Posted by Ishan Jain <ja...@gmail.com>.
I need to just get the price of a stock which is stored in hdfs with
timestamp and make a graph with the prices of that stock over time.
On Mon, Jun 12, 2017 at 1:03 PM, Jörn Franke <jo...@gmail.com> wrote:
> First you need the user requirements - without them answering your
> questions will be difficult
>
> > On 12. Jun 2017, at 07:08, ishan-jain <ja...@gmail.com> wrote:
> >
> > I am new to BIG Data .Just been working for a month.
> > I have HDFS data of stock prices. I need to perform data analysis(maybe
> some
> > ML) and visualizations(Graphs and charts). For that i need Mapreduce
> > functions. Which approach should i use?
> > 1. Stream data from IGFS into ignite cache and work on it?
> > 2. Use Hive with Tez and LLap function.(Should i use it with ignite or
> > independent and directly on HDFS. No info available on the net.)
> > 3. Use presto (Which is the better variant?(Hive or presto))
> > 4. Some other fast way with IGFS if possible.
> > 5. Also which open source tools should i use to accomplish this.
> > Any help would be appreciated.
> >
> >
> >
> > --
> > View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/Data-Analysis-and-visualization-tp13614.html
> > Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
Re: Data Analysis and visualization
Posted by ishan-jain <ja...@gmail.com>.
I need to basically have a sql query remote access from tools like tableau or
zeppelin and have fast mapreduce funtions
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Data-Analysis-and-visualization-tp13614p13639.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Data Analysis and visualization
Posted by Jörn Franke <jo...@gmail.com>.
First you need the user requirements - without them answering your questions will be difficult
> On 12. Jun 2017, at 07:08, ishan-jain <ja...@gmail.com> wrote:
>
> I am new to BIG Data .Just been working for a month.
> I have HDFS data of stock prices. I need to perform data analysis(maybe some
> ML) and visualizations(Graphs and charts). For that i need Mapreduce
> functions. Which approach should i use?
> 1. Stream data from IGFS into ignite cache and work on it?
> 2. Use Hive with Tez and LLap function.(Should i use it with ignite or
> independent and directly on HDFS. No info available on the net.)
> 3. Use presto (Which is the better variant?(Hive or presto))
> 4. Some other fast way with IGFS if possible.
> 5. Also which open source tools should i use to accomplish this.
> Any help would be appreciated.
>
>
>
> --
> View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Data-Analysis-and-visualization-tp13614.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.