You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by "Munoz, Robert" <Ro...@pimco.com> on 2019/10/24 23:40:39 UTC
RE: Support for hdfs caching
Hello,
With the removal of Hadoop accelerator from ignite binaries, are there any other options for using HDFS as persistence store?
Thanks,
Robert Munoz
This message contains confidential information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute, alter or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmissions cannot be guaranteed to be secure or without error as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender, therefore, does not accept liability for any errors or omissions in the contents of this message which arise during or as a result of e-mail transmission. If verification is required, please request a hard-copy version. This message is provided for information purposes and should not be construed as a solicitation or offer to buy or sell any securities or related financial instruments in any jurisdiction. Securities are offered in the U.S. through PIMCO Investments LLC, distributor and a company of PIMCO LLC.
The individual providing the information herein is an employee of Pacific Investment Management Company LLC ("PIMCO"), an SEC-registered investment adviser. To the extent such individual advises you regarding a PIMCO investment strategy, he or she does so as an associated person of PIMCO. To the extent that any information is provided to you related to a PIMCO-sponsored investment fund ("PIMCO Fund"), it is being provided to you in the individual's capacity as a registered representative of PIMCO Investments LLC ("PI"), an SEC-registered broker-dealer. PI is not registered, and does not intend to register, as a municipal advisor and therefore does not provide advice with respect to the investment of the proceeds of municipal securities or municipal escrow investments. In addition, unless otherwise agreed by PIMCO, this communication and any related attachments are being provided on the express basis that they will not cause PIMCO LLC, or its affiliates, to become an investment advice fiduciary under ERISA or the Internal Revenue Code.
Re: Support for hdfs caching
Posted by Denis Magda <dm...@apache.org>.
Hi Robert,
The architecture [1] suggested here is an ideal one. You deploy Ignite
separately as an in-memory store and use one of the replication techniques
to update the HDFS cluster:
* Hadoop Connector form [1].
* Implement a custom version of a CacheStore [2] interface that will push
all the changes from Ignite to Hadoop.
Does this work for you?
[1]
https://www.gridgain.com/docs/latest/integrations/datalake-accelerator/getting-started
[2]
https://apacheignite.readme.io/docs/3rd-party-store#section-custom-cachestore
-
Denis
On Fri, Oct 25, 2019 at 8:21 AM Munoz, Robert <Ro...@pimco.com>
wrote:
> Hi Denis,
>
>
>
> Thank you for responding. The use case is to deploy ignite cluster on our
> Hadoop cluster nodes and make use of persistence to HDFS since the data
> disks have already been allocated for HDFS. I am new to ignite so am
> unfamiliar with its file/folder structure but the goal would be to use
> ignite as in memory cache for sql tables and to persist the data to HDFS as
> write through / read through . This should not impact the current HDFS
> configuration as we have users/applications using HDFS already. The
> implementation should be standard sql and not hbase get/put operations. Let
> me know if this helps.
>
>
>
>
>
>
>
> Thanks,
>
>
>
> Robert Munoz
>
> 949-220-5169
>
> https://pimco.webex.com/meet/robert.munoz
>
>
>
>
>
>
>
> *From:* Denis Magda <dm...@apache.org>
> *Sent:* Thursday, October 24, 2019 10:08 PM
> *To:* user@ignite.apache.org
> *Subject:* Re: Support for hdfs caching
>
>
>
> Robert,
>
>
>
> Could you please describe your use case? There several options that might
> suit your needs.
>
>
>
> Denis
>
> On Thursday, October 24, 2019, Munoz, Robert <Ro...@pimco.com>
> wrote:
>
> Hello,
>
>
>
> With the removal of Hadoop accelerator from ignite binaries, are there any
> other options for using HDFS as persistence store?
>
>
>
>
>
> Thanks,
>
>
>
> Robert Munoz
>
>
>
> This message contains confidential information and is intended only for
> the individual named. If you are not the named addressee, you should not
> disseminate, distribute, alter or copy this e-mail. Please notify the
> sender immediately by e-mail if you have received this e-mail by mistake
> and delete this e-mail from your system. E-mail transmissions cannot be
> guaranteed to be secure or without error as information could be
> intercepted, corrupted, lost, destroyed, arrive late or incomplete, or
> contain viruses. The sender, therefore, does not accept liability for any
> errors or omissions in the contents of this message which arise during or
> as a result of e-mail transmission. If verification is required, please
> request a hard-copy version. This message is provided for information
> purposes and should not be construed as a solicitation or offer to buy or
> sell any securities or related financial instruments in any jurisdiction.
> Securities are offered in the U.S. through PIMCO Investments LLC,
> distributor and a company of PIMCO LLC.
>
> The individual providing the information herein is an employee of Pacific
> Investment Management Company LLC ("PIMCO"), an SEC-registered investment
> adviser. To the extent such individual advises you regarding a PIMCO
> investment strategy, he or she does so as an associated person of PIMCO.
> To the extent that any information is provided to you related to a
> PIMCO-sponsored investment fund ("PIMCO Fund"), it is being provided to you
> in the individual's capacity as a registered representative of PIMCO
> Investments LLC ("PI"), an SEC-registered broker-dealer. PI is not
> registered, and does not intend to register, as a municipal advisor and
> therefore does not provide advice with respect to the investment of the
> proceeds of municipal securities or municipal escrow investments. In
> addition, unless otherwise agreed by PIMCO, this communication and any
> related attachments are being provided on the express basis that they will
> not cause PIMCO LLC, or its affiliates, to become an investment advice
> fiduciary under ERISA or the Internal Revenue Code.
>
>
>
> --
>
> -
>
> Denis
>
>
>
RE: Support for hdfs caching
Posted by "Munoz, Robert" <Ro...@pimco.com>.
Hi Denis,
Thank you for responding. The use case is to deploy ignite cluster on our Hadoop cluster nodes and make use of persistence to HDFS since the data disks have already been allocated for HDFS. I am new to ignite so am unfamiliar with its file/folder structure but the goal would be to use ignite as in memory cache for sql tables and to persist the data to HDFS as write through / read through . This should not impact the current HDFS configuration as we have users/applications using HDFS already. The implementation should be standard sql and not hbase get/put operations. Let me know if this helps.
Thanks,
Robert Munoz
949-220-5169
https://pimco.webex.com/meet/robert.munoz
From: Denis Magda <dm...@apache.org>
Sent: Thursday, October 24, 2019 10:08 PM
To: user@ignite.apache.org
Subject: Re: Support for hdfs caching
Robert,
Could you please describe your use case? There several options that might suit your needs.
Denis
On Thursday, October 24, 2019, Munoz, Robert <Ro...@pimco.com>> wrote:
Hello,
With the removal of Hadoop accelerator from ignite binaries, are there any other options for using HDFS as persistence store?
Thanks,
Robert Munoz
This message contains confidential information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute, alter or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmissions cannot be guaranteed to be secure or without error as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender, therefore, does not accept liability for any errors or omissions in the contents of this message which arise during or as a result of e-mail transmission. If verification is required, please request a hard-copy version. This message is provided for information purposes and should not be construed as a solicitation or offer to buy or sell any securities or related financial instruments in any jurisdiction. Securities are offered in the U.S. through PIMCO Investments LLC, distributor and a company of PIMCO LLC.
The individual providing the information herein is an employee of Pacific Investment Management Company LLC ("PIMCO"), an SEC-registered investment adviser. To the extent such individual advises you regarding a PIMCO investment strategy, he or she does so as an associated person of PIMCO. To the extent that any information is provided to you related to a PIMCO-sponsored investment fund ("PIMCO Fund"), it is being provided to you in the individual's capacity as a registered representative of PIMCO Investments LLC ("PI"), an SEC-registered broker-dealer. PI is not registered, and does not intend to register, as a municipal advisor and therefore does not provide advice with respect to the investment of the proceeds of municipal securities or municipal escrow investments. In addition, unless otherwise agreed by PIMCO, this communication and any related attachments are being provided on the express basis that they will not cause PIMCO LLC, or its affiliates, to become an investment advice fiduciary under ERISA or the Internal Revenue Code.
--
-
Denis
Re: Support for hdfs caching
Posted by Denis Magda <dm...@apache.org>.
Robert,
Could you please describe your use case? There several options that might
suit your needs.
Denis
On Thursday, October 24, 2019, Munoz, Robert <Ro...@pimco.com> wrote:
> Hello,
>
>
>
> With the removal of Hadoop accelerator from ignite binaries, are there any
> other options for using HDFS as persistence store?
>
>
>
>
>
> Thanks,
>
>
>
> Robert Munoz
>
>
>
> This message contains confidential information and is intended only for
> the individual named. If you are not the named addressee, you should not
> disseminate, distribute, alter or copy this e-mail. Please notify the
> sender immediately by e-mail if you have received this e-mail by mistake
> and delete this e-mail from your system. E-mail transmissions cannot be
> guaranteed to be secure or without error as information could be
> intercepted, corrupted, lost, destroyed, arrive late or incomplete, or
> contain viruses. The sender, therefore, does not accept liability for any
> errors or omissions in the contents of this message which arise during or
> as a result of e-mail transmission. If verification is required, please
> request a hard-copy version. This message is provided for information
> purposes and should not be construed as a solicitation or offer to buy or
> sell any securities or related financial instruments in any jurisdiction.
> Securities are offered in the U.S. through PIMCO Investments LLC,
> distributor and a company of PIMCO LLC.
>
> The individual providing the information herein is an employee of Pacific
> Investment Management Company LLC ("PIMCO"), an SEC-registered investment
> adviser. To the extent such individual advises you regarding a PIMCO
> investment strategy, he or she does so as an associated person of PIMCO.
> To the extent that any information is provided to you related to a
> PIMCO-sponsored investment fund ("PIMCO Fund"), it is being provided to you
> in the individual's capacity as a registered representative of PIMCO
> Investments LLC ("PI"), an SEC-registered broker-dealer. PI is not
> registered, and does not intend to register, as a municipal advisor and
> therefore does not provide advice with respect to the investment of the
> proceeds of municipal securities or municipal escrow investments. In
> addition, unless otherwise agreed by PIMCO, this communication and any
> related attachments are being provided on the express basis that they will
> not cause PIMCO LLC, or its affiliates, to become an investment advice
> fiduciary under ERISA or the Internal Revenue Code.
>
--
-
Denis