You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@arrow.apache.org by Priyanshu Mittal <pr...@gmail.com> on 2021/02/19 06:15:18 UTC
How to connect arrow flights to spark clusters
Hi,
I want to connect my arrow flight with apache spark clusters so that I can
transfer my data between spark clusters and arrow flights. Basically, I
have to transfer my arrow data to spark clusters using arrow flights. So,
How can I do that?
Any help or resources would be appreciated.
Thanks,
Priyanshu
Re: How to connect arrow flights to spark clusters
Posted by Priyanshu Mittal <pr...@gmail.com>.
Ok, thanks for the clarification.
On Tue, 2 Mar, 2021, 11:30 pm Micah Kornfield, <em...@gmail.com>
wrote:
> It is unofficial from the perspective of the Arrow project., I pointed to
> it as an example.
>
> On Tue, Mar 2, 2021 at 9:08 AM Priyanshu Mittal <pr...@gmail.com>
> wrote:
>
>> One more query here,
>> Although I'm still not able to figure out how to build and run the code
>> in this repo https://github.com/rymurr/flight-spark-source but hopefully
>> if it works, then I would like to know that is the code in this repo for
>> flight spark connector is some sample code written by an individual or it
>> is official spark flight connector code supported by arrow or some other
>> community?
>>
>> Thanks,
>> Priyanshu
>>
>> On Tue, 2 Mar, 2021, 8:52 am Priyanshu Mittal, <pr...@gmail.com>
>> wrote:
>>
>>> Hi Micah,
>>>
>>> I tried to understand this repo
>>> https://github.com/rymurr/flight-spark-source but I'm not able to build
>>> and run this code. Readme file also doesn't explain how to run this code..
>>> So, can anyone help me in running this code? And also is there any
>>> documentation related to it?
>>>
>>> Further, I found very less resources for usage of Arrow flight and like
>>> how we can use it in our project. So If there are any implementations of
>>> Arrow flight or Spark Flight connector, It would be great if you can share
>>> that with us.
>>>
>>> Thanks,
>>> Priyanshu
>>>
>>> On Sun, Feb 28, 2021 at 10:40 PM Priyanshu Mittal <
>>> prmittal.nch@gmail.com> wrote:
>>>
>>>> Hi,
>>>> Thanks for sharing that. I'm looking into it.
>>>>
>>>> On Sat, 27 Feb, 2021, 10:21 am Micah Kornfield, <em...@gmail.com>
>>>> wrote:
>>>>
>>>>> Ryan Murray prototyped something like this a while ago [1]. I'm not
>>>>> sure what the current state of this is.
>>>>>
>>>>> [1] https://github.com/rymurr/flight-spark-source
>>>>>
>>>>> On Thu, Feb 18, 2021 at 10:15 PM Priyanshu Mittal <
>>>>> prmittal.nch@gmail.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>> I want to connect my arrow flight with apache spark clusters so that
>>>>>> I can transfer my data between spark clusters and arrow flights. Basically,
>>>>>> I have to transfer my arrow data to spark clusters using arrow flights. So,
>>>>>> How can I do that?
>>>>>> Any help or resources would be appreciated.
>>>>>>
>>>>>> Thanks,
>>>>>> Priyanshu
>>>>>>
>>>>>
Re: How to connect arrow flights to spark clusters
Posted by Micah Kornfield <em...@gmail.com>.
It is unofficial from the perspective of the Arrow project., I pointed to
it as an example.
On Tue, Mar 2, 2021 at 9:08 AM Priyanshu Mittal <pr...@gmail.com>
wrote:
> One more query here,
> Although I'm still not able to figure out how to build and run the code in
> this repo https://github.com/rymurr/flight-spark-source but hopefully if
> it works, then I would like to know that is the code in this repo for
> flight spark connector is some sample code written by an individual or it
> is official spark flight connector code supported by arrow or some other
> community?
>
> Thanks,
> Priyanshu
>
> On Tue, 2 Mar, 2021, 8:52 am Priyanshu Mittal, <pr...@gmail.com>
> wrote:
>
>> Hi Micah,
>>
>> I tried to understand this repo
>> https://github.com/rymurr/flight-spark-source but I'm not able to build
>> and run this code. Readme file also doesn't explain how to run this code..
>> So, can anyone help me in running this code? And also is there any
>> documentation related to it?
>>
>> Further, I found very less resources for usage of Arrow flight and like
>> how we can use it in our project. So If there are any implementations of
>> Arrow flight or Spark Flight connector, It would be great if you can share
>> that with us.
>>
>> Thanks,
>> Priyanshu
>>
>> On Sun, Feb 28, 2021 at 10:40 PM Priyanshu Mittal <pr...@gmail.com>
>> wrote:
>>
>>> Hi,
>>> Thanks for sharing that. I'm looking into it.
>>>
>>> On Sat, 27 Feb, 2021, 10:21 am Micah Kornfield, <em...@gmail.com>
>>> wrote:
>>>
>>>> Ryan Murray prototyped something like this a while ago [1]. I'm not
>>>> sure what the current state of this is.
>>>>
>>>> [1] https://github.com/rymurr/flight-spark-source
>>>>
>>>> On Thu, Feb 18, 2021 at 10:15 PM Priyanshu Mittal <
>>>> prmittal.nch@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>> I want to connect my arrow flight with apache spark clusters so that I
>>>>> can transfer my data between spark clusters and arrow flights. Basically, I
>>>>> have to transfer my arrow data to spark clusters using arrow flights. So,
>>>>> How can I do that?
>>>>> Any help or resources would be appreciated.
>>>>>
>>>>> Thanks,
>>>>> Priyanshu
>>>>>
>>>>
Re: How to connect arrow flights to spark clusters
Posted by Priyanshu Mittal <pr...@gmail.com>.
One more query here,
Although I'm still not able to figure out how to build and run the code in
this repo https://github.com/rymurr/flight-spark-source but hopefully if it
works, then I would like to know that is the code in this repo for flight
spark connector is some sample code written by an individual or it is
official spark flight connector code supported by arrow or some other
community?
Thanks,
Priyanshu
On Tue, 2 Mar, 2021, 8:52 am Priyanshu Mittal, <pr...@gmail.com>
wrote:
> Hi Micah,
>
> I tried to understand this repo
> https://github.com/rymurr/flight-spark-source but I'm not able to build
> and run this code. Readme file also doesn't explain how to run this code..
> So, can anyone help me in running this code? And also is there any
> documentation related to it?
>
> Further, I found very less resources for usage of Arrow flight and like
> how we can use it in our project. So If there are any implementations of
> Arrow flight or Spark Flight connector, It would be great if you can share
> that with us.
>
> Thanks,
> Priyanshu
>
> On Sun, Feb 28, 2021 at 10:40 PM Priyanshu Mittal <pr...@gmail.com>
> wrote:
>
>> Hi,
>> Thanks for sharing that. I'm looking into it.
>>
>> On Sat, 27 Feb, 2021, 10:21 am Micah Kornfield, <em...@gmail.com>
>> wrote:
>>
>>> Ryan Murray prototyped something like this a while ago [1]. I'm not
>>> sure what the current state of this is.
>>>
>>> [1] https://github.com/rymurr/flight-spark-source
>>>
>>> On Thu, Feb 18, 2021 at 10:15 PM Priyanshu Mittal <
>>> prmittal.nch@gmail.com> wrote:
>>>
>>>> Hi,
>>>> I want to connect my arrow flight with apache spark clusters so that I
>>>> can transfer my data between spark clusters and arrow flights. Basically, I
>>>> have to transfer my arrow data to spark clusters using arrow flights. So,
>>>> How can I do that?
>>>> Any help or resources would be appreciated.
>>>>
>>>> Thanks,
>>>> Priyanshu
>>>>
>>>
Re: How to connect arrow flights to spark clusters
Posted by Priyanshu Mittal <pr...@gmail.com>.
Hi Micah,
I tried to understand this repo
https://github.com/rymurr/flight-spark-source but I'm not able to build and
run this code. Readme file also doesn't explain how to run this code.. So,
can anyone help me in running this code? And also is there any
documentation related to it?
Further, I found very less resources for usage of Arrow flight and like how
we can use it in our project. So If there are any implementations of Arrow
flight or Spark Flight connector, It would be great if you can share that
with us.
Thanks,
Priyanshu
On Sun, Feb 28, 2021 at 10:40 PM Priyanshu Mittal <pr...@gmail.com>
wrote:
> Hi,
> Thanks for sharing that. I'm looking into it.
>
> On Sat, 27 Feb, 2021, 10:21 am Micah Kornfield, <em...@gmail.com>
> wrote:
>
>> Ryan Murray prototyped something like this a while ago [1]. I'm not sure
>> what the current state of this is.
>>
>> [1] https://github.com/rymurr/flight-spark-source
>>
>> On Thu, Feb 18, 2021 at 10:15 PM Priyanshu Mittal <pr...@gmail.com>
>> wrote:
>>
>>> Hi,
>>> I want to connect my arrow flight with apache spark clusters so that I
>>> can transfer my data between spark clusters and arrow flights. Basically, I
>>> have to transfer my arrow data to spark clusters using arrow flights. So,
>>> How can I do that?
>>> Any help or resources would be appreciated.
>>>
>>> Thanks,
>>> Priyanshu
>>>
>>
Re: How to connect arrow flights to spark clusters
Posted by Priyanshu Mittal <pr...@gmail.com>.
Hi,
Thanks for sharing that. I'm looking into it.
On Sat, 27 Feb, 2021, 10:21 am Micah Kornfield, <em...@gmail.com>
wrote:
> Ryan Murray prototyped something like this a while ago [1]. I'm not sure
> what the current state of this is.
>
> [1] https://github.com/rymurr/flight-spark-source
>
> On Thu, Feb 18, 2021 at 10:15 PM Priyanshu Mittal <pr...@gmail.com>
> wrote:
>
>> Hi,
>> I want to connect my arrow flight with apache spark clusters so that I
>> can transfer my data between spark clusters and arrow flights. Basically, I
>> have to transfer my arrow data to spark clusters using arrow flights. So,
>> How can I do that?
>> Any help or resources would be appreciated.
>>
>> Thanks,
>> Priyanshu
>>
>
Re: How to connect arrow flights to spark clusters
Posted by Micah Kornfield <em...@gmail.com>.
Ryan Murray prototyped something like this a while ago [1]. I'm not sure
what the current state of this is.
[1] https://github.com/rymurr/flight-spark-source
On Thu, Feb 18, 2021 at 10:15 PM Priyanshu Mittal <pr...@gmail.com>
wrote:
> Hi,
> I want to connect my arrow flight with apache spark clusters so that I can
> transfer my data between spark clusters and arrow flights. Basically, I
> have to transfer my arrow data to spark clusters using arrow flights. So,
> How can I do that?
> Any help or resources would be appreciated.
>
> Thanks,
> Priyanshu
>