[Pyspark] How to download Zip file from SFTP location and put in into Azure Data Lake and unzip it

[Heta Desai <he...@1rivet.com>]

Hello,

I have zip files on SFTP location. I want to download/copy those files and put into Azure Data Lake. Once the zip files get stored into Azure Data Lake, I want to unzip those files and read using Data Frames.

The file format inside zip is excel. SO, once files are unzipped, I want to read excel files using spark DataFrames.

Please help me with the solution as soon as possible.


Thanks,
[cid:1885f47f-6a84-4099-9a4f-eabe570a41ac]      ​Heta Desai | Data | Sr Associate L1
e.heta.desai@1rivet.com<ma...@1rivet.com> | t. +91 966.225.4954
​

This email, including attachments, may include confidential and/or proprietary information, and may be used only by the person or entity to which it is addressed. If you are not the intended recipient, please advise the sender immediately and delete this message and any attachments. Unless otherwise specifically stated in this email, transaction related information in this email, including attachments, is not to be construed as an offer, solicitation or the basis or confirmation for any contract for the purchase/sale of any services. All email sent to or from this address will be received by 1Rivet US, Inc and is subject to archival retention and review by someone other than the recipient.

Re: [Pyspark] How to download Zip file from SFTP location and put in into Azure Data Lake and unzip it

[Wes Peng <we...@stackdev.eu>]

How large is the file? From my experience, reading the excel file from 
data lake and loading as dataframe, works great.

Thanks

On 2022-01-18 22:16, Heta Desai wrote:
> Hello,
> 
>  I have zip files on SFTP location. I want to download/copy those
> files and put into Azure Data Lake. Once the zip files get stored into
> Azure Data Lake, I want to unzip those files and read using Data
> Frames.
> 
>  The file format inside zip is excel. SO, once files are unzipped, I
> want to read excel files using spark DataFrames.
> 
>  Please help me with the solution as soon as possible.
> 
>  Thanks,
> 
>  ​Heta Desai | Data | Sr Associate L1
> e.heta.desai@1rivet.com | t. +91 966.225.4954
> 
>  ​
> 
>  This email, including attachments, may include confidential and/or
> proprietary information, and may be used only by the person or entity
> to which it is addressed. If you are not the intended recipient,
> please advise the sender immediately and delete this message and any
> attachments. Unless otherwise specifically stated in this email,
> transaction related information in this email, including attachments,
> is not to be construed as an offer, solicitation or the basis or
> confirmation for any contract for the purchase/sale of any services.
> All email sent to or from this address will be received by 1Rivet US,
> Inc and is subject to archival retention and review by someone other
> than the recipient.

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org