You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hop.apache.org by GitBox <gi...@apache.org> on 2022/12/13 22:56:41 UTC
[GitHub] [hop] usbrandon opened a new issue, #2059: [Feature Request]: File Metadata - Add support for compressed csv (gzip, etc)
usbrandon opened a new issue, #2059:
URL: https://github.com/apache/hop/issues/2059
### What would you like to happen?
The Text File Input and Output transforms support compression. It would be wonderful if this step supported reading compressed CSV files.
Use case:
A terrible system allows me to download CSV reports, but sometimes they add columns to existing reports.
I have a python script that downloads the CSVs and compresses them to save on AWS costs (transit/IO/etc). I would like to use this transform to look for the columns and any changes and inject them into the Text File Input Step so it can read the new layout, and also a Select Transform to ignore the new columns until they are vetted.
### Issue Priority
Priority: 3
### Issue Component
Component: Transforms
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: File Metadata - Add support for compressed csv (gzip, etc) (hop)
Posted by "hansva (via GitHub)" <gi...@apache.org>.
hansva closed issue #2059: [Feature Request]: File Metadata - Add support for compressed csv (gzip, etc)
URL: https://github.com/apache/hop/issues/2059
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: File Metadata - Add support for compressed csv (gzip, etc) (hop)
Posted by "sramazzina (via GitHub)" <gi...@apache.org>.
sramazzina commented on issue #2059:
URL: https://github.com/apache/hop/issues/2059#issuecomment-5455472485
Hi @usbrandon, thanks for the patience on this one — it's been open for a while.
Good news: this already works today, thanks to Apache VFS. Hop registers layered file system providers for `gz`, `zip`, `tar`, `tgz` and `bz2`, so File Metadata can read a compressed CSV without any change to the transform. You just point it at the file with a layered URI instead of a plain path.
For gzip — which is probably what your Python script produces:
```
gz:file:///path/to/report.csv.gz!/report.csv
```
For a CSV inside a zip archive:
```
zip:file:///path/to/report.zip!/report.csv
```
The part after `!/` is the name of the file *inside* the archive. For zip it has to match exactly — it's case sensitive, and if the archive was created with a folder structure you need the full internal path (`!/data/report.csv`). `unzip -l yourfile.zip` shows you the exact name to use. Gzip is more forgiving, since a gzip stream only ever contains one file.
We tested this against the transform's actual read path (charset detection and delimiter detection) and the results on a gzipped and a zipped CSV are identical to the ones on the plain file — same charset, same delimiter, same field types. It also works when the file name comes from an incoming field, so it fits an injection setup like the one you described: File Metadata reads the compressed file, and the Text File Input downstream handles the decompression itself with its own Compression setting.
Two rough edges worth knowing about:
- If you forget the `!/name` part and pass just the archive, you get `... !/ because it is not a file`, which isn't very self-explanatory — it means "tell me which file inside the archive you want".
- If the internal name doesn't match, the transform doesn't fail: it emits an empty row and moves on, so it looks like nothing happened.
Since compression itself is already covered, we'll use this issue to track a small fix limited to those two cases: a clearer message when an archive is passed without naming a file inside it, and a warning in the log instead of a silently empty row when the file can't be found. Nothing else changes in the transform.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: File Metadata - Add support for compressed csv (gzip, etc) (hop)
Posted by "sramazzina (via GitHub)" <gi...@apache.org>.
sramazzina commented on issue #2059:
URL: https://github.com/apache/hop/issues/2059#issuecomment-5453038247
.take-issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: File Metadata - Add support for compressed csv (gzip, etc) (hop)
Posted by "usbrandon (via GitHub)" <gi...@apache.org>.
usbrandon commented on issue #2059:
URL: https://github.com/apache/hop/issues/2059#issuecomment-5456053596
Thank you for the thoughtful contribution and follow up. I appreciate you.
On Fri, Aug 28, 2026 at 12:11 Sergio Ramazzina ***@***.***>
wrote:
> *sramazzina* left a comment (apache/hop#2059)
> <https://github.com/apache/hop/issues/2059#issuecomment-5455472485>
>
> Hi @usbrandon <https://github.com/usbrandon>, thanks for the patience on
> this one — it's been open for a while.
>
> Good news: this already works today, thanks to Apache VFS. Hop registers
> layered file system providers for gz, zip, tar, tgz and bz2, so File
> Metadata can read a compressed CSV without any change to the transform. You
> just point it at the file with a layered URI instead of a plain path.
>
> For gzip — which is probably what your Python script produces:
>
> gz:file:///path/to/report.csv.gz!/report.csv
>
> For a CSV inside a zip archive:
>
> zip:file:///path/to/report.zip!/report.csv
>
> The part after !/ is the name of the file *inside* the archive. For zip
> it has to match exactly — it's case sensitive, and if the archive was
> created with a folder structure you need the full internal path (
> !/data/report.csv). unzip -l yourfile.zip shows you the exact name to
> use. Gzip is more forgiving, since a gzip stream only ever contains one
> file.
>
> We tested this against the transform's actual read path (charset detection
> and delimiter detection) and the results on a gzipped and a zipped CSV are
> identical to the ones on the plain file — same charset, same delimiter,
> same field types. It also works when the file name comes from an incoming
> field, so it fits an injection setup like the one you described: File
> Metadata reads the compressed file, and the Text File Input downstream
> handles the decompression itself with its own Compression setting.
>
> Two rough edges worth knowing about:
>
> - If you forget the !/name part and pass just the archive, you get ...
> !/ because it is not a file, which isn't very self-explanatory — it
> means "tell me which file inside the archive you want".
> - If the internal name doesn't match, the transform doesn't fail: it
> emits an empty row and moves on, so it looks like nothing happened.
>
> Since compression itself is already covered, we'll use this issue to track
> a small fix limited to those two cases: a clearer message when an archive
> is passed without naming a file inside it, and a warning in the log instead
> of a silently empty row when the file can't be found. Nothing else changes
> in the transform.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/hop/issues/2059?email_source=notifications&email_token=AAJNF5SGDWSI64NPAELD2SL5MG4KRA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKNBVGU2DOMRUHA22M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLDGN5XXIZLSL5RWY2LDNM#issuecomment-5455472485>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAJNF5SGLKTQKRCNHCVPCMT5MG4KRAVCNFSNUABFKJSXA33TNF2G64TZHMZDCMBVG44TAMRRHNEXG43VMU5TCNBZGUZDSMBRGMZKC5QC>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org