You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@hop.apache.org by "cwhommes (via GitHub)" <gi...@apache.org> on 2025/01/14 15:45:48 UTC
[D] Downloading files with variable filenames based on current date (hop)
GitHub user cwhommes created a discussion: Downloading files with variable filenames based on current date
I would appreciate someone pointing me in the right direction. I need to download seven files off a webserver daily. The names of these files contain the date they were generated (for example 20250114-customer.csv). The files are generated daily.
How would I go about setting up a workflow and pipeline where the name of the file is dependend on the current date (TODAY())? So for example, I start retrieving yesterday's file and subsequently the files of the six days before?
GitHub link: https://github.com/apache/hop/discussions/4778
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Downloading files with variable filenames based on current date (hop)
Posted by "cwhommes (via GitHub)" <gi...@apache.org>.
GitHub user cwhommes edited a comment on the discussion: Downloading files with variable filenames based on current date
Hi Bart, the files I need to access are stored on a webserver that I can only access through the internet ('get http://.../file.csv'). Right now I use a HTTP action to download a file to local storage. From what I understand from the Hop documentation the VFS is specifically for Google, Dropbox, s3 etc. Or am I missing something?
I will experiment with the variables and see if I can make it work and report back. Thanks for your help.
GitHub link: https://github.com/apache/hop/discussions/4778#discussioncomment-11835556
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Downloading files with variable filenames based on current date (hop)
Posted by "bamaer (via GitHub)" <gi...@apache.org>.
GitHub user bamaer added a comment to the discussion: Downloading files with variable filenames based on current date
VFS works with the file systems in the list in the doc page I shared. S3, Google Storage, Dropbox etc are Apache Hop additions, but VFS (part of the Apache Commons project) supports a large number of file systems out of the box.
GitHub link: https://github.com/apache/hop/discussions/4778#discussioncomment-11841055
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Downloading files with variable filenames based on current date (hop)
Posted by "bamaer (via GitHub)" <gi...@apache.org>.
GitHub user bamaer added a comment to the discussion: Downloading files with variable filenames based on current date
you didn't specify any details about what this web server would look like, but [VFS](https://hop.apache.org//manual/latest/vfs.html#_apache_vfs_file_system_types) supports a set of file system types that let you download files directly. VFS is supported in most workflow actions and pipeline transforms, e.g the "copy files" action to copy file locally, or the "text file input" transform to read file contents directly, without the need to download.
You can include your date in the correct format as a variable or workflow or pipeline parameter.
GitHub link: https://github.com/apache/hop/discussions/4778#discussioncomment-11834839
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Downloading files with variable filenames based on current date (hop)
Posted by "cwhommes (via GitHub)" <gi...@apache.org>.
GitHub user cwhommes edited a comment on the discussion: Downloading files with variable filenames based on current date
Hi Bart, the files I need to access are stored on a webserver that I can only access through the internet. Right now I use a HTTP action to download a file to local storage. From what I understand from the Hop documentation the VFS is specifically for Google, Dropbox, s3 etc. Or am I missing something?
I will experiment with the variables and see if I can make it work and report back. Thanks for your help.
GitHub link: https://github.com/apache/hop/discussions/4778#discussioncomment-11835556
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Downloading files with variable filenames based on current date (hop)
Posted by "cwhommes (via GitHub)" <gi...@apache.org>.
GitHub user cwhommes edited a discussion: Downloading files with variable filenames based on current date
I would appreciate someone pointing me in the right direction. I need to download seven files off a webserver daily. The names of these files contain the date on which they were generated (for example 20250114-customer.csv). The files are generated daily.
How would I go about setting up a workflow and pipeline where the name of the file is dependend on the current date (TODAY())? So for example, I start retrieving yesterday's file and subsequently the files of the six days before?
GitHub link: https://github.com/apache/hop/discussions/4778
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Downloading files with variable filenames based on current date (hop)
Posted by "cwhommes (via GitHub)" <gi...@apache.org>.
GitHub user cwhommes added a comment to the discussion: Downloading files with variable filenames based on current date
Hi Bart, the files I need to access are stored on a webserver. Right now I use a HTTP action to download a file to local storage. From what I understand from the Hop documentation the VFS is specifically for Google, Dropbox, s3 etc. Or am I missing something?
I will experiment with the variables and see if I can make it work and report back. Thanks for your help.
GitHub link: https://github.com/apache/hop/discussions/4778#discussioncomment-11835556
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org