You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hop.apache.org by gi...@apache.org on 2021/03/19 11:28:18 UTC
[incubator-hop-docs] branch asf-site updated: Documentation updated
to e5f7e6d
This is an automated email from the ASF dual-hosted git repository.
git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/incubator-hop-docs.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 2e30bda Documentation updated to e5f7e6d
2e30bda is described below
commit 2e30bda382cf7733d68c33420cb165dbf19d5ba9
Author: jenkins <bu...@apache.org>
AuthorDate: Fri Mar 19 11:28:14 2021 +0000
Documentation updated to e5f7e6d
---
.../ROOT/pages/technology/google/google-info.adoc | 53 +---------------
.../modules/ROOT/pages/vfs/aws-s3-vfs.adoc | 60 ++++++++++++++++++
.../ROOT/pages/vfs/azure-blob-storage-vfs.adoc | 72 ++++++++++++++++++++++
.../ROOT/pages/vfs/google-cloud-storage-vfs.adoc | 54 ++++++++++++++++
.../google-info.adoc => vfs/google-drive-vfs.adoc} | 40 +++---------
5 files changed, 198 insertions(+), 81 deletions(-)
diff --git a/hop-user-manual/modules/ROOT/pages/technology/google/google-info.adoc b/hop-user-manual/modules/ROOT/pages/technology/google/google-info.adoc
index 99d04de..69c28be 100644
--- a/hop-user-manual/modules/ROOT/pages/technology/google/google-info.adoc
+++ b/hop-user-manual/modules/ROOT/pages/technology/google/google-info.adoc
@@ -33,58 +33,9 @@ It can be very convenient to save data into a Google sheet or indeed to read fro
== Google Drive
-Hop has a Virtual File System (VFS) plugin for Google Drive. You need to generate a credentials file to make it work. Follow the Google documentation to see how that is done. You also need to specify a folder in which security tokens are going to be saved. You can specify both in the Hop system configuration options. This can be done in the Hop GUI: go to the "Google Drive" tab in the Options dialog (from the Tools menu). You can also use the `hop-conf` script with the following options:
-
-[source,shell script]
-----
- -gdc, --google-drive-credentials-file=<credentialsFile>
- Configure the path to a Google Drive credentials JSON
- file
- -gdt, --google-drive-tokens-folder=<tokensFolder>
- Configure the path to a Google Drive tokens folder
-----
-
-Once done you will see a `googleDrive` entry in the central `hop-config.json` file:
-
-[source,json]
-----
-{
- "googleDrive" : {
- "credentialsFile" : "/path/to/google-drive-credentials.json",
- "tokensFolder" : "/path/to/tokens"
- }
-}
-----
-
-When you try to run the first time you'll see a message on the console where you executed Hop GUI or Hop Run saying something like:
-
-[source]
-----
-Please open the following address in your browser:
- https://accounts.google.com/o/oauth2/auth?access_type=offline&client_id=yourClientId&redirect_uri=http://localhost:8888/Callback&response_type=code&scope=https://www.googleapis.com/auth/drive
-----
-
-Open that URL in a browser and authenticate the given client ID. You'll then get a token in the configured `tokens folder` with which you can work.
+Hop has a Virtual File System (VFS) plugin for Google Drive. For more information see the VFS section of the documentation.
== Google Cloud Storage
-Hop has a Virtual File System (VFS) plugin for Google Cloud Storage.
-
-You need to generate a key file for a service account to make it work. Go to the Google Cloud console to do this. Once you have a key file for your service account, with permissions to access your GCP storage, point to it with either a system environment variable called `GOOGLE_APPLICATION_CREDENTIALS` (standard Google way of doing this) or in the Options dialog in the 'Google Cloud' tab. You can also use `hop-conf`:
-
-[source,shell script]
-----
- -gck, --google-cloud-service-account-key-file=<serviceAccountKeyFile>
- Configure the path to a Google Cloud service account JSON key file
-----
-
-Once done you will see a `googleCloud` entry in the central `hop-config.json` file:
+Hop has a Virtual File System (VFS) plugin for Google Cloud Storage. For more information see the VFS section of the documentation.
-[source,json]
-----
-{
- "googleCloud" : {
- "serviceAccountKeyFile" : "/path/to/your/google-key.json"
- }
-}
-----
diff --git a/hop-user-manual/modules/ROOT/pages/vfs/aws-s3-vfs.adoc b/hop-user-manual/modules/ROOT/pages/vfs/aws-s3-vfs.adoc
new file mode 100644
index 0000000..55675c6
--- /dev/null
+++ b/hop-user-manual/modules/ROOT/pages/vfs/aws-s3-vfs.adoc
@@ -0,0 +1,60 @@
+////
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+////
+
+:documentationPath: /vfs/
+:language: en_US
+
+= AWS S3
+
+== Scheme
+
+The scheme you can use to access your files in Azure Blob Storage is
+
+`**s3://**`
+
+== Configuration
+
+The configuration of the Amazon Web Services Simple Cloud Storage can be done through a variety of ways. Most require you to have an `Access Key` and a `Secret Key`.
+
+**Best practice is to create a specific IAM user for Apache Hop** so that if needed you can fine-tune the permissions (set it to read-only for example) or indeed delete the user if it's no longer needed.
+
+For a complete list see https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html[Working with credentials] in the AWS documentation.
+
+Below are 2 popular ways of configuring access.
+
+=== Credentials file
+
+You can create a file in your home folder in the `.aws/credentials` file which then contains content like this:
+
+[source,properties]
+----
+[default]
+aws_access_key_id = yourSecretKey
+aws_secret_access_key = a-long/series-of-characters-for-your-access-key
+----
+
+=== Variables
+
+You can set the following system environment variables:
+
+* `AWS_ACCESS_KEY_ID` : set it to your AWS access key
+* `AWS_SECRET_ACCESS_KEY` : set it to your secret access key
+
+== Usage and testing
+
+To test if the configuration works you can simply upload a small CSV file in an S3 bucket and then use File/Open in Hop GUI. Then you type in `s3://` as a file location and hit enter (or click the refresh button). Browse to the CSV file you uploaded and open it. If all is configured correctly you should be able to see the content in the Hop GUI.
+
diff --git a/hop-user-manual/modules/ROOT/pages/vfs/azure-blob-storage-vfs.adoc b/hop-user-manual/modules/ROOT/pages/vfs/azure-blob-storage-vfs.adoc
new file mode 100644
index 0000000..c3cb865
--- /dev/null
+++ b/hop-user-manual/modules/ROOT/pages/vfs/azure-blob-storage-vfs.adoc
@@ -0,0 +1,72 @@
+////
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+////
+
+:documentationPath: /vfs/
+:language: en_US
+
+= Azure Storage
+
+== Scheme
+
+The scheme you can use to access your files in Azure Blob Storage is
+
+`**azure://**`
+
+== Configuration
+
+To get access to your Azure storage files you need to configure a few things:
+
+* The name of your storage account in Azure
+* The key of your storage account in Azure
+
+You can find both in the Storage Accounts section of your Azure portal.
+
+* A file block size : This needs to be a multiple of 512 bytes.
+
+All 3 options can be set in either the Hop GUI options dialog (Menu: Tools / Options) or using the following Hop Conf (`hop-conf.sh` or `hop-conf.bat`) command line options:
+
+[source,shell script]
+----
+ -aza, --azure-account=<account>
+ The account to use for the Azure VFS
+ -azi, --azure-block-increment=<blockIncrement>
+ The block increment size for new files on Azure,
+ multiples of 512 only.
+ -azk, --azure-key=<key>
+ The key to use for the Azure VFS
+
+----
+
+Once done you will see an `azure` entry in the central `hop-config.json` file:
+
+[source,json]
+----
+{
+ "azure" : {
+ "account" : "storage-account-name",
+ "key" : "a/key-comprised-of-a-long-set-of-characters-and-numbers==",
+ "blockIncrement" : "1024"
+ }
+}
+----
+
+== Usage and testing
+
+To test if the configuration works you can simply upload a small CSV file in an Azure Storage folder and then use File/Open in Hop GUI. Then you type in `azure://` as a file location and hit enter (or click the refresh button). Browse to the CSV file you uploaded and open it. If all is configured correctly you should be able to see the content in the Hop GUI.
+
+
+
diff --git a/hop-user-manual/modules/ROOT/pages/vfs/google-cloud-storage-vfs.adoc b/hop-user-manual/modules/ROOT/pages/vfs/google-cloud-storage-vfs.adoc
new file mode 100644
index 0000000..13d79f2
--- /dev/null
+++ b/hop-user-manual/modules/ROOT/pages/vfs/google-cloud-storage-vfs.adoc
@@ -0,0 +1,54 @@
+////
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+////
+
+:documentationPath: /vfs/
+:language: en_US
+
+= Google Cloud Storage VFS
+
+== Scheme
+
+The scheme you can use to access your files in Google Cloud Storage is
+
+`**gs://**`
+
+== Configuration
+
+You need to generate a key file for a service account to make it work. Go to the Google Cloud console to do this. Once you have a key file for your service account, with permissions to access your GCP storage, point to it with either a system environment variable called `GOOGLE_APPLICATION_CREDENTIALS` (standard Google way of doing this) or in the Options dialog in the 'Google Cloud' tab. You can also use `hop-conf`:
+
+[source,shell script]
+----
+ -gck, --google-cloud-service-account-key-file=<serviceAccountKeyFile>
+ Configure the path to a Google Cloud service account JSON key file
+----
+
+Once done you will see a `googleCloud` entry in the central `hop-config.json` file:
+
+[source,json]
+----
+{
+ "googleCloud" : {
+ "serviceAccountKeyFile" : "/path/to/your/google-key.json"
+ }
+}
+----
+
+== Usage and testing
+
+To test if the configuration works type use File/Open in the GUI and type in googledrive:// as a file location. Then hit enter (or click the refresh button). Browse to a CSV, JSON or text file you uploaded and open it. If all is configured correctly you should be able to see the content in the Hop GUI.
+
+
diff --git a/hop-user-manual/modules/ROOT/pages/technology/google/google-info.adoc b/hop-user-manual/modules/ROOT/pages/vfs/google-drive-vfs.adoc
similarity index 50%
copy from hop-user-manual/modules/ROOT/pages/technology/google/google-info.adoc
copy to hop-user-manual/modules/ROOT/pages/vfs/google-drive-vfs.adoc
index 99d04de..4e929a8 100644
--- a/hop-user-manual/modules/ROOT/pages/technology/google/google-info.adoc
+++ b/hop-user-manual/modules/ROOT/pages/vfs/google-drive-vfs.adoc
@@ -14,26 +14,21 @@ KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
////
-:documentationPath: /technology/google/
-:language: en_US
-
-= Google Technology
-== Introduction
-
-The Google technology stack is supported in Hop through a number of plugins. We briefly touch upon them below.
+:documentationPath: /vfs/
+:language: en_US
-=== Google Analytics
+= Google Drive VFS
-You can use the `Google Analytics` transform to get information from this service.
+== Scheme
-== Google Sheets
+The scheme you can use to access your files in Google Drive is
-It can be very convenient to save data into a Google sheet or indeed to read from it. You can do this using the `Google Sheets Input` and `Google Sheets Output` transforms.
+`**googlesheets://**`
-== Google Drive
+== Configuration
-Hop has a Virtual File System (VFS) plugin for Google Drive. You need to generate a credentials file to make it work. Follow the Google documentation to see how that is done. You also need to specify a folder in which security tokens are going to be saved. You can specify both in the Hop system configuration options. This can be done in the Hop GUI: go to the "Google Drive" tab in the Options dialog (from the Tools menu). You can also use the `hop-conf` script with the following options:
+You need to generate a credentials file to make it work. Follow the Google documentation to see how that is done. You also need to specify a folder in which security tokens are going to be saved. You can specify both in the Hop system configuration options. This can be done in the Hop GUI: go to the "Google Drive" tab in the Options dialog (from the Tools menu). You can also use the `hop-conf` script with the following options:
[source,shell script]
----
@@ -66,25 +61,10 @@ Please open the following address in your browser:
Open that URL in a browser and authenticate the given client ID. You'll then get a token in the configured `tokens folder` with which you can work.
-== Google Cloud Storage
-Hop has a Virtual File System (VFS) plugin for Google Cloud Storage.
+== Usage and testing
-You need to generate a key file for a service account to make it work. Go to the Google Cloud console to do this. Once you have a key file for your service account, with permissions to access your GCP storage, point to it with either a system environment variable called `GOOGLE_APPLICATION_CREDENTIALS` (standard Google way of doing this) or in the Options dialog in the 'Google Cloud' tab. You can also use `hop-conf`:
+To test if the configuration works you can simply put a small CSV file in Google Drive and then use File/Open in Hop GUI. Then you type in googledrive:// as a file location and hit enter (or click the refresh button). Browse to the CSV file you uploaded and open it. If all is configured correctly you should be able to see the content in the Hop GUI.
-[source,shell script]
-----
- -gck, --google-cloud-service-account-key-file=<serviceAccountKeyFile>
- Configure the path to a Google Cloud service account JSON key file
-----
-Once done you will see a `googleCloud` entry in the central `hop-config.json` file:
-[source,json]
-----
-{
- "googleCloud" : {
- "serviceAccountKeyFile" : "/path/to/your/google-key.json"
- }
-}
-----