You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hop.apache.org by ha...@apache.org on 2022/11/24 09:48:56 UTC
[hop] branch master updated: MDI/Tika IT fix
This is an automated email from the ASF dual-hosted git repository.
hansva pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hop.git
The following commit(s) were added to refs/heads/master by this push:
new 7f7a44c8ec MDI/Tika IT fix
new a04b2cf5c0 Merge pull request #1822 from mattcasters/cypher-builder
7f7a44c8ec is described below
commit 7f7a44c8ec8b5c043e77892f7bf8e1ada6eac97a
Author: Matt Casters <ma...@gmail.com>
AuthorDate: Wed Nov 23 18:14:15 2022 +0100
MDI/Tika IT fix
---
.../beam/dataflowPipeline/google-dataflow-pipeline.adoc | 10 +++++++---
integration-tests/mdi/datasets/golden-apache-tika.csv | 2 +-
2 files changed, 8 insertions(+), 4 deletions(-)
diff --git a/docs/hop-user-manual/modules/ROOT/pages/pipeline/beam/dataflowPipeline/google-dataflow-pipeline.adoc b/docs/hop-user-manual/modules/ROOT/pages/pipeline/beam/dataflowPipeline/google-dataflow-pipeline.adoc
index 251595a4da..5d9a7695f6 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/beam/dataflowPipeline/google-dataflow-pipeline.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/beam/dataflowPipeline/google-dataflow-pipeline.adoc
@@ -27,22 +27,26 @@ Apache Hop pipelines can be scheduled and triggered in various ways. In this sec
Before we can add a new pipeline in the Google Cloud Platform https://console.cloud.google.com/dataflow/pipelines[console] we need to create a Google Storage bucket that contains 3 types of files.
-==== Hop pipelines:
+=== Hop pipelines
+
The pipelines you created using the Hop Gui and wish to schedule in Google Dataflow.
Tip:: You can also create a Hop project using a Google Storage bucket this way you can directly create and edit Hop pipelines in GS
-==== Hop Metadata:
+=== Hop Metadata
+
For the pipeline to be able to use Hop metadata objects and other run configurations we need to generate a hop metadata.json file.
This file can be generated from the GUI under Tools -> Export metadata to JSON or using the export-metadata function from the xref:hop-tools/hop-conf/hop-conf.adoc[Hop conf] tool.
-==== Beam Flex template metadata file:
+=== Beam Flex template metadata file
+
The final part to get everything working is a metadata file used by Dataflow to stitch all the parts together. You can find the file you need xref:pipeline/beam/dataflowPipeline/hopFlexTemplateMetadata.json[here].
Important:: You can change the docker image used in the metadata file
== Creating a Dataflow pipeline
+
Now we can go back to the https://console.cloud.google.com/dataflow/pipelines[console] and "Create data pipeline"
image::beam/beam-dataflow-template.png[]
diff --git a/integration-tests/mdi/datasets/golden-apache-tika.csv b/integration-tests/mdi/datasets/golden-apache-tika.csv
index 6f617b56e0..0c6b1e83d4 100644
--- a/integration-tests/mdi/datasets/golden-apache-tika.csv
+++ b/integration-tests/mdi/datasets/golden-apache-tika.csv
@@ -80,7 +80,7 @@ fileSize,filename,rowNumber,shortFilename,extension,path,hiddenFlag,lastModifica
24291,zip:s3:///apache-hop/tika-test-files.zip!/testLotus123.wk1,79,testLotus123.wk1,wk1,zip:s3:///apache-hop/tika-test-files.zip!/,false,Fri Dec 03 15:28:10 CET 2021,zip:s3:///apache-hop/tika-test-files.zip!/testLotus123.wk1,zip:s3:///apache-hop/tika-test-files.zip!/,207-131-225-53-126-239-184-189-241-84-40-80-214-109-128-7-214-32-228-5-11-87-21-220-131-244-169-33-211-108-233-206-71-208-209-60-93-133-242-176-255-131-24-210-135-126-236-47-99-185-49-189-71-65-122-129-165-56-50-122-249-39-218-62
18635,zip:s3:///apache-hop/tika-test-files.zip!/testLotus123.wk3,80,testLotus123.wk3,wk3,zip:s3:///apache-hop/tika-test-files.zip!/,false,Fri Dec 03 15:28:10 CET 2021,zip:s3:///apache-hop/tika-test-files.zip!/testLotus123.wk3,zip:s3:///apache-hop/tika-test-files.zip!/,207-131-225-53-126-239-184-189-241-84-40-80-214-109-128-7-214-32-228-5-11-87-21-220-131-244-169-33-211-108-233-206-71-208-209-60-93-133-242-176-255-131-24-210-135-126-236-47-99-185-49-189-71-65-122-129-165-56-50-122-249-39-218-62
852,zip:s3:///apache-hop/tika-test-files.zip!/testLotus123.wks,81,testLotus123.wks,wks,zip:s3:///apache-hop/tika-test-files.zip!/,false,Fri Dec 03 15:28:10 CET 2021,zip:s3:///apache-hop/tika-test-files.zip!/testLotus123.wks,zip:s3:///apache-hop/tika-test-files.zip!/,207-131-225-53-126-239-184-189-241-84-40-80-214-109-128-7-214-32-228-5-11-87-21-220-131-244-169-33-211-108-233-206-71-208-209-60-93-133-242-176-255-131-24-210-135-126-236-47-99-185-49-189-71-65-122-129-165-56-50-122-249-39-218-62
-1848,zip:s3:///apache-hop/tika-test-files.zip!/testLotusEml.eml,82,testLotusEml.eml,eml,zip:s3:///apache-hop/tika-test-files.zip!/,false,Fri Dec 03 15:28:10 CET 2021,zip:s3:///apache-hop/tika-test-files.zip!/testLotusEml.eml,zip:s3:///apache-hop/tika-test-files.zip!/,60-83-135-113-93-87-57-124-130-154-110-204-28-229-114-16-19-6-183-174-22-89-98-148-119-66-171-94-199-193-231-35-5-217-104-118-111-216-131-117-251-225-11-96-242-113-34-215-10-208-42-97-244-23-51-75-91-185-53-244-96-236-200-14
+1848,zip:s3:///apache-hop/tika-test-files.zip!/testLotusEml.eml,82,testLotusEml.eml,eml,zip:s3:///apache-hop/tika-test-files.zip!/,false,Fri Dec 03 15:28:10 CET 2021,zip:s3:///apache-hop/tika-test-files.zip!/testLotusEml.eml,zip:s3:///apache-hop/tika-test-files.zip!/,110-47-50-6-102-122-21-100-210-240-68-143-253-153-177-88-137-214-207-31-203-106-41-27-38-255-240-97-160-15-132-68-115-157-195-68-75-63-104-89-32-212-211-119-76-238-30-150-67-183-89-95-12-153-56-38-74-71-74-67-238-45-135-40
1448,zip:s3:///apache-hop/tika-test-files.zip!/testMARC.mrc,83,testMARC.mrc,mrc,zip:s3:///apache-hop/tika-test-files.zip!/,false,Fri Dec 03 15:28:10 CET 2021,zip:s3:///apache-hop/tika-test-files.zip!/testMARC.mrc,zip:s3:///apache-hop/tika-test-files.zip!/,207-131-225-53-126-239-184-189-241-84-40-80-214-109-128-7-214-32-228-5-11-87-21-220-131-244-169-33-211-108-233-206-71-208-209-60-93-133-242-176-255-131-24-210-135-126-236-47-99-185-49-189-71-65-122-129-165-56-50-122-249-39-218-62
13759,zip:s3:///apache-hop/tika-test-files.zip!/testMHTMLFirefox.mhtml,84,testMHTMLFirefox.mhtml,mhtml,zip:s3:///apache-hop/tika-test-files.zip!/,false,Fri Dec 03 15:28:10 CET 2021,zip:s3:///apache-hop/tika-test-files.zip!/testMHTMLFirefox.mhtml,zip:s3:///apache-hop/tika-test-files.zip!/,246-44-212-84-186-21-32-103-230-183-232-81-76-89-99-191-238-146-99-228-72-97-105-237-98-143-75-13-47-137-222-230-153-96-42-121-252-235-76-237-159-85-166-147-194-214-170-248-147-61-135-93-149-248-249-182- [...]
82969,zip:s3:///apache-hop/tika-test-files.zip!/testMKV.mkv,85,testMKV.mkv,mkv,zip:s3:///apache-hop/tika-test-files.zip!/,false,Fri Dec 03 15:28:10 CET 2021,zip:s3:///apache-hop/tika-test-files.zip!/testMKV.mkv,zip:s3:///apache-hop/tika-test-files.zip!/,207-131-225-53-126-239-184-189-241-84-40-80-214-109-128-7-214-32-228-5-11-87-21-220-131-244-169-33-211-108-233-206-71-208-209-60-93-133-242-176-255-131-24-210-135-126-236-47-99-185-49-189-71-65-122-129-165-56-50-122-249-39-218-62