You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/08/10 07:15:51 UTC
[GitHub] [beam] piotr-szuberski opened a new pull request #12509: [BEAM-9898][BEAM-9899] SnowflakeIO.Write for cross-language with python wrapper and integration test
piotr-szuberski opened a new pull request #12509:
URL: https://github.com/apache/beam/pull/12509
This PR adds:
1. External transform registrar for SnowflakeIO.Write for cross-language usage
2. Python wrapper for 1)
3. Integration test for SnowflakeIO python wrappers.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2
--- | --- | --- | --- | --- | --- | ---
Go | [](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | ---
Java | [](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/)
Python | [](https://ci-beam.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/) | ---
XLang | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/) | ---
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] purbanow commented on pull request #12509: [BEAM-9898][BEAM-9899] SnowflakeIO.Write for cross-language with python wrapper and integration test
Posted by GitBox <gi...@apache.org>.
purbanow commented on pull request #12509:
URL: https://github.com/apache/beam/pull/12509#issuecomment-682356712
Run Java PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski commented on pull request #12509: [BEAM-9898][BEAM-9899] SnowflakeIO.Write for cross-language with python wrapper and integration test
Posted by GitBox <gi...@apache.org>.
piotr-szuberski commented on pull request #12509:
URL: https://github.com/apache/beam/pull/12509#issuecomment-674747920
@chamikaramj ping
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski commented on a change in pull request #12509: [BEAM-9898][BEAM-9899] SnowflakeIO.Write for cross-language with python wrapper and integration test
Posted by GitBox <gi...@apache.org>.
piotr-szuberski commented on a change in pull request #12509:
URL: https://github.com/apache/beam/pull/12509#discussion_r471932071
##########
File path: sdks/python/apache_beam/io/external/xlang_snowflakeio_it_test.py
##########
@@ -0,0 +1,269 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""
+Integration test for cross-language snowflake io operations.
+
+Example run:
+
+python setup.py nosetests --tests=apache_beam.io.external.snowflake_test \
+--test-pipeline-options="
+ --server_name=<SNOWFLAKE_SERVER_NAME>
+ --username=<SNOWFLAKE_USERNAME>
+ --password=<SNOWFLAKE_PASSWORD>
+ --private_key_path=<PATH_TO_PRIVATE_KEY>
+ --private_key_passphrase=<PASSWORD_TO_PRIVATE_KEY>
+ --o_auth_token=<TOKEN>
+ --staging_bucket_name=<GCP_BUCKET_PATH>
+ --storage_integration_name=<SNOWFLAKE_STORAGE_INTEGRATION_NAME>
+ --database=<DATABASE>
+ --schema=<SCHEMA>
+ --role=<ROLE>
+ --warehouse=<WAREHOUSE>
+ --table=<TABLE_NAME>
+ --runner=FlinkRunner"
+"""
+
+# pytype: skip-file
+
+from __future__ import absolute_import
+
+import argparse
+import binascii
+import logging
+import unittest
+from typing import ByteString
+from typing import NamedTuple
+
+import apache_beam as beam
+from apache_beam import coders
+from apache_beam.io.snowflake import CreateDisposition
+from apache_beam.io.snowflake import ReadFromSnowflake
+from apache_beam.io.snowflake import WriteDisposition
+from apache_beam.io.snowflake import WriteToSnowflake
+from apache_beam.options.pipeline_options import PipelineOptions
+from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import equal_to
+
+# pylint: disable=wrong-import-order, wrong-import-position, ungrouped-imports
+try:
+ from apache_beam.io.gcp.gcsfilesystem import GCSFileSystem
+except ImportError:
+ GCSFileSystem = None
+# pylint: enable=wrong-import-order, wrong-import-position, ungrouped-imports
+
+SCHEMA_STRING = """
+{"schema":[
+ {"dataType":{"type":"integer","precision":38,"scale":0},"name":"number_column","nullable":false},
+ {"dataType":{"type":"boolean"},"name":"boolean_column","nullable":false},
+ {"dataType":{"type":"binary","size":100},"name":"bytes_column","nullable":true}
+]}
+"""
+
+TestRow = NamedTuple(
+ 'TestRow',
+ [
+ ('number_column', int),
+ ('boolean_column', bool),
+ ('bytes_column', ByteString),
+ ])
+
+coders.registry.register_coder(TestRow, coders.RowCoder)
+
+NUM_RECORDS = 100
+
+
+@unittest.skipIf(GCSFileSystem is None, 'GCP dependencies are not installed')
+@unittest.skipIf(
+ TestPipeline().get_option('server_name') is None,
+ 'Snowflake IT test requires external configuration to be run.')
+class SnowflakeTest(unittest.TestCase):
+ def test_snowflake_write_read(self):
+ self.run_write()
+ self.run_read()
+
+ def run_write(self):
+ def user_data_mapper(test_row):
+ return [
+ str(test_row.number_column).encode('utf-8'),
+ str(test_row.boolean_column).encode('utf-8'),
+ binascii.hexlify(test_row.bytes_column),
+ ]
+
+ with TestPipeline(options=PipelineOptions(self.pipeline_args)) as p:
+ p.not_use_test_runner_api = True
+ _ = (
+ p
+ | 'Impulse' >> beam.Impulse()
+ | 'Generate' >> beam.FlatMap(lambda x: range(NUM_RECORDS)) # pylint: disable=range-builtin-not-iterating
+ | 'Map to TestRow' >> beam.Map(
+ lambda num: TestRow(
+ num, num % 2 == 0, b"test" + str(num).encode()))
+ | WriteToSnowflake(
Review comment:
I'm not sure what kind of verification could be added here. run_read function verfies the values written here. write_disposition=TRUNCATE means that the table has been recreated before writing to it. But I can be missing something.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12509: [BEAM-9898][BEAM-9899] SnowflakeIO.Write for cross-language with python wrapper and integration test
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12509:
URL: https://github.com/apache/beam/pull/12509#issuecomment-681189359
# [Codecov](https://codecov.io/gh/apache/beam/pull/12509?src=pr&el=h1) Report
> :exclamation: No coverage uploaded for pull request base (`master@86b8326`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit).
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/12509?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12509 +/- ##
=========================================
Coverage ? 35.99%
=========================================
Files ? 798
Lines ? 87539
Branches ? 0
=========================================
Hits ? 31509
Misses ? 56030
Partials ? 0
```
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12509?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12509?src=pr&el=footer). Last update [86b8326...2cd2dd8](https://codecov.io/gh/apache/beam/pull/12509?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] purbanow commented on pull request #12509: [BEAM-9898][BEAM-9899] SnowflakeIO.Write for cross-language with python wrapper and integration test
Posted by GitBox <gi...@apache.org>.
purbanow commented on pull request #12509:
URL: https://github.com/apache/beam/pull/12509#issuecomment-681652903
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski commented on pull request #12509: [BEAM-9898][BEAM-9899] SnowflakeIO.Write for cross-language with python wrapper and integration test
Posted by GitBox <gi...@apache.org>.
piotr-szuberski commented on pull request #12509:
URL: https://github.com/apache/beam/pull/12509#issuecomment-671534409
Run Portable_Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12509: [BEAM-9898][BEAM-9899] SnowflakeIO.Write for cross-language with python wrapper and integration test
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12509:
URL: https://github.com/apache/beam/pull/12509#issuecomment-681189359
# [Codecov](https://codecov.io/gh/apache/beam/pull/12509?src=pr&el=h1) Report
> :exclamation: No coverage uploaded for pull request base (`master@86b8326`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit).
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/12509?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12509 +/- ##
=========================================
Coverage ? 35.99%
=========================================
Files ? 725
Lines ? 86629
Branches ? 0
=========================================
Hits ? 31185
Misses ? 55444
Partials ? 0
```
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12509?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12509?src=pr&el=footer). Last update [86b8326...2cd2dd8](https://codecov.io/gh/apache/beam/pull/12509?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12509: [BEAM-9898][BEAM-9899] SnowflakeIO.Write for cross-language with python wrapper and integration test
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12509:
URL: https://github.com/apache/beam/pull/12509#issuecomment-681189359
# [Codecov](https://codecov.io/gh/apache/beam/pull/12509?src=pr&el=h1) Report
> :exclamation: No coverage uploaded for pull request base (`master@86b8326`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit).
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/12509?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12509 +/- ##
=========================================
Coverage ? 35.99%
=========================================
Files ? 798
Lines ? 87539
Branches ? 0
=========================================
Hits ? 31509
Misses ? 56030
Partials ? 0
```
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12509?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12509?src=pr&el=footer). Last update [86b8326...2cd2dd8](https://codecov.io/gh/apache/beam/pull/12509?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12509: [BEAM-9898][BEAM-9899] SnowflakeIO.Write for cross-language with python wrapper and integration test
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12509:
URL: https://github.com/apache/beam/pull/12509#issuecomment-681189359
# [Codecov](https://codecov.io/gh/apache/beam/pull/12509?src=pr&el=h1) Report
> :exclamation: No coverage uploaded for pull request base (`master@86b8326`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit).
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/12509?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12509 +/- ##
=========================================
Coverage ? 35.99%
=========================================
Files ? 798
Lines ? 87539
Branches ? 0
=========================================
Hits ? 31509
Misses ? 56030
Partials ? 0
```
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12509?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12509?src=pr&el=footer). Last update [86b8326...2cd2dd8](https://codecov.io/gh/apache/beam/pull/12509?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mwalenia merged pull request #12509: [BEAM-9898][BEAM-9899] SnowflakeIO.Write for cross-language with python wrapper and integration test
Posted by GitBox <gi...@apache.org>.
mwalenia merged pull request #12509:
URL: https://github.com/apache/beam/pull/12509
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski commented on pull request #12509: [BEAM-9898][BEAM-9899] SnowflakeIO.Write for cross-language with python wrapper and integration test
Posted by GitBox <gi...@apache.org>.
piotr-szuberski commented on pull request #12509:
URL: https://github.com/apache/beam/pull/12509#issuecomment-671200150
@chamikaramj Could I ask you for review?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski removed a comment on pull request #12509: [BEAM-9898][BEAM-9899] SnowflakeIO.Write for cross-language with python wrapper and integration test
Posted by GitBox <gi...@apache.org>.
piotr-szuberski removed a comment on pull request #12509:
URL: https://github.com/apache/beam/pull/12509#issuecomment-671534409
Run Portable_Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12509: [BEAM-9898][BEAM-9899] SnowflakeIO.Write for cross-language with python wrapper and integration test
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12509:
URL: https://github.com/apache/beam/pull/12509#issuecomment-681189359
# [Codecov](https://codecov.io/gh/apache/beam/pull/12509?src=pr&el=h1) Report
> :exclamation: No coverage uploaded for pull request base (`master@86b8326`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit).
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/12509?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12509 +/- ##
=========================================
Coverage ? 35.99%
=========================================
Files ? 798
Lines ? 87539
Branches ? 0
=========================================
Hits ? 31509
Misses ? 56030
Partials ? 0
```
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12509?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12509?src=pr&el=footer). Last update [86b8326...2cd2dd8](https://codecov.io/gh/apache/beam/pull/12509?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] slawomir-andrian commented on a change in pull request #12509: [BEAM-9898][BEAM-9899] SnowflakeIO.Write for cross-language with python wrapper and integration test
Posted by GitBox <gi...@apache.org>.
slawomir-andrian commented on a change in pull request #12509:
URL: https://github.com/apache/beam/pull/12509#discussion_r477081860
##########
File path: sdks/python/apache_beam/io/external/xlang_snowflakeio_it_test.py
##########
@@ -0,0 +1,269 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""
+Integration test for cross-language snowflake io operations.
+
+Example run:
+
+python setup.py nosetests --tests=apache_beam.io.external.snowflake_test \
+--test-pipeline-options="
+ --server_name=<SNOWFLAKE_SERVER_NAME>
+ --username=<SNOWFLAKE_USERNAME>
+ --password=<SNOWFLAKE_PASSWORD>
+ --private_key_path=<PATH_TO_PRIVATE_KEY>
+ --private_key_passphrase=<PASSWORD_TO_PRIVATE_KEY>
+ --o_auth_token=<TOKEN>
+ --staging_bucket_name=<GCP_BUCKET_PATH>
+ --storage_integration_name=<SNOWFLAKE_STORAGE_INTEGRATION_NAME>
+ --database=<DATABASE>
+ --schema=<SCHEMA>
+ --role=<ROLE>
+ --warehouse=<WAREHOUSE>
+ --table=<TABLE_NAME>
+ --runner=FlinkRunner"
+"""
+
+# pytype: skip-file
+
+from __future__ import absolute_import
+
+import argparse
+import binascii
+import logging
+import unittest
+from typing import ByteString
+from typing import NamedTuple
+
+import apache_beam as beam
+from apache_beam import coders
+from apache_beam.io.snowflake import CreateDisposition
+from apache_beam.io.snowflake import ReadFromSnowflake
+from apache_beam.io.snowflake import WriteDisposition
+from apache_beam.io.snowflake import WriteToSnowflake
+from apache_beam.options.pipeline_options import PipelineOptions
+from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import equal_to
+
+# pylint: disable=wrong-import-order, wrong-import-position, ungrouped-imports
+try:
+ from apache_beam.io.gcp.gcsfilesystem import GCSFileSystem
+except ImportError:
+ GCSFileSystem = None
+# pylint: enable=wrong-import-order, wrong-import-position, ungrouped-imports
+
+SCHEMA_STRING = """
+{"schema":[
+ {"dataType":{"type":"integer","precision":38,"scale":0},"name":"number_column","nullable":false},
+ {"dataType":{"type":"boolean"},"name":"boolean_column","nullable":false},
+ {"dataType":{"type":"binary","size":100},"name":"bytes_column","nullable":true}
+]}
+"""
+
+TestRow = NamedTuple(
+ 'TestRow',
+ [
+ ('number_column', int),
+ ('boolean_column', bool),
+ ('bytes_column', ByteString),
+ ])
+
+coders.registry.register_coder(TestRow, coders.RowCoder)
+
+NUM_RECORDS = 100
+
+
+@unittest.skipIf(GCSFileSystem is None, 'GCP dependencies are not installed')
+@unittest.skipIf(
+ TestPipeline().get_option('server_name') is None,
+ 'Snowflake IT test requires external configuration to be run.')
+class SnowflakeTest(unittest.TestCase):
+ def test_snowflake_write_read(self):
+ self.run_write()
+ self.run_read()
+
+ def run_write(self):
+ def user_data_mapper(test_row):
+ return [
+ str(test_row.number_column).encode('utf-8'),
+ str(test_row.boolean_column).encode('utf-8'),
+ binascii.hexlify(test_row.bytes_column),
+ ]
+
+ with TestPipeline(options=PipelineOptions(self.pipeline_args)) as p:
+ p.not_use_test_runner_api = True
+ _ = (
+ p
+ | 'Impulse' >> beam.Impulse()
+ | 'Generate' >> beam.FlatMap(lambda x: range(NUM_RECORDS)) # pylint: disable=range-builtin-not-iterating
+ | 'Map to TestRow' >> beam.Map(
+ lambda num: TestRow(
+ num, num % 2 == 0, b"test" + str(num).encode()))
+ | WriteToSnowflake(
Review comment:
@chamikaramj What are your thoughts? Would you mind responding to @piotr-szuberski comment? Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] commented on pull request #12509: [BEAM-9898][BEAM-9899] SnowflakeIO.Write for cross-language with python wrapper and integration test
Posted by GitBox <gi...@apache.org>.
codecov[bot] commented on pull request #12509:
URL: https://github.com/apache/beam/pull/12509#issuecomment-681189359
# [Codecov](https://codecov.io/gh/apache/beam/pull/12509?src=pr&el=h1) Report
> :exclamation: No coverage uploaded for pull request base (`master@86b8326`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit).
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/12509?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12509 +/- ##
=========================================
Coverage ? 35.99%
=========================================
Files ? 725
Lines ? 86629
Branches ? 0
=========================================
Hits ? 31185
Misses ? 55444
Partials ? 0
```
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12509?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12509?src=pr&el=footer). Last update [86b8326...2cd2dd8](https://codecov.io/gh/apache/beam/pull/12509?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] purbanow commented on pull request #12509: [BEAM-9898][BEAM-9899] SnowflakeIO.Write for cross-language with python wrapper and integration test
Posted by GitBox <gi...@apache.org>.
purbanow commented on pull request #12509:
URL: https://github.com/apache/beam/pull/12509#issuecomment-681740011
Run Java PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12509: [BEAM-9898][BEAM-9899] SnowflakeIO.Write for cross-language with python wrapper and integration test
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12509:
URL: https://github.com/apache/beam/pull/12509#issuecomment-681189359
# [Codecov](https://codecov.io/gh/apache/beam/pull/12509?src=pr&el=h1) Report
> :exclamation: No coverage uploaded for pull request base (`master@86b8326`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit).
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/12509?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12509 +/- ##
=========================================
Coverage ? 35.99%
=========================================
Files ? 798
Lines ? 87539
Branches ? 0
=========================================
Hits ? 31509
Misses ? 56030
Partials ? 0
```
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12509?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12509?src=pr&el=footer). Last update [86b8326...2cd2dd8](https://codecov.io/gh/apache/beam/pull/12509?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on pull request #12509: [BEAM-9898][BEAM-9899] SnowflakeIO.Write for cross-language with python wrapper and integration test
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on pull request #12509:
URL: https://github.com/apache/beam/pull/12509#issuecomment-681182380
Retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] edited a comment on pull request #12509: [BEAM-9898][BEAM-9899] SnowflakeIO.Write for cross-language with python wrapper and integration test
Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #12509:
URL: https://github.com/apache/beam/pull/12509#issuecomment-681189359
# [Codecov](https://codecov.io/gh/apache/beam/pull/12509?src=pr&el=h1) Report
> :exclamation: No coverage uploaded for pull request base (`master@86b8326`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit).
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/beam/pull/12509?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #12509 +/- ##
=========================================
Coverage ? 35.99%
=========================================
Files ? 725
Lines ? 86629
Branches ? 0
=========================================
Hits ? 31185
Misses ? 55444
Partials ? 0
```
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/12509?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/12509?src=pr&el=footer). Last update [86b8326...2cd2dd8](https://codecov.io/gh/apache/beam/pull/12509?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on a change in pull request #12509: [BEAM-9898][BEAM-9899] SnowflakeIO.Write for cross-language with python wrapper and integration test
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on a change in pull request #12509:
URL: https://github.com/apache/beam/pull/12509#discussion_r471881235
##########
File path: sdks/python/apache_beam/io/external/xlang_snowflakeio_it_test.py
##########
@@ -0,0 +1,269 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""
+Integration test for cross-language snowflake io operations.
+
+Example run:
+
+python setup.py nosetests --tests=apache_beam.io.external.snowflake_test \
+--test-pipeline-options="
+ --server_name=<SNOWFLAKE_SERVER_NAME>
+ --username=<SNOWFLAKE_USERNAME>
+ --password=<SNOWFLAKE_PASSWORD>
+ --private_key_path=<PATH_TO_PRIVATE_KEY>
+ --private_key_passphrase=<PASSWORD_TO_PRIVATE_KEY>
+ --o_auth_token=<TOKEN>
+ --staging_bucket_name=<GCP_BUCKET_PATH>
+ --storage_integration_name=<SNOWFLAKE_STORAGE_INTEGRATION_NAME>
+ --database=<DATABASE>
+ --schema=<SCHEMA>
+ --role=<ROLE>
+ --warehouse=<WAREHOUSE>
+ --table=<TABLE_NAME>
+ --runner=FlinkRunner"
+"""
+
+# pytype: skip-file
+
+from __future__ import absolute_import
+
+import argparse
+import binascii
+import logging
+import unittest
+from typing import ByteString
+from typing import NamedTuple
+
+import apache_beam as beam
+from apache_beam import coders
+from apache_beam.io.snowflake import CreateDisposition
+from apache_beam.io.snowflake import ReadFromSnowflake
+from apache_beam.io.snowflake import WriteDisposition
+from apache_beam.io.snowflake import WriteToSnowflake
+from apache_beam.options.pipeline_options import PipelineOptions
+from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import equal_to
+
+# pylint: disable=wrong-import-order, wrong-import-position, ungrouped-imports
+try:
+ from apache_beam.io.gcp.gcsfilesystem import GCSFileSystem
+except ImportError:
+ GCSFileSystem = None
+# pylint: enable=wrong-import-order, wrong-import-position, ungrouped-imports
+
+SCHEMA_STRING = """
+{"schema":[
+ {"dataType":{"type":"integer","precision":38,"scale":0},"name":"number_column","nullable":false},
+ {"dataType":{"type":"boolean"},"name":"boolean_column","nullable":false},
+ {"dataType":{"type":"binary","size":100},"name":"bytes_column","nullable":true}
+]}
+"""
+
+TestRow = NamedTuple(
+ 'TestRow',
+ [
+ ('number_column', int),
+ ('boolean_column', bool),
+ ('bytes_column', ByteString),
+ ])
+
+coders.registry.register_coder(TestRow, coders.RowCoder)
+
+NUM_RECORDS = 100
+
+
+@unittest.skipIf(GCSFileSystem is None, 'GCP dependencies are not installed')
+@unittest.skipIf(
+ TestPipeline().get_option('server_name') is None,
+ 'Snowflake IT test requires external configuration to be run.')
+class SnowflakeTest(unittest.TestCase):
+ def test_snowflake_write_read(self):
+ self.run_write()
+ self.run_read()
+
+ def run_write(self):
+ def user_data_mapper(test_row):
+ return [
+ str(test_row.number_column).encode('utf-8'),
+ str(test_row.boolean_column).encode('utf-8'),
+ binascii.hexlify(test_row.bytes_column),
+ ]
+
+ with TestPipeline(options=PipelineOptions(self.pipeline_args)) as p:
+ p.not_use_test_runner_api = True
+ _ = (
+ p
+ | 'Impulse' >> beam.Impulse()
+ | 'Generate' >> beam.FlatMap(lambda x: range(NUM_RECORDS)) # pylint: disable=range-builtin-not-iterating
+ | 'Map to TestRow' >> beam.Map(
+ lambda num: TestRow(
+ num, num % 2 == 0, b"test" + str(num).encode()))
+ | WriteToSnowflake(
Review comment:
Should we verify the output written here ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski commented on a change in pull request #12509: [BEAM-9898][BEAM-9899] SnowflakeIO.Write for cross-language with python wrapper and integration test
Posted by GitBox <gi...@apache.org>.
piotr-szuberski commented on a change in pull request #12509:
URL: https://github.com/apache/beam/pull/12509#discussion_r471932071
##########
File path: sdks/python/apache_beam/io/external/xlang_snowflakeio_it_test.py
##########
@@ -0,0 +1,269 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""
+Integration test for cross-language snowflake io operations.
+
+Example run:
+
+python setup.py nosetests --tests=apache_beam.io.external.snowflake_test \
+--test-pipeline-options="
+ --server_name=<SNOWFLAKE_SERVER_NAME>
+ --username=<SNOWFLAKE_USERNAME>
+ --password=<SNOWFLAKE_PASSWORD>
+ --private_key_path=<PATH_TO_PRIVATE_KEY>
+ --private_key_passphrase=<PASSWORD_TO_PRIVATE_KEY>
+ --o_auth_token=<TOKEN>
+ --staging_bucket_name=<GCP_BUCKET_PATH>
+ --storage_integration_name=<SNOWFLAKE_STORAGE_INTEGRATION_NAME>
+ --database=<DATABASE>
+ --schema=<SCHEMA>
+ --role=<ROLE>
+ --warehouse=<WAREHOUSE>
+ --table=<TABLE_NAME>
+ --runner=FlinkRunner"
+"""
+
+# pytype: skip-file
+
+from __future__ import absolute_import
+
+import argparse
+import binascii
+import logging
+import unittest
+from typing import ByteString
+from typing import NamedTuple
+
+import apache_beam as beam
+from apache_beam import coders
+from apache_beam.io.snowflake import CreateDisposition
+from apache_beam.io.snowflake import ReadFromSnowflake
+from apache_beam.io.snowflake import WriteDisposition
+from apache_beam.io.snowflake import WriteToSnowflake
+from apache_beam.options.pipeline_options import PipelineOptions
+from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import equal_to
+
+# pylint: disable=wrong-import-order, wrong-import-position, ungrouped-imports
+try:
+ from apache_beam.io.gcp.gcsfilesystem import GCSFileSystem
+except ImportError:
+ GCSFileSystem = None
+# pylint: enable=wrong-import-order, wrong-import-position, ungrouped-imports
+
+SCHEMA_STRING = """
+{"schema":[
+ {"dataType":{"type":"integer","precision":38,"scale":0},"name":"number_column","nullable":false},
+ {"dataType":{"type":"boolean"},"name":"boolean_column","nullable":false},
+ {"dataType":{"type":"binary","size":100},"name":"bytes_column","nullable":true}
+]}
+"""
+
+TestRow = NamedTuple(
+ 'TestRow',
+ [
+ ('number_column', int),
+ ('boolean_column', bool),
+ ('bytes_column', ByteString),
+ ])
+
+coders.registry.register_coder(TestRow, coders.RowCoder)
+
+NUM_RECORDS = 100
+
+
+@unittest.skipIf(GCSFileSystem is None, 'GCP dependencies are not installed')
+@unittest.skipIf(
+ TestPipeline().get_option('server_name') is None,
+ 'Snowflake IT test requires external configuration to be run.')
+class SnowflakeTest(unittest.TestCase):
+ def test_snowflake_write_read(self):
+ self.run_write()
+ self.run_read()
+
+ def run_write(self):
+ def user_data_mapper(test_row):
+ return [
+ str(test_row.number_column).encode('utf-8'),
+ str(test_row.boolean_column).encode('utf-8'),
+ binascii.hexlify(test_row.bytes_column),
+ ]
+
+ with TestPipeline(options=PipelineOptions(self.pipeline_args)) as p:
+ p.not_use_test_runner_api = True
+ _ = (
+ p
+ | 'Impulse' >> beam.Impulse()
+ | 'Generate' >> beam.FlatMap(lambda x: range(NUM_RECORDS)) # pylint: disable=range-builtin-not-iterating
+ | 'Map to TestRow' >> beam.Map(
+ lambda num: TestRow(
+ num, num % 2 == 0, b"test" + str(num).encode()))
+ | WriteToSnowflake(
Review comment:
I'm not sure what kind of verification could be added here. run_read function verfies the values written here. write_disposition=TRUNCATE means that the table has been recreated before writing to it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org