You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by gi...@apache.org on 2020/09/29 00:03:53 UTC
[beam] branch asf-site updated: Publishing website 2020/09/29
00:03:34 at commit cee7388
This is an automated email from the ASF dual-hosted git repository.
git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 7ece216 Publishing website 2020/09/29 00:03:34 at commit cee7388
7ece216 is described below
commit 7ece2161d7b7c84098028575d663454ce0e768d0
Author: jenkins <bu...@apache.org>
AuthorDate: Tue Sep 29 00:03:34 2020 +0000
Publishing website 2020/09/29 00:03:34 at commit cee7388
---
website/generated-content/documentation/index.xml | 613 ++++++++++++++++++++-
.../documentation/io/built-in/snowflake/index.html | 164 +++++-
website/generated-content/sitemap.xml | 2 +-
3 files changed, 746 insertions(+), 33 deletions(-)
diff --git a/website/generated-content/documentation/index.xml b/website/generated-content/documentation/index.xml
index fa73c2c..6497ebe 100644
--- a/website/generated-content/documentation/index.xml
+++ b/website/generated-content/documentation/index.xml
@@ -1549,8 +1549,9 @@ limitations under the License.
</p>
<h3 id="key-pair">Key pair</h3>
<p>To use this authentication method, you must first generate a key pair and associate the public key with the Snowflake user that will connect using the IO transform. For instructions, see the <a href="https://docs.snowflake.com/en/user-guide/jdbc-configure.html">Snowflake documentation</a>.</p>
-<p>To use key pair authentication with SnowflakeIO, invoke your pipeline with following Pipeline options:
-<pre><code>--username=&lt;USERNAME&gt; --privateKeyPath=&lt;PATH_TO_P8_FILE&gt; --privateKeyPassphrase=&lt;PASSWORD_FOR_KEY&gt;</code></pre>
+<p>To use key pair authentication with SnowflakeIO, invoke your pipeline with one of the following set of Pipeline options:
+<pre><code>--username=&lt;USERNAME&gt; --privateKeyPath=&lt;PATH_TO_P8_FILE&gt; --privateKeyPassphrase=&lt;PASSWORD_FOR_KEY&gt;
+--username=&lt;USERNAME&gt; --rawPrivateKey=&lt;PRIVATE_KEY&gt; --privateKeyPassphrase=&lt;PASSWORD_FOR_KEY&gt;</code></pre>
</p>
<h3 id="oauth-token">OAuth token</h3>
<p>SnowflakeIO also supports OAuth token.</p>
@@ -1564,7 +1565,7 @@ limitations under the License.
<p>Create the DataSource configuration:
<div class=language-java>
<div class="highlight"><pre class="chroma"><code class="language-java" data-lang="java"> <span class="n">SnowflakeIO</span><span class="o">.</span><span class="na">DataSourceConfiguration</span>
-<span class="o">.</span><span class="na">create</span><span class="o">(</span><span class="n">SnowflakeCredentialsFactory</span><span class="o">.</span><span class="na">of</span><span class="o">(</span><span class="n">options</span><span class="o">))</span>
+<span class="o">.</span><span class="na">create</span><span class="o">()</span>
<span class="o">.</span><span class="na">withUrl</span><span class="o">(</span><span class="n">options</span><span class="o">.</span><span class="na">getUrl</span><span class="o">())</span>
<span class="o">.</span><span class="na">withServerName</span><span class="o">(</span><span class="n">options</span><span class="o">.</span><span class="na">getServerName</span><span class="o">())</span>
<span class="o">.</span><span class="na">withDatabase</span><span class="o">(</span><span class="n">options</span><span class="o">.</span><span class="na">getDatabase</span><span class="o">())</span>
@@ -1603,10 +1604,40 @@ Where parameters can be:</p>
<li>Example: <code>.withSchema(&quot;PUBLIC&quot;)</code></li>
</ul>
</li>
+<li><code>.withUsernamePasswordAuth(username, password)</code>
+<ul>
+<li>Sets username/password authentication.</li>
+<li>Example: <code>.withUsernamePasswordAuth(&quot;USERNAME&quot;, &quot;PASSWORD&quot;)</code></li>
+</ul>
+</li>
+<li><code>.withOAuth(token)</code>
+<ul>
+<li>Sets OAuth authentication.</li>
+<li>Example: <code>.withOAuth(&quot;TOKEN&quot;)</code></li>
+</ul>
+</li>
+<li><code>.withKeyPairAuth(username, privateKey)</code>
+<ul>
+<li>Sets key pair authentication using username and <a href="https://docs.oracle.com/javase/8/docs/api/java/security/PrivateKey.html">PrivateKey</a></li>
+<li>Example: <code>.withKeyPairAuth(&quot;USERNAME&quot;,</code> <a href="https://docs.oracle.com/javase/8/docs/api/java/security/PrivateKey.html">PrivateKey</a><code>)</code></li>
+</ul>
+</li>
+<li><code>.withKeyPairPathAuth(username, privateKeyPath, privateKeyPassphrase)</code>
+<ul>
+<li>Sets key pair authentication using username, path to private key file and passphrase.</li>
+<li>Example: <code>.withKeyPairPathAuth(&quot;USERNAME&quot;, &quot;PATH/TO/KEY.P8&quot;, &quot;PASSPHRASE&quot;)</code></li>
+</ul>
+</li>
+<li><code>.withKeyPairRawAuth(username, rawPrivateKey, privateKeyPassphrase)</code>
+<ul>
+<li>Sets key pair authentication using username, private key and passphrase.</li>
+<li>Example: <code>.withKeyPairRawAuth(&quot;USERNAME&quot;, &quot;PRIVATE_KEY&quot;, &quot;PASSPHRASE&quot;)</code></li>
+</ul>
+</li>
</ul>
<p><strong>Note</strong> - either <code>.withUrl(...)</code> or <code>.withServerName(...)</code> <strong>is required</strong>.</p>
<h2 id="pipeline-options">Pipeline options</h2>
-<p>Use Beam’s <a href="https://beam.apache.org/releases/javadoc/2.17.0/org/apache/beam/sdk/options/PipelineOptions.html">Pipeline options</a> to set options via the command line.</p>
+<p>Use Beam’s <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/options/PipelineOptions.html">Pipeline options</a> to set options via the command line.</p>
<h3 id="snowflake-pipeline-options">Snowflake Pipeline options</h3>
<p>Snowflake IO library supports following options that can be passed via the <a href="https://beam.apache.org/documentation/io/built-in/snowflake/#running-main-command-with-pipeline-options">command line</a> by default when a Pipeline uses them:</p>
<p><code>--url</code> Snowflake&rsquo;s JDBC-like url including account name and region without any parameters.</p>
@@ -1615,6 +1646,7 @@ Where parameters can be:</p>
<p><code>--oauthToken</code> Required for OAuth authentication only.</p>
<p><code>--password</code> Required for username/password authentication only.</p>
<p><code>--privateKeyPath</code> Path to Private Key file. Required for Private Key authentication only.</p>
+<p><code>--rawPrivateKey</code> Private Key. Required for Private Key authentication only.</p>
<p><code>--privateKeyPassphrase</code> Private Key&rsquo;s passphrase. Required for Private Key authentication only.</p>
<p><code>--stagingBucketName</code> External bucket path ending with <code>/</code>. I.e. <code>gs://bucket/</code>. Sub-directories are allowed.</p>
<p><code>--storageIntegrationName</code> Storage integration name</p>
@@ -1627,6 +1659,117 @@ Where parameters can be:</p>
<p><code>--authenticator</code> Authenticator to use. Optional.</p>
<p><code>--portNumber</code> Port number. Optional.</p>
<p><code>--loginTimeout</code> Login timeout. Optional.</p>
+<p><code>--snowPipe</code> SnowPipe name. Optional.</p>
+<h3 id="running-main-command-with-pipeline-options">Running main command with Pipeline options</h3>
+<p>To pass Pipeline options via the command line, use <code>--args</code> in a gradle command as follows:</p>
+<p>
+<pre><code>./gradle run
+--args=&#34;
+--serverName=&lt;SNOWFLAKE SERVER NAME&gt;
+Example: --serverName=account.region.gcp.snowflakecomputing.com
+--username=&lt;SNOWFLAKE USERNAME&gt;
+Example: --username=testuser
+--password=&lt;SNOWFLAKE PASSWORD&gt;
+Example: --password=mypassword
+--database=&lt;SNOWFLAKE DATABASE&gt;
+Example: --database=TEST_DATABASE
+--schema=&lt;SNOWFLAKE SCHEMA&gt;
+Example: --schema=public
+--table=&lt;SNOWFLAKE TABLE IN DATABASE&gt;
+Example: --table=TEST_TABLE
+--query=&lt;IF NOT TABLE THEN QUERY&gt;
+Example: --query=‘SELECT column FROM TABLE’
+--storageIntegrationName=&lt;SNOWFLAKE STORAGE INTEGRATION NAME&gt;
+Example: --storageIntegrationName=my_integration
+--stagingBucketName=&lt;GCS BUCKET NAME&gt;
+Example: --stagingBucketName=gs://my_gcp_bucket/
+--runner=&lt;DirectRunner/DataflowRunner&gt;
+Example: --runner=DataflowRunner
+--project=&lt;FOR DATAFLOW RUNNER: GCP PROJECT NAME&gt;
+Example: --project=my_project
+--tempLocation=&lt;FOR DATAFLOW RUNNER: GCS TEMP LOCATION STARTING
+WITH gs://…&gt;
+Example: --tempLocation=gs://my_bucket/temp/
+--region=&lt;FOR DATAFLOW RUNNER: GCP REGION&gt;
+Example: --region=us-east-1
+--appName=&lt;OPTIONAL: DATAFLOW JOB NAME PREFIX&gt;
+Example: --appName=my_job&#34;</code></pre>
+Then in the code it is possible to access the parameters with arguments using the options.getStagingBucketName(); command.</p>
+<h3 id="running-test-command-with-pipeline-options">Running test command with Pipeline options</h3>
+<p>To pass Pipeline options via the command line, use <code>-DintegrationTestPipelineOptions</code> in a gradle command as follows:
+<pre><code>./gradlew test --tests nameOfTest
+-DintegrationTestPipelineOptions=&#39;[
+&#34;--serverName=&lt;SNOWFLAKE SERVER NAME&gt;&#34;,
+Example: --serverName=account.region.gcp.snowflakecomputing.com
+&#34;--username=&lt;SNOWFLAKE USERNAME&gt;&#34;,
+Example: --username=testuser
+&#34;--password=&lt;SNOWFLAKE PASSWORD&gt;&#34;,
+Example: --password=mypassword
+&#34;--schema=&lt;SNOWFLAKE SCHEMA&gt;&#34;,
+Example: --schema=PUBLIC
+&#34;--table=&lt;SNOWFLAKE TABLE IN DATABASE&gt;&#34;,
+Example: --table=TEST_TABLE
+&#34;--database=&lt;SNOWFLAKE DATABASE&gt;&#34;,
+Example: --database=TEST_DATABASE
+&#34;--storageIntegrationName=&lt;SNOWFLAKE STORAGE INTEGRATION NAME&gt;&#34;,
+Example: --storageIntegrationName=my_integration
+&#34;--stagingBucketName=&lt;GCS BUCKET NAME&gt;&#34;,
+Example: --stagingBucketName=gs://my_gcp_bucket
+&#34;--externalLocation=&lt;GCS BUCKET URL STARTING WITH GS://&gt;&#34;,
+Example: --tempLocation=gs://my_bucket/temp/
+]&#39; --no-build-cache</code></pre>
+</p>
+<p>Where all parameters are starting with “&ndash;”, they are surrounded with double quotation and separated with comma:</p>
+<ul>
+<li>
+<p><code>--serverName=&lt;SNOWFLAKE SERVER NAME&gt;</code></p>
+<ul>
+<li>Specifies the full name of your account (provided by Snowflake). Note that your full account name might include additional segments that identify the region and cloud platform where your account is hosted.</li>
+<li>Example: <code>--serverName=xy12345.eu-west-1.gcp..snowflakecomputing.com</code></li>
+</ul>
+</li>
+<li>
+<p><code>--username=&lt;SNOWFLAKE USERNAME&gt;</code></p>
+<ul>
+<li>Specifies the login name of the user.</li>
+<li>Example: <code>--username=my_username</code></li>
+</ul>
+</li>
+<li>
+<p><code>--password=&lt;SNOWFLAKE PASSWORD&gt;</code></p>
+<ul>
+<li>Specifies the password for the specified user.</li>
+<li>Example: <code>--password=my_secret</code></li>
+</ul>
+</li>
+<li>
+<p><code>--schema=&lt;SNOWFLAKE SCHEMA&gt;</code></p>
+<ul>
+<li>Specifies the schema to use for the specified database once connected. The specified schema should be an existing schema for which the specified user’s role has privileges.</li>
+<li>Example: <code>--schema=PUBLIC</code></li>
+</ul>
+</li>
+<li>
+<p><code>--table=&lt;SNOWFLAKE TABLE IN DATABASE&gt;</code></p>
+<ul>
+<li>Example: <code>--table=MY_TABLE</code></li>
+</ul>
+</li>
+<li>
+<p><code>--database=&lt;SNOWFLAKE DATABASE&gt;</code></p>
+<ul>
+<li>Specifies the database to use once connected. The specified database should be an existing database for which the specified user’s role has privileges.</li>
+<li>Example: <code>--database=MY_DATABASE</code></li>
+</ul>
+</li>
+<li>
+<p><code>--storageIntegrationName=&lt;SNOWFLAKE STORAGE INTEGRATION NAME&gt;</code></p>
+<ul>
+<li>Name of storage integration created in <a href="https://docs.snowflake.com/en/sql-reference/sql/create-storage-integration.html">Snowflake</a> for a cloud storage of choice.</li>
+<li>Example: <code>--storageIntegrationName=my_google_integration</code></li>
+</ul>
+</li>
+</ul>
<h2 id="running-pipelines-on-dataflow">Running pipelines on Dataflow</h2>
<p>By default, pipelines are run on <a href="https://beam.apache.org/documentation/runners/direct/">Direct Runner</a> on your local machine. To run a pipeline on <a href="https://cloud.google.com/dataflow/">Google Dataflow</a>, you must provide the following Pipeline options:</p>
<ul>
@@ -1661,18 +1804,93 @@ Where parameters can be:</p>
</ul>
</li>
</ul>
-<p>More pipeline options for Dataflow can be found <a href="https://beam.apache.org/releases/javadoc/2.17.0/org/apache/beam/runners/dataflow/options/DataflowPipelineOptions.html">here</a>.</p>
+<p>More pipeline options for Dataflow can be found <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/runners/dataflow/options/DataflowPipelineOptions.html">here</a>.</p>
<p><strong>Note</strong>: To properly authenticate with Google Cloud, please use <a href="https://cloud.google.com/sdk/gcloud/">gcloud</a> or follow the <a href="https://cloud.google.com/docs/authentication/">Google Cloud documentation</a>.</p>
-<p><strong>Important</strong>: Please acknowledge [Google Dataflow pricing](Important: Please acknowledge Google Dataflow pricing).</p>
+<p><strong>Important</strong>: Please acknowledge <a href="https://cloud.google.com/dataflow/pricing">Google Dataflow pricing</a></p>
+<h3 id="running-pipeline-templates-on-dataflow">Running pipeline templates on Dataflow</h3>
+<p>Google Dataflow is supporting <a href="https://cloud.google.com/dataflow/docs/guides/templates/overview">template</a> creation which means staging pipelines on Cloud Storage and running them with ability to pass runtime parameters that are only available during pipeline execution.</p>
+<p>The process of creating own Dataflow template is following</p>
+<ol>
+<li>Create your own pipeline.</li>
+<li>Create <a href="https://cloud.google.com/dataflow/docs/guides/templates/creating-templates#creating-and-staging-templates">Dataflow template</a> with checking which options SnowflakeIO is supporting at runtime.</li>
+<li>Run a Dataflow template using <a href="https://cloud.google.com/dataflow/docs/guides/templates/running-templates#using-the-cloud-console">Cloud Console</a>, <a href="https://cloud.google.com/dataflow/docs/guides/templates/running-templates#using-the-rest-api">REST API</a> or <a href="https://cloud.google.com/dataflow/docs/guides/templates/running-templates#using-gcloud">gcloud</a>.</li>
+</ol>
+<p>Currently, SnowflakeIO supports following options at runtime:</p>
+<ul>
+<li>
+<p><code>--serverName</code> Full server name with account, zone and domain.</p>
+</li>
+<li>
+<p><code>--username</code> Required for username/password and Private Key authentication.</p>
+</li>
+<li>
+<p><code>--password</code> Required for username/password authentication only.</p>
+</li>
+<li>
+<p><code>--rawPrivateKey</code> Private Key file. Required for Private Key authentication only.</p>
+</li>
+<li>
+<p><code>--privateKeyPassphrase</code> Private Key&rsquo;s passphrase. Required for Private Key authentication only.</p>
+</li>
+<li>
+<p><code>--stagingBucketName</code> external bucket path ending with <code>/</code>. I.e. <code>gs://bucket/</code>. Sub-directories are allowed.</p>
+</li>
+<li>
+<p><code>--storageIntegrationName</code> Storage integration name.</p>
+</li>
+<li>
+<p><code>--warehouse</code> Warehouse to use. Optional.</p>
+</li>
+<li>
+<p><code>--database</code> Database name to connect to. Optional.</p>
+</li>
+<li>
+<p><code>--schema</code> Schema to use. Optional.</p>
+</li>
+<li>
+<p><code>--table</code> Table to use. Optional. Note: table is not in default pipeline options.</p>
+</li>
+<li>
+<p><code>--query</code> Query to use. Optional. Note: query is not in default pipeline options.</p>
+</li>
+<li>
+<p><code>--role</code> Role to use. Optional.</p>
+</li>
+<li>
+<p><code>--snowPipe</code> SnowPipe name. Optional.</p>
+</li>
+</ul>
+<p><strong>Note</strong>: table and query is not in pipeline options by default, it may be added by <a href="https://beam.apache.org/documentation/io/built-in/snowflake/#extending-pipeline-options">extending</a> PipelineOptions.</p>
+<p>Currently, SnowflakeIO <strong>doesn&rsquo;t support</strong> following options at runtime:</p>
+<ul>
+<li>
+<p><code>--url</code> Snowflake&rsquo;s JDBC-like url including account name and region without any parameters.</p>
+</li>
+<li>
+<p><code>--oauthToken</code> Required for OAuth authentication only.</p>
+</li>
+<li>
+<p><code>--privateKeyPath</code> Path to Private Key file. Required for Private Key authentication only.</p>
+</li>
+<li>
+<p><code>--authenticator</code> Authenticator to use. Optional.</p>
+</li>
+<li>
+<p><code>--portNumber</code> Port number. Optional.</p>
+</li>
+<li>
+<p><code>--loginTimeout</code> Login timeout. Optional.</p>
+</li>
+</ul>
<h2 id="writing-to-snowflake-tables">Writing to Snowflake tables</h2>
-<p>One of the functions of SnowflakeIO is writing to Snowflake tables. This transformation enables you to finish the Beam pipeline with an output operation that sends the user&rsquo;s <a href="https://beam.apache.org/releases/javadoc/2.17.0/org/apache/beam/sdk/values/PCollection.html">PCollection</a> to your Snowflake database.</p>
+<p>One of the functions of SnowflakeIO is writing to Snowflake tables. This transformation enables you to finish the Beam pipeline with an output operation that sends the user&rsquo;s <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/values/PCollection.html">PCollection</a> to your Snowflake database.</p>
<h3 id="batch-write-from-a-bounded-source">Batch write (from a bounded source)</h3>
<p>The basic .<code>write()</code> operation usage is as follows:
<div class=language-java>
<div class="highlight"><pre class="chroma"><code class="language-java" data-lang="java"><span class="n">data</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span>
<span class="n">SnowflakeIO</span><span class="o">.&lt;</span><span class="n">type</span><span class="o">&gt;</span><span class="n">write</span><span class="o">()</span>
<span class="o">.</span><span class="na">withDataSourceConfiguration</span><span class="o">(</span><span class="n">dc</span><span class="o">)</span>
-<span class="o">.</span><span class="na">to</span><span class="o">(</span><span class="s">&#34;MY_TABLE&#34;</span><span class="o">)</span>
+<span class="o">.</span><span class="na">toTable</span><span class="o">(</span><span class="s">&#34;MY_TABLE&#34;</span><span class="o">)</span>
<span class="o">.</span><span class="na">withStagingBucketName</span><span class="o">(</span><span class="s">&#34;BUCKET NAME&#34;</span><span class="o">)</span>
<span class="o">.</span><span class="na">withStorageIntegrationName</span><span class="o">(</span><span class="s">&#34;STORAGE INTEGRATION NAME&#34;</span><span class="o">)</span>
<span class="o">.</span><span class="na">withUserDataMapper</span><span class="o">(</span><span class="n">mapper</span><span class="o">)</span>
@@ -1685,7 +1903,7 @@ Replace type with the data type of the PCollection object to write; for example,
<p><code>.withDataSourceConfiguration()</code> Accepts a DatasourceConfiguration object.</p>
</li>
<li>
-<p><code>.to()</code> Accepts the target Snowflake table name.</p>
+<p><code>.toTable()</code> Accepts the target Snowflake table name.</p>
</li>
<li>
<p><code>.withStagingBucketName()</code> Accepts a cloud bucket path ended with slash.
@@ -1708,6 +1926,150 @@ Then:
</ul>
<p><strong>Note</strong>:
SnowflakeIO uses COPY statements behind the scenes to write (using <a href="https://docs.snowflake.net/manuals/sql-reference/sql/copy-into-table.html">COPY to table</a>). StagingBucketName will be used to save CSV files which will end up in Snowflake. Those CSV files will be saved under the “stagingBucketName” path.</p>
+<p><strong>Optional</strong> for batching:</p>
+<ul>
+<li><code>.withQuotationMark()</code>
+<ul>
+<li>Default value: <code>‘</code> (single quotation mark).</li>
+<li>Accepts String with one character. It will surround all text (String) fields saved to CSV. It should be one of the accepted characters by <a href="https://docs.snowflake.com/en/sql-reference/sql/create-file-format.html">Snowflake’s</a> <a href="https://docs.snowflake.com/en/sql-reference/sql/create-file-format.html">FIELD_OPTIONALLY_ENCLOSED_BY</a> parameter (double quotation mark, single quotation mark or none).</li>
+<li>Example: <code>.withQuotationMark(&quot;'&quot;)</code></li>
+</ul>
+</li>
+</ul>
+<h3 id="streaming-write--from-unbounded-source">Streaming write (from unbounded source)</h3>
+<p>It is required to create a <a href="https://docs.snowflake.com/en/user-guide/data-load-snowpipe.html">SnowPipe</a> in the Snowflake console. SnowPipe should use the same integration and the same bucket as specified by .withStagingBucketName and .withStorageIntegrationName methods. The write operation might look as follows:
+<div class=language-java>
+<div class="highlight"><pre class="chroma"><code class="language-java" data-lang="java"><span class="n">data</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span>
+<span class="n">SnowflakeIO</span><span class="o">.&lt;</span><span class="n">type</span><span class="o">&gt;</span><span class="n">write</span><span class="o">()</span>
+<span class="o">.</span><span class="na">withStagingBucketName</span><span class="o">(</span><span class="s">&#34;BUCKET NAME&#34;</span><span class="o">)</span>
+<span class="o">.</span><span class="na">withStorageIntegrationName</span><span class="o">(</span><span class="s">&#34;STORAGE INTEGRATION NAME&#34;</span><span class="o">)</span>
+<span class="o">.</span><span class="na">withDataSourceConfiguration</span><span class="o">(</span><span class="n">dc</span><span class="o">)</span>
+<span class="o">.</span><span class="na">withUserDataMapper</span><span class="o">(</span><span class="n">mapper</span><span class="o">)</span>
+<span class="o">.</span><span class="na">withSnowPipe</span><span class="o">(</span><span class="s">&#34;MY_SNOW_PIPE&#34;</span><span class="o">)</span>
+<span class="o">.</span><span class="na">withFlushTimeLimit</span><span class="o">(</span><span class="n">Duration</span><span class="o">.</span><span class="na">millis</span><span class="o">(</span><span class="n">time</span><span class="o">))</span>
+<span class="o">.</span><span class="na">withFlushRowLimit</span><span class="o">(</span><span class="n">rowsNumber</span><span class="o">)</span>
+<span class="o">.</span><span class="na">withShardsNumber</span><span class="o">(</span><span class="n">shardsNumber</span><span class="o">)</span>
+<span class="o">)</span></code></pre></div>
+</div>
+</p>
+<h4 id="parameters">Parameters</h4>
+<p><strong>Required</strong> for streaming:</p>
+<ul>
+<li>
+<p><code> .withDataSourceConfiguration()</code></p>
+<ul>
+<li>Accepts a DatasourceConfiguration object.</li>
+</ul>
+</li>
+<li>
+<p><code>.toTable()</code></p>
+<ul>
+<li>Accepts the target Snowflake table name.</li>
+<li>Example: <code>.toTable(&quot;MY_TABLE)</code></li>
+</ul>
+</li>
+<li>
+<p><code>.withStagingBucketName()</code></p>
+<ul>
+<li>Accepts a cloud bucket path ended with slash.</li>
+<li>Example: <code>.withStagingBucketName(&quot;gs://mybucket/my/dir/&quot;)</code></li>
+</ul>
+</li>
+<li>
+<p><code>.withStorageIntegrationName()</code></p>
+<ul>
+<li>Accepts a name of a Snowflake storage integration object created according to Snowflake documentationt.</li>
+<li>Example:
+<pre><code>CREATE OR REPLACE STORAGE INTEGRATION test_integration
+TYPE = EXTERNAL_STAGE
+STORAGE_PROVIDER = GCS
+ENABLED = TRUE
+STORAGE_ALLOWED_LOCATIONS = (&#39;gcs://bucket/&#39;);</code></pre>
+Then:
+<pre><code>.withStorageIntegrationName(test_integration)</code></pre>
+</li>
+</ul>
+</li>
+<li>
+<p><code>.withSnowPipe()</code></p>
+<ul>
+<li>
+<p>Accepts the target SnowPipe name. <code>.withSnowPipe()</code> accepts the exact name of snowpipe.
+Example:
+<pre><code>CREATE OR REPLACE PIPE test_database.public.test_gcs_pipe
+AS COPY INTO stream_table from @streamstage;</code></pre>
+</p>
+</li>
+<li>
+<p>Then:
+<pre><code>.withSnowPipe(test_gcs_pipe)</code></pre>
+</p>
+</li>
+</ul>
+</li>
+</ul>
+<p><strong>Note</strong>: this is important to provide <strong>schema</strong> and <strong>database</strong> names.</p>
+<ul>
+<li><code>.withUserDataMapper()</code>
+<ul>
+<li>Accepts the <a href="https://beam.apache.org/documentation/io/built-in/snowflake/#userdatamapper-function">UserDataMapper</a> function that will map a user&rsquo;s PCollection to an array of String values <code>(String[]).</code></li>
+</ul>
+</li>
+</ul>
+<p><strong>Note</strong>:</p>
+<p>SnowflakeIO uses COPY statements behind the scenes to write (using <a href="https://docs.snowflake.net/manuals/sql-reference/sql/copy-into-table.html">COPY to table</a>). StagingBucketName will be used to save CSV files which will end up in Snowflake. Those CSV files will be saved under the “stagingBucketName” path.</p>
+<p><strong>Optional</strong> for streaming:</p>
+<ul>
+<li>
+<p><code>.withFlushTimeLimit()</code></p>
+<ul>
+<li>Default value: 30 seconds</li>
+<li>Accepts Duration objects with the specified time after each the streaming write will be repeated</li>
+<li>Example: <code>.withFlushTimeLimit(Duration.millis(180000))</code></li>
+</ul>
+</li>
+<li>
+<p><code>.withFlushRowLimit()</code></p>
+<ul>
+<li>Default value: 10,000 rows</li>
+<li>Limit of rows written to each staged file</li>
+<li>Example: <code>.withFlushRowLimit(500000)</code></li>
+</ul>
+</li>
+<li>
+<p><code>.withShardNumber()</code></p>
+<ul>
+<li>Default value: 1 shard</li>
+<li>Number of files that will be saved in every flush (for purposes of parallel write).</li>
+<li>Example: <code>.withShardNumber(5)</code></li>
+</ul>
+</li>
+<li>
+<p><code>.withQuotationMark()</code></p>
+<ul>
+<li>Default value: <code>‘</code> (single quotation mark).</li>
+<li>Accepts String with one character. It will surround all text (String) fields saved to CSV. It should be one of the accepted characters by <a href="https://docs.snowflake.com/en/sql-reference/sql/create-file-format.html">Snowflake’s</a> <a href="https://docs.snowflake.com/en/sql-reference/sql/create-file-format.html">FIELD_OPTIONALLY_ENCLOSED_BY</a> parameter (double quotation mark, single quotation mark or none). Example: .withQuotationMark(&quot;&quot;) (no qu [...]
+</ul>
+</li>
+<li>
+<p><code>.withDebugMode()</code></p>
+<ul>
+<li>Accepts:
+<ul>
+<li><code>SnowflakeIO.StreamingLogLevel.INFO</code> - shows whole info about loaded files</li>
+<li><code>SnowflakeIO.StreamingLogLevel.ERROR</code> - shows only errors.</li>
+</ul>
+</li>
+<li>Shows logs about streamed files to Snowflake similarly to <a href="https://docs.snowflake.com/en/user-guide/data-load-snowpipe-rest-apis.html#endpoint-insertreport">insertReport</a>. Enabling debug mode may influence performance.</li>
+<li>Example: <code>.withDebugMode(SnowflakeIO.StreamingLogLevel.INFO)</code></li>
+</ul>
+</li>
+</ul>
+<p><strong>Important notice</strong>: Streaming accepts only <strong>key pair authentication</strong>.</p>
+<h4 id="flush-time-duration--number-of-rows">Flush time: duration &amp; number of rows</h4>
+<p>Duration: streaming write will write periodically files on stage according to time duration specified in flush time limit (for example. every 1 minute).</p>
+<p>Number of rows: files staged for write will have number of rows specified in flush row limit unless the flush time limit will be reached (for example if the limit is 1000 rows and buffor collected 99 rows and the 1 minute flush time passes, the rows will be sent to SnowPipe for insertion).</p>
+<p>Size of staged files will depend on the rows size and used compression (GZIP).</p>
<h3 id="userdatamapper-function">UserDataMapper function</h3>
<p>The UserDataMapper function is required to map data from a PCollection to an array of String values before the <code>write()</code> operation saves the data to temporary .csv files. For example:
<div class=language-java>
@@ -1725,7 +2087,7 @@ SnowflakeIO uses COPY statements behind the scenes to write (using <a href="h
<span class="n">data</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span>
<span class="n">SnowflakeIO</span><span class="o">.&lt;~&gt;</span><span class="n">write</span><span class="o">()</span>
<span class="o">.</span><span class="na">withDataSourceConfiguration</span><span class="o">(</span><span class="n">dc</span><span class="o">)</span>
-<span class="o">.</span><span class="na">to</span><span class="o">(</span><span class="s">&#34;MY_TABLE&#34;</span><span class="o">)</span>
+<span class="o">.</span><span class="na">toTable</span><span class="o">(</span><span class="s">&#34;MY_TABLE&#34;</span><span class="o">)</span>
<span class="o">.</span><span class="na">withStagingBucketName</span><span class="o">(</span><span class="s">&#34;BUCKET NAME&#34;</span><span class="o">)</span>
<span class="o">.</span><span class="na">withStorageIntegrationName</span><span class="o">(</span><span class="s">&#34;STORAGE INTEGRATION NAME&#34;</span><span class="o">)</span>
<span class="o">.</span><span class="na">withUserDataMapper</span><span class="o">(</span><span class="n">mapper</span><span class="o">)</span>
@@ -1751,7 +2113,7 @@ SnowflakeIO uses COPY statements behind the scenes to write (using <a href="h
<div class="highlight"><pre class="chroma"><code class="language-java" data-lang="java"><span class="n">data</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span>
<span class="n">SnowflakeIO</span><span class="o">.&lt;~&gt;</span><span class="n">write</span><span class="o">()</span>
<span class="o">.</span><span class="na">withDataSourceConfiguration</span><span class="o">(</span><span class="n">dc</span><span class="o">)</span>
-<span class="o">.</span><span class="na">to</span><span class="o">(</span><span class="s">&#34;MY_TABLE&#34;</span><span class="o">)</span>
+<span class="o">.</span><span class="na">toTable</span><span class="o">(</span><span class="s">&#34;MY_TABLE&#34;</span><span class="o">)</span>
<span class="o">.</span><span class="na">withStagingBucketName</span><span class="o">(</span><span class="s">&#34;BUCKET NAME&#34;</span><span class="o">)</span>
<span class="o">.</span><span class="na">withStorageIntegrationName</span><span class="o">(</span><span class="s">&#34;STORAGE INTEGRATION NAME&#34;</span><span class="o">)</span>
<span class="o">.</span><span class="na">withUserDataMapper</span><span class="o">(</span><span class="n">mapper</span><span class="o">)</span>
@@ -1774,7 +2136,7 @@ SnowflakeIO uses COPY statements behind the scenes to write (using <a href="h
<div class="highlight"><pre class="chroma"><code class="language-java" data-lang="java"><span class="n">data</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span>
<span class="n">SnowflakeIO</span><span class="o">.&lt;~&gt;</span><span class="n">write</span><span class="o">()</span>
<span class="o">.</span><span class="na">withDataSourceConfiguration</span><span class="o">(</span><span class="n">dc</span><span class="o">)</span>
-<span class="o">.</span><span class="na">to</span><span class="o">(</span><span class="s">&#34;MY_TABLE&#34;</span><span class="o">)</span>
+<span class="o">.</span><span class="na">toTable</span><span class="o">(</span><span class="s">&#34;MY_TABLE&#34;</span><span class="o">)</span>
<span class="o">.</span><span class="na">withStagingBucketName</span><span class="o">(</span><span class="s">&#34;BUCKET NAME&#34;</span><span class="o">)</span>
<span class="o">.</span><span class="na">withStorageIntegrationName</span><span class="o">(</span><span class="s">&#34;STORAGE INTEGRATION NAME&#34;</span><span class="o">)</span>
<span class="o">.</span><span class="na">withUserDataMapper</span><span class="o">(</span><span class="n">mapper</span><span class="o">)</span>
@@ -1795,7 +2157,7 @@ A table schema is a list of <code>SFColumn</code> objects with name and ty
<span class="n">data</span><span class="o">.</span><span class="na">apply</span><span class="o">(</span>
<span class="n">SnowflakeIO</span><span class="o">.&lt;~&gt;</span><span class="n">write</span><span class="o">()</span>
<span class="o">.</span><span class="na">withDataSourceConfiguration</span><span class="o">(</span><span class="n">dc</span><span class="o">)</span>
-<span class="o">.</span><span class="na">to</span><span class="o">(</span><span class="s">&#34;MY_TABLE&#34;</span><span class="o">)</span>
+<span class="o">.</span><span class="na">toTable</span><span class="o">(</span><span class="s">&#34;MY_TABLE&#34;</span><span class="o">)</span>
<span class="o">.</span><span class="na">withStagingBucketName</span><span class="o">(</span><span class="s">&#34;BUCKET NAME&#34;</span><span class="o">)</span>
<span class="o">.</span><span class="na">withStorageIntegrationName</span><span class="o">(</span><span class="s">&#34;STORAGE INTEGRATION NAME&#34;</span><span class="o">)</span>
<span class="o">.</span><span class="na">withUserDataMapper</span><span class="o">(</span><span class="n">mapper</span><span class="o">)</span>
@@ -1804,7 +2166,7 @@ A table schema is a list of <code>SFColumn</code> objects with name and ty
</div>
</p>
<h2 id="reading-from-snowflake">Reading from Snowflake</h2>
-<p>One of the functions of SnowflakeIO is reading Snowflake tables - either full tables via table name or custom data via query. Output of the read transform is a <a href="https://beam.apache.org/releases/javadoc/2.17.0/org/apache/beam/sdk/values/PCollection.html">PCollection</a> of user-defined data type.</p>
+<p>One of the functions of SnowflakeIO is reading Snowflake tables - either full tables via table name or custom data via query. Output of the read transform is a <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/values/PCollection.html">PCollection</a> of user-defined data type.</p>
<h3 id="general-usage-1">General usage</h3>
<p>The basic <code>.read()</code> operation usage:
<div class=language-java>
@@ -1861,14 +2223,14 @@ Then:
<li>
<p><code>.withCoder(coder)</code></p>
<ul>
-<li>Accepts the <a href="https://beam.apache.org/releases/javadoc/2.0.0/org/apache/beam/sdk/coders/Coder.html">Coder</a> for USER_DATA_TYPE.</li>
+<li>Accepts the <a href="https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/coders/Coder.html">Coder</a> for USER_DATA_TYPE.</li>
</ul>
</li>
</ul>
<p><strong>Note</strong>:
SnowflakeIO uses COPY statements behind the scenes to read (using <a href="https://docs.snowflake.net/manuals/sql-reference/sql/copy-into-location.html">COPY to location</a>) files staged in cloud storage.StagingBucketName will be used as a temporary location for storing CSV files. Those temporary directories will be named <code>sf_copy_csv_DATE_TIME_RANDOMSUFFIX</code> and they will be removed automatically once Read operation finishes.</p>
<h3 id="csvmapper">CSVMapper</h3>
-<p>SnowflakeIO uses a <a href="https://docs.snowflake.net/manuals/sql-reference/sql/copy-into-location.html">COPY INTO <location></a> statement to move data from a Snowflake table to Google Cloud Storage as CSV files. These files are then downloaded via <a href="https://beam.apache.org/releases/javadoc/2.3.0/index.html?org/apache/beam/sdk/io/FileIO.html">FileIO</a> and processed line by line. Each line is split into an array of Strings using the <a href="http://openc [...]
+<p>SnowflakeIO uses a <a href="https://docs.snowflake.net/manuals/sql-reference/sql/copy-into-location.html">COPY INTO <location></a> statement to move data from a Snowflake table to Google Cloud Storage as CSV files. These files are then downloaded via <a href="https://beam.apache.org/releases/javadoc/current/index.html?org/apache/beam/sdk/io/FileIO.html">FileIO</a> and processed line by line. Each line is split into an array of Strings using the <a href="http://ope [...]
<p>The CSVMapper’s job is to give the user the possibility to convert the array of Strings to a user-defined type, ie. GenericRecord for Avro or Parquet files, or custom POJO.</p>
<p>Example implementation of CsvMapper for GenericRecord:
<div class=language-java>
@@ -1883,7 +2245,224 @@ SnowflakeIO uses COPY statements behind the scenes to read (using <a href="ht
<span class="o">};</span>
<span class="o">}</span></code></pre></div>
</div>
-</p></description></item><item><title>Documentation: ApproximateQuantiles</title><link>/documentation/transforms/java/aggregation/approximatequantiles/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/documentation/transforms/java/aggregation/approximatequantiles/</guid><description>
+</p>
+<h2 id="using-snowflakeio-in-python-sdk">Using SnowflakeIO in Python SDK</h2>
+<h3 id="intro">Intro</h3>
+<p>Snowflake cross-language implementation is supporting both reading and writing operations for Python programming language, thanks to
+cross-language which is part of <a href="https://beam.apache.org/roadmap/portability/">Portability Framework Roadmap</a> which aims to provide full interoperability
+across the Beam ecosystem. From a developer perspective it means the possibility of combining transforms written in different languages(Java/Python/Go).</p>
+<p>For more information about cross-language please see <a href="https://beam.apache.org/roadmap/connectors-multi-sdk/">multi sdk efforts</a>
+and <a href="https://beam.apache.org/roadmap/connectors-multi-sdk/#cross-language-transforms-api-and-expansion-service">Cross-language transforms API and expansion service</a> articles.</p>
+<h3 id="reading-from-snowflake-1">Reading from Snowflake</h3>
+<p>One of the functions of SnowflakeIO is reading Snowflake tables - either full tables via table name or custom data via query. Output of the read transform is a <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.pvalue.html#apache_beam.pvalue.PCollection">PCollection</a> of user-defined data type.</p>
+<h4 id="general-usage-2">General usage</h4>
+<div class=language-py>
+<div class="highlight"><pre class="chroma"><code class="language-py" data-lang="py"><span class="n">OPTIONS</span> <span class="o">=</span> <span class="p">[</span><span class="s2">&#34;--runner=FlinkRunner&#34;</span><span class="p">]</span>
+<span class="k">with</span> <span class="n">TestPipeline</span><span class="p">(</span><span class="n">options</span><span class="o">=</span><span class="n">PipelineOptions</span><span class="p">(</span><span class="n">OPTIONS</span><span class="p">))</span> <span class="k">as</span> <span class="n">p</span><span class="p">:</span>
+<span class="p">(</span><span class="n">p</span>
+<span class="o">|</span> <span class="n">ReadFromSnowflake</span><span class="p">(</span><span class="o">...</span><span class="p">)</span>
+<span class="o">|</span> <span class="o">&lt;</span><span class="n">FURTHER</span> <span class="n">TRANSFORMS</span><span class="o">&gt;</span><span class="p">)</span></code></pre></div>
+</div>
+<h4 id="required-parameters">Required parameters</h4>
+<ul>
+<li>
+<p><code>server_name</code> Full Snowflake server name with an account, zone, and domain.</p>
+</li>
+<li>
+<p><code>schema</code> Name of the Snowflake schema in the database to use.</p>
+</li>
+<li>
+<p><code>database</code> Name of the Snowflake database to use.</p>
+</li>
+<li>
+<p><code>staging_bucket_name</code> Name of the Google Cloud Storage bucket. Bucket will be used as a temporary location for storing CSV files. Those temporary directories will be named <code>sf_copy_csv_DATE_TIME_RANDOMSUFFIX</code> and they will be removed automatically once Read operation finishes.</p>
+</li>
+<li>
+<p><code>storage_integration_name</code> Is the name of a Snowflake storage integration object created according to <a href="https://docs.snowflake.net/manuals/sql-reference/sql/create-storage-integration.html">Snowflake documentation</a>.</p>
+</li>
+<li>
+<p><code>csv_mapper</code> Specifies a function which must translate user-defined object to array of strings. SnowflakeIO uses a <a href="https://docs.snowflake.net/manuals/sql-reference/sql/copy-into-location.html">COPY INTO <location></a> statement to move data from a Snowflake table to Google Cloud Storage as CSV files. These files are then downloaded via <a href="https://beam.apache.org/releases/javadoc/current/index.html?org/apache/beam/sdk/io/FileIO.html">FileI [...]
+Example:
+<div class=language-py>
+<div class="highlight"><pre class="chroma"><code class="language-py" data-lang="py"><span class="k">def</span> <span class="nf">csv_mapper</span><span class="p">(</span><span class="n">strings_array</span><span class="p">):</span>
+<span class="k">return</span> <span class="n">User</span><span class="p">(</span><span class="n">strings_array</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="nb">int</span><span class="p">(</span><span class="n">strings_array</span><span class="p">[</span><span class="mi">1</span><span class="p">])))</span></code></pre></div>
+</div>
+</p>
+</li>
+<li>
+<p><code>table</code> or <code>query</code> Specifies a Snowflake table name or custom SQL query</p>
+</li>
+</ul>
+<h4 id="authentication-parameters">Authentication parameters</h4>
+<p>It’s required to pass one of the following combinations of valid parameters for authentication:</p>
+<ul>
+<li>
+<p><code>username</code> and <code>password</code> Specifies username and password for username/password authentication method.</p>
+</li>
+<li>
+<p><code>private_key_path</code> and <code>private_key_passphrase</code> Specifies a path to private key and passphrase for key/pair authentication method.</p>
+</li>
+<li>
+<p><code>raw_private_key</code> and <code>private_key_passphrase</code> Specifies a private key and passphrase for key/pair authentication method.</p>
+</li>
+<li>
+<p><code>o_auth_token</code> Specifies access token for OAuth authentication method.</p>
+</li>
+</ul>
+<h4 id="additional-parameters">Additional parameters</h4>
+<ul>
+<li>
+<p><code>role</code> specifies Snowflake role. If not specified the user&rsquo;s default will be used.</p>
+</li>
+<li>
+<p><code>warehouse</code> specifies Snowflake warehouse name. If not specified the user&rsquo;s default will be used.</p>
+</li>
+<li>
+<p><code>expansion_service</code> specifies URL of expansion service.</p>
+</li>
+</ul>
+<h3 id="writing-to-snowflake">Writing to Snowflake</h3>
+<p>One of the functions of SnowflakeIO is writing to Snowflake tables. This transformation enables you to finish the Beam pipeline with an output operation that sends the user&rsquo;s <a href="https://beam.apache.org/releases/pydoc/current/apache_beam.pvalue.html#apache_beam.pvalue.PCollection">PCollection</a> to your Snowflake database.</p>
+<h4 id="general-usage-3">General usage</h4>
+<div class=language-py>
+<div class="highlight"><pre class="chroma"><code class="language-py" data-lang="py"><span class="n">OPTIONS</span> <span class="o">=</span> <span class="p">[</span><span class="s2">&#34;--runner=FlinkRunner&#34;</span><span class="p">]</span>
+<span class="k">with</span> <span class="n">TestPipeline</span><span class="p">(</span><span class="n">options</span><span class="o">=</span><span class="n">PipelineOptions</span><span class="p">(</span><span class="n">OPTIONS</span><span class="p">))</span> <span class="k">as</span> <span class="n">p</span><span class="p">:</span>
+<span class="p">(</span><span class="n">p</span>
+<span class="o">|</span> <span class="o">&lt;</span><span class="n">SOURCE</span> <span class="n">OF</span> <span class="n">DATA</span><span class="o">&gt;</span>
+<span class="o">|</span> <span class="n">WriteToSnowflake</span><span class="p">(</span>
+<span class="n">server_name</span><span class="o">=&lt;</span><span class="n">SNOWFLAKE</span> <span class="n">SERVER</span> <span class="n">NAME</span><span class="o">&gt;</span><span class="p">,</span>
+<span class="n">username</span><span class="o">=&lt;</span><span class="n">SNOWFLAKE</span> <span class="n">USERNAME</span><span class="o">&gt;</span><span class="p">,</span>
+<span class="n">password</span><span class="o">=&lt;</span><span class="n">SNOWFLAKE</span> <span class="n">PASSWORD</span><span class="o">&gt;</span><span class="p">,</span>
+<span class="n">o_auth_token</span><span class="o">=&lt;</span><span class="n">OAUTH</span> <span class="n">TOKEN</span><span class="o">&gt;</span><span class="p">,</span>
+<span class="n">private_key_path</span><span class="o">=&lt;</span><span class="n">PATH</span> <span class="n">TO</span> <span class="n">P8</span> <span class="n">FILE</span><span class="o">&gt;</span><span class="p">,</span>
+<span class="n">raw_private_key</span><span class="o">=&lt;</span><span class="n">PRIVATE_KEY</span><span class="o">&gt;</span>
+<span class="n">private_key_passphrase</span><span class="o">=&lt;</span><span class="n">PASSWORD</span> <span class="n">FOR</span> <span class="n">KEY</span><span class="o">&gt;</span><span class="p">,</span>
+<span class="n">schema</span><span class="o">=&lt;</span><span class="n">SNOWFLAKE</span> <span class="n">SCHEMA</span><span class="o">&gt;</span><span class="p">,</span>
+<span class="n">database</span><span class="o">=&lt;</span><span class="n">SNOWFLAKE</span> <span class="n">DATABASE</span><span class="o">&gt;</span><span class="p">,</span>
+<span class="n">staging_bucket_name</span><span class="o">=&lt;</span><span class="n">GCS</span> <span class="n">BUCKET</span> <span class="n">NAME</span><span class="o">&gt;</span><span class="p">,</span>
+<span class="n">storage_integration_name</span><span class="o">=&lt;</span><span class="n">SNOWFLAKE</span> <span class="n">STORAGE</span> <span class="n">INTEGRATION</span> <span class="n">NAME</span><span class="o">&gt;</span><span class="p">,</span>
+<span class="n">create_disposition</span><span class="o">=&lt;</span><span class="n">CREATE</span> <span class="n">DISPOSITION</span><span class="o">&gt;</span><span class="p">,</span>
+<span class="n">write_disposition</span><span class="o">=&lt;</span><span class="n">WRITE</span> <span class="n">DISPOSITION</span><span class="o">&gt;</span><span class="p">,</span>
+<span class="n">table_schema</span><span class="o">=&lt;</span><span class="n">SNOWFLAKE</span> <span class="n">TABLE</span> <span class="n">SCHEMA</span><span class="o">&gt;</span><span class="p">,</span>
+<span class="n">user_data_mapper</span><span class="o">=&lt;</span><span class="n">USER</span> <span class="n">DATA</span> <span class="n">MAPPER</span> <span class="n">FUNCTION</span><span class="o">&gt;</span><span class="p">,</span>
+<span class="n">table</span><span class="o">=&lt;</span><span class="n">SNOWFLAKE</span> <span class="n">TABLE</span><span class="o">&gt;</span><span class="p">,</span>
+<span class="n">query</span><span class="o">=&lt;</span><span class="n">IF</span> <span class="n">NOT</span> <span class="n">TABLE</span> <span class="n">THEN</span> <span class="n">QUERY</span><span class="o">&gt;</span><span class="p">,</span>
+<span class="n">role</span><span class="o">=&lt;</span><span class="n">SNOWFLAKE</span> <span class="n">ROLE</span><span class="o">&gt;</span><span class="p">,</span>
+<span class="n">warehouse</span><span class="o">=&lt;</span><span class="n">SNOWFLAKE</span> <span class="n">WAREHOUSE</span><span class="o">&gt;</span><span class="p">,</span>
+<span class="n">expansion_service</span><span class="o">=&lt;</span><span class="n">EXPANSION</span> <span class="n">SERVICE</span> <span class="n">ADDRESS</span><span class="o">&gt;</span><span class="p">))</span></code></pre></div>
+</div>
+<h4 id="required-parameters-1">Required parameters</h4>
+<ul>
+<li>
+<p><code>server_name</code> Full Snowflake server name with account, zone and domain.</p>
+</li>
+<li>
+<p><code>schema</code> Name of the Snowflake schema in the database to use.</p>
+</li>
+<li>
+<p><code>database</code> Name of the Snowflake database to use.</p>
+</li>
+<li>
+<p><code>staging_bucket_name</code> Path to Google Cloud Storage bucket ended with slash. Bucket will be used to save CSV files which will end up in Snowflake. Those CSV files will be saved under “staging_bucket_name” path.</p>
+</li>
+<li>
+<p><code>storage_integration_name</code> Is the name of a Snowflake storage integration object created according to <a href="https://docs.snowflake.net/manuals/sql-reference/sql/create-storage-integration.html">Snowflake documentation</a>.</p>
+</li>
+<li>
+<p><code>user_data_mapper</code> Specifies a function which maps data from a PCollection to an array of String values before the write operation saves the data to temporary .csv files.
+Example:
+<div class=language-py>
+<div class="highlight"><pre class="chroma"><code class="language-py" data-lang="py"><span class="k">def</span> <span class="nf">user_data_mapper</span><span class="p">(</span><span class="n">user</span><span class="p">):</span>
+<span class="k">return</span> <span class="p">[</span><span class="n">user</span><span class="o">.</span><span class="n">name</span><span class="p">,</span> <span class="nb">str</span><span class="p">(</span><span class="n">user</span><span class="o">.</span><span class="n">age</span><span class="p">)]</span></code></pre></div>
+</div>
+</p>
+</li>
+<li>
+<p><code>table</code> or <code>query</code> Specifies a Snowflake table name or custom SQL query</p>
+</li>
+</ul>
+<h4 id="authentication-parameters-1">Authentication parameters</h4>
+<p>It’s required to pass one of the following combination of valid parameters for authentication:</p>
+<ul>
+<li>
+<p><code>username</code> and <code>password</code> Specifies username/password authentication method.</p>
+</li>
+<li>
+<p><code>private_key_path</code> and <code>private_key_passphrase</code> Specifies a path to private key and passphrase for key/pair authentication method.</p>
+</li>
+<li>
+<p><code>raw_private_key</code> and <code>private_key_passphrase</code> Specifies a private key and passphrase for key/pair authentication method.</p>
+</li>
+<li>
+<p><code>o_auth_token</code> Specifies access token for OAuth authentication method.</p>
+</li>
+</ul>
+<h4 id="additional-parameters-1">Additional parameters</h4>
+<ul>
+<li>
+<p><code>role</code> specifies Snowflake role. If not specified the user&rsquo;s default will be used.</p>
+</li>
+<li>
+<p><code>warehouse</code> specifies Snowflake warehouse name. If not specified the user&rsquo;s default will be used.</p>
+</li>
+<li>
+<p><code>create_disposition</code> Defines the behaviour of the write operation if the target table does not exist. The following values are supported:</p>
+<ul>
+<li>CREATE_IF_NEEDED - default behaviour. The write operation checks whether the specified target table exists; if it does not, the write operation attempts to create the table Specify the schema for the target table using the table_schema parameter.</li>
+<li>CREATE_NEVER - The write operation fails if the target table does not exist.</li>
+</ul>
+</li>
+<li>
+<p><code>write_disposition</code> Defines the write behaviour based on the table where data will be written to. The following values are supported:</p>
+<ul>
+<li>APPEND - Default behaviour. Written data is added to the existing rows in the table,</li>
+<li>EMPTY - The target table must be empty; otherwise, the write operation fails,</li>
+<li>TRUNCATE - The write operation deletes all rows from the target table before writing to it.</li>
+</ul>
+</li>
+<li>
+<p><code>table_schema</code> When the <code>create_disposition</code> parameter is set to CREATE_IF_NEEDED, the table_schema parameter enables specifying the schema for the created target table. A table schema is a JSON array with the following structure:
+<div class=language-py>
+<div class="highlight"><pre class="chroma"><code class="language-py" data-lang="py"><span class="p">{</span><span class="s2">&#34;schema&#34;</span><span class="p">:</span> <span class="p">[</span>
+<span class="p">{</span>
+<span class="s2">&#34;dataType&#34;</span><span class="p">:{</span><span class="s2">&#34;type&#34;</span><span class="p">:</span><span class="s2">&#34;&lt;COLUMN DATA TYPE&gt;&#34;</span><span class="p">},</span>
+<span class="s2">&#34;name&#34;</span><span class="p">:</span><span class="s2">&#34;&lt;COLUMN NAME&gt; &#34;</span><span class="p">,</span>
+<span class="s2">&#34;nullable&#34;</span><span class="p">:</span> <span class="o">&lt;</span><span class="n">NULLABLE</span><span class="o">&gt;</span>
+<span class="p">},</span>
+<span class="o">...</span>
+<span class="p">]}</span></code></pre></div>
+</div>
+All supported data types:
+<pre><code>{&#34;type&#34;:&#34;date&#34;},
+{&#34;type&#34;:&#34;datetime&#34;},
+{&#34;type&#34;:&#34;time&#34;},
+{&#34;type&#34;:&#34;timestamp&#34;},
+{&#34;type&#34;:&#34;timestamp_ltz&#34;},
+{&#34;type&#34;:&#34;timestamp_ntz&#34;},
+{&#34;type&#34;:&#34;timestamp_tz&#34;},
+{&#34;type&#34;:&#34;boolean&#34;},
+{&#34;type&#34;:&#34;decimal&#34;,&#34;precision&#34;:38,&#34;scale&#34;:1},
+{&#34;type&#34;:&#34;double&#34;},
+{&#34;type&#34;:&#34;float&#34;},
+{&#34;type&#34;:&#34;integer&#34;,&#34;precision&#34;:38,&#34;scale&#34;:0},
+{&#34;type&#34;:&#34;number&#34;,&#34;precision&#34;:38,&#34;scale&#34;:1},
+{&#34;type&#34;:&#34;numeric&#34;,&#34;precision&#34;:38,&#34;scale&#34;:2},
+{&#34;type&#34;:&#34;real&#34;},
+{&#34;type&#34;:&#34;array&#34;},
+{&#34;type&#34;:&#34;object&#34;},
+{&#34;type&#34;:&#34;variant&#34;},
+{&#34;type&#34;:&#34;binary&#34;,&#34;size&#34;:null},
+{&#34;type&#34;:&#34;char&#34;,&#34;length&#34;:1},
+{&#34;type&#34;:&#34;string&#34;,&#34;length&#34;:null},
+{&#34;type&#34;:&#34;text&#34;,&#34;length&#34;:null},
+{&#34;type&#34;:&#34;varbinary&#34;,&#34;size&#34;:null},
+{&#34;type&#34;:&#34;varchar&#34;,&#34;length&#34;:100}]</code></pre>
+You can read about Snowflake data types at <a href="https://docs.snowflake.com/en/sql-reference/data-types.html">Snowflake data types</a>.</p>
+</li>
+<li>
+<p><code>expansion_service</code> Specifies URL of expansion service.</p>
+</li>
+</ul></description></item><item><title>Documentation: ApproximateQuantiles</title><link>/documentation/transforms/java/aggregation/approximatequantiles/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/documentation/transforms/java/aggregation/approximatequantiles/</guid><description>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
diff --git a/website/generated-content/documentation/io/built-in/snowflake/index.html b/website/generated-content/documentation/io/built-in/snowflake/index.html
index de40a6d..542e6c0 100644
--- a/website/generated-content/documentation/io/built-in/snowflake/index.html
+++ b/website/generated-content/documentation/io/built-in/snowflake/index.html
@@ -1,39 +1,106 @@
<!doctype html><html lang=en class=no-js><head><meta charset=utf-8><meta http-equiv=x-ua-compatible content="IE=edge"><meta name=viewport content="width=device-width,initial-scale=1"><title>Apache Snowflake I/O connector</title><meta name=description content="Apache Beam is an open source, unified model and set of language-specific SDKs for defining and executing data processing workflows, and also data ingestion and integration flows, supporting Enterprise Integration Patterns (EIPs) an [...]
<span class=sr-only>Toggle navigation</span>
<span class=icon-bar></span><span class=icon-bar></span><span class=icon-bar></span></button>
-<a href=/ class=navbar-brand><img alt=Brand style=height:25px src=/images/beam_logo_navbar.png></a></div><div class="navbar-mask closed"></div><div id=navbar class="navbar-container closed"><ul class="nav navbar-nav"><li><a href=/get-started/beam-overview/>Get Started</a></li><li><a href=/documentation/>Documentation</a></li><li><a href=/documentation/sdks/java/>Languages</a></li><li><a href=/documentation/runners/capability-matrix/>RUNNERS</a></li><li><a href=/roadmap/>Roadmap</a></li>< [...]
+<a href=/ class=navbar-brand><img alt=Brand style=height:25px src=/images/beam_logo_navbar.png></a></div><div class="navbar-mask closed"></div><div id=navbar class="navbar-container closed"><ul class="nav navbar-nav"><li><a href=/get-started/beam-overview/>Get Started</a></li><li><a href=/documentation/>Documentation</a></li><li><a href=/documentation/sdks/java/>Languages</a></li><li><a href=/documentation/runners/capability-matrix/>RUNNERS</a></li><li><a href=/roadmap/>Roadmap</a></li>< [...]
<span class=o>.</span><span class=na>fromArgs</span><span class=o>(</span><span class=n>args</span><span class=o>)</span>
<span class=o>.</span><span class=na>withValidation</span><span class=o>()</span>
<span class=o>.</span><span class=na>as</span><span class=o>(</span><span class=n>SnowflakePipelineOptions</span><span class=o>.</span><span class=na>class</span><span class=o>);</span>
<span class=n>SnowflakeCredentials</span> <span class=n>credentials</span> <span class=o>=</span> <span class=n>SnowflakeCredentialsFactory</span><span class=o>.</span><span class=na>of</span><span class=o>(</span><span class=n>options</span><span class=o>);</span>
<span class=n>SnowflakeIO</span><span class=o>.</span><span class=na>DataSourceConfiguration</span><span class=o>.</span><span class=na>create</span><span class=o>(</span><span class=n>credentials</span><span class=o>)</span>
- <span class=o>.(</span><span class=n>other</span> <span class=n>DataSourceConfiguration</span> <span class=n>options</span><span class=o>)</span></code></pre></div></div></p><h3 id=username-and-password>Username and password</h3><p>To use username/password authentication in SnowflakeIO, invoke your pipeline with the following Pipeline options:<pre><code>--username=<USERNAME> --password=<PASSWORD></code></pre></p><h3 id=key-pair>Key pair</h3><p>To use this authenticati [...]
- <span class=o>.</span><span class=na>create</span><span class=o>(</span><span class=n>SnowflakeCredentialsFactory</span><span class=o>.</span><span class=na>of</span><span class=o>(</span><span class=n>options</span><span class=o>))</span>
+ <span class=o>.(</span><span class=n>other</span> <span class=n>DataSourceConfiguration</span> <span class=n>options</span><span class=o>)</span></code></pre></div></div></p><h3 id=username-and-password>Username and password</h3><p>To use username/password authentication in SnowflakeIO, invoke your pipeline with the following Pipeline options:<pre><code>--username=<USERNAME> --password=<PASSWORD></code></pre></p><h3 id=key-pair>Key pair</h3><p>To use this authenticati [...]
+--username=<USERNAME> --rawPrivateKey=<PRIVATE_KEY> --privateKeyPassphrase=<PASSWORD_FOR_KEY></code></pre></p><h3 id=oauth-token>OAuth token</h3><p>SnowflakeIO also supports OAuth token.</p><p><strong>IMPORTANT</strong>: SnowflakeIO requires a valid OAuth access token. It will neither be able to refresh the token nor obtain it using a web-based flow. For information on configuring an OAuth integration and obtaining the token, see the <a href=https://docs.snowflake.com/e [...]
+ <span class=o>.</span><span class=na>create</span><span class=o>()</span>
<span class=o>.</span><span class=na>withUrl</span><span class=o>(</span><span class=n>options</span><span class=o>.</span><span class=na>getUrl</span><span class=o>())</span>
<span class=o>.</span><span class=na>withServerName</span><span class=o>(</span><span class=n>options</span><span class=o>.</span><span class=na>getServerName</span><span class=o>())</span>
<span class=o>.</span><span class=na>withDatabase</span><span class=o>(</span><span class=n>options</span><span class=o>.</span><span class=na>getDatabase</span><span class=o>())</span>
<span class=o>.</span><span class=na>withWarehouse</span><span class=o>(</span><span class=n>options</span><span class=o>.</span><span class=na>getWarehouse</span><span class=o>())</span>
- <span class=o>.</span><span class=na>withSchema</span><span class=o>(</span><span class=n>options</span><span class=o>.</span><span class=na>getSchema</span><span class=o>());</span></code></pre></div></div>Where parameters can be:</p><ul><li><code>.withUrl(...)</code><ul><li>JDBC-like URL for your Snowflake account, including account name and region, without any parameters.</li><li>Example: <code>.withUrl("jdbc:snowflake://account.snowflakecomputing.com")</code></li></ul></l [...]
+ <span class=o>.</span><span class=na>withSchema</span><span class=o>(</span><span class=n>options</span><span class=o>.</span><span class=na>getSchema</span><span class=o>());</span></code></pre></div></div>Where parameters can be:</p><ul><li><code>.withUrl(...)</code><ul><li>JDBC-like URL for your Snowflake account, including account name and region, without any parameters.</li><li>Example: <code>.withUrl("jdbc:snowflake://account.snowflakecomputing.com")</code></li></ul></l [...]
+ --args="
+ --serverName=<SNOWFLAKE SERVER NAME>
+ Example: --serverName=account.region.gcp.snowflakecomputing.com
+ --username=<SNOWFLAKE USERNAME>
+ Example: --username=testuser
+ --password=<SNOWFLAKE PASSWORD>
+ Example: --password=mypassword
+ --database=<SNOWFLAKE DATABASE>
+ Example: --database=TEST_DATABASE
+ --schema=<SNOWFLAKE SCHEMA>
+ Example: --schema=public
+ --table=<SNOWFLAKE TABLE IN DATABASE>
+ Example: --table=TEST_TABLE
+ --query=<IF NOT TABLE THEN QUERY>
+ Example: --query=‘SELECT column FROM TABLE’
+ --storageIntegrationName=<SNOWFLAKE STORAGE INTEGRATION NAME>
+ Example: --storageIntegrationName=my_integration
+ --stagingBucketName=<GCS BUCKET NAME>
+ Example: --stagingBucketName=gs://my_gcp_bucket/
+ --runner=<DirectRunner/DataflowRunner>
+ Example: --runner=DataflowRunner
+ --project=<FOR DATAFLOW RUNNER: GCP PROJECT NAME>
+ Example: --project=my_project
+ --tempLocation=<FOR DATAFLOW RUNNER: GCS TEMP LOCATION STARTING
+ WITH gs://…>
+ Example: --tempLocation=gs://my_bucket/temp/
+ --region=<FOR DATAFLOW RUNNER: GCP REGION>
+ Example: --region=us-east-1
+ --appName=<OPTIONAL: DATAFLOW JOB NAME PREFIX>
+ Example: --appName=my_job"</code></pre>Then in the code it is possible to access the parameters with arguments using the options.getStagingBucketName(); command.</p><h3 id=running-test-command-with-pipeline-options>Running test command with Pipeline options</h3><p>To pass Pipeline options via the command line, use <code>-DintegrationTestPipelineOptions</code> in a gradle command as follows:<pre><code>./gradlew test --tests nameOfTest
+-DintegrationTestPipelineOptions='[
+ "--serverName=<SNOWFLAKE SERVER NAME>",
+ Example: --serverName=account.region.gcp.snowflakecomputing.com
+ "--username=<SNOWFLAKE USERNAME>",
+ Example: --username=testuser
+ "--password=<SNOWFLAKE PASSWORD>",
+ Example: --password=mypassword
+ "--schema=<SNOWFLAKE SCHEMA>",
+ Example: --schema=PUBLIC
+ "--table=<SNOWFLAKE TABLE IN DATABASE>",
+ Example: --table=TEST_TABLE
+ "--database=<SNOWFLAKE DATABASE>",
+ Example: --database=TEST_DATABASE
+ "--storageIntegrationName=<SNOWFLAKE STORAGE INTEGRATION NAME>",
+ Example: --storageIntegrationName=my_integration
+ "--stagingBucketName=<GCS BUCKET NAME>",
+ Example: --stagingBucketName=gs://my_gcp_bucket
+ "--externalLocation=<GCS BUCKET URL STARTING WITH GS://>",
+ Example: --tempLocation=gs://my_bucket/temp/
+]' --no-build-cache</code></pre></p><p>Where all parameters are starting with “–”, they are surrounded with double quotation and separated with comma:</p><ul><li><p><code>--serverName=<SNOWFLAKE SERVER NAME></code></p><ul><li>Specifies the full name of your account (provided by Snowflake). Note that your full account name might include additional segments that identify the region and cloud platform where your account is hosted.</li><li>Example: <code>--serverName=xy12345.eu- [...]
<span class=n>SnowflakeIO</span><span class=o>.<</span><span class=n>type</span><span class=o>></span><span class=n>write</span><span class=o>()</span>
<span class=o>.</span><span class=na>withDataSourceConfiguration</span><span class=o>(</span><span class=n>dc</span><span class=o>)</span>
- <span class=o>.</span><span class=na>to</span><span class=o>(</span><span class=s>"MY_TABLE"</span><span class=o>)</span>
+ <span class=o>.</span><span class=na>toTable</span><span class=o>(</span><span class=s>"MY_TABLE"</span><span class=o>)</span>
<span class=o>.</span><span class=na>withStagingBucketName</span><span class=o>(</span><span class=s>"BUCKET NAME"</span><span class=o>)</span>
<span class=o>.</span><span class=na>withStorageIntegrationName</span><span class=o>(</span><span class=s>"STORAGE INTEGRATION NAME"</span><span class=o>)</span>
<span class=o>.</span><span class=na>withUserDataMapper</span><span class=o>(</span><span class=n>mapper</span><span class=o>)</span>
-<span class=o>)</span></code></pre></div></div>Replace type with the data type of the PCollection object to write; for example, SnowflakeIO.<string> for an input PCollection of Strings.</p><p>All the below parameters are required:</p><ul><li><p><code>.withDataSourceConfiguration()</code> Accepts a DatasourceConfiguration object.</p></li><li><p><code>.to()</code> Accepts the target Snowflake table name.</p></li><li><p><code>.withStagingBucketName()</code> Accepts a cloud bucket path ended [...]
+<span class=o>)</span></code></pre></div></div>Replace type with the data type of the PCollection object to write; for example, SnowflakeIO.<string> for an input PCollection of Strings.</p><p>All the below parameters are required:</p><ul><li><p><code>.withDataSourceConfiguration()</code> Accepts a DatasourceConfiguration object.</p></li><li><p><code>.toTable()</code> Accepts the target Snowflake table name.</p></li><li><p><code>.withStagingBucketName()</code> Accepts a cloud bucket path [...]
-Example: <code>.withStagingBucketName("gs://mybucket/my/dir/")</code></p></li><li><p><code>.withStorageIntegrationName()</code> Accepts a name of a Snowflake storage integration object created according to Snowflake documentationt. Example:<pre><code>CREATE OR REPLACE STORAGE INTEGRATION test_integration
TYPE = EXTERNAL_STAGE
STORAGE_PROVIDER = GCS
ENABLED = TRUE
STORAGE_ALLOWED_LOCATIONS = ('gcs://bucket/');</code></pre>Then:<pre><code>.withStorageIntegrationName(test_integration)</code></pre></p></li><li><p><code>.withUserDataMapper()</code> Accepts the UserDataMapper function that will map a user’s PCollection to an array of String values <code>(String[])</code>.</p></li></ul><p><strong>Note</strong>:
-SnowflakeIO uses COPY statements behind the scenes to write (using <a href=https://docs.snowflake.net/manuals/sql-reference/sql/copy-into-table.html>COPY to table</a>). StagingBucketName will be used to save CSV files which will end up in Snowflake. Those CSV files will be saved under the “stagingBucketName” path.</p><h3 id=userdatamapper-function>UserDataMapper function</h3><p>The UserDataMapper function is required to map data from a PCollection to an array of String values before the [...]
+SnowflakeIO uses COPY statements behind the scenes to write (using <a href=https://docs.snowflake.net/manuals/sql-reference/sql/copy-into-table.html>COPY to table</a>). StagingBucketName will be used to save CSV files which will end up in Snowflake. Those CSV files will be saved under the “stagingBucketName” path.</p><p><strong>Optional</strong> for batching:</p><ul><li><code>.withQuotationMark()</code><ul><li>Default value: <code>‘</code> (single quotation mark).</li><li>Accepts String [...]
+ <span class=n>SnowflakeIO</span><span class=o>.<</span><span class=n>type</span><span class=o>></span><span class=n>write</span><span class=o>()</span>
+ <span class=o>.</span><span class=na>withStagingBucketName</span><span class=o>(</span><span class=s>"BUCKET NAME"</span><span class=o>)</span>
+ <span class=o>.</span><span class=na>withStorageIntegrationName</span><span class=o>(</span><span class=s>"STORAGE INTEGRATION NAME"</span><span class=o>)</span>
+ <span class=o>.</span><span class=na>withDataSourceConfiguration</span><span class=o>(</span><span class=n>dc</span><span class=o>)</span>
+ <span class=o>.</span><span class=na>withUserDataMapper</span><span class=o>(</span><span class=n>mapper</span><span class=o>)</span>
+ <span class=o>.</span><span class=na>withSnowPipe</span><span class=o>(</span><span class=s>"MY_SNOW_PIPE"</span><span class=o>)</span>
+ <span class=o>.</span><span class=na>withFlushTimeLimit</span><span class=o>(</span><span class=n>Duration</span><span class=o>.</span><span class=na>millis</span><span class=o>(</span><span class=n>time</span><span class=o>))</span>
+ <span class=o>.</span><span class=na>withFlushRowLimit</span><span class=o>(</span><span class=n>rowsNumber</span><span class=o>)</span>
+ <span class=o>.</span><span class=na>withShardsNumber</span><span class=o>(</span><span class=n>shardsNumber</span><span class=o>)</span>
+<span class=o>)</span></code></pre></div></div></p><h4 id=parameters>Parameters</h4><p><strong>Required</strong> for streaming:</p><ul><li><p><code>.withDataSourceConfiguration()</code></p><ul><li>Accepts a DatasourceConfiguration object.</li></ul></li><li><p><code>.toTable()</code></p><ul><li>Accepts the target Snowflake table name.</li><li>Example: <code>.toTable("MY_TABLE)</code></li></ul></li><li><p><code>.withStagingBucketName()</code></p><ul><li>Accepts a cloud bucket path ended wi [...]
+TYPE = EXTERNAL_STAGE
+STORAGE_PROVIDER = GCS
+ENABLED = TRUE
+STORAGE_ALLOWED_LOCATIONS = ('gcs://bucket/');</code></pre>Then:<pre><code>.withStorageIntegrationName(test_integration)</code></pre></li></ul></li><li><p><code>.withSnowPipe()</code></p><ul><li><p>Accepts the target SnowPipe name. <code>.withSnowPipe()</code> accepts the exact name of snowpipe.
+Example:<pre><code>CREATE OR REPLACE PIPE test_database.public.test_gcs_pipe
+AS COPY INTO stream_table from @streamstage;</code></pre></p></li><li><p>Then:<pre><code>.withSnowPipe(test_gcs_pipe)</code></pre></p></li></ul></li></ul><p><strong>Note</strong>: this is important to provide <strong>schema</strong> and <strong>database</strong> names.</p><ul><li><code>.withUserDataMapper()</code><ul><li>Accepts the <a href=https://beam.apache.org/documentation/io/built-in/snowflake/#userdatamapper-function>UserDataMapper</a> function that will map a user’s PCollec [...]
<span class=k>return</span> <span class=o>(</span><span class=n>SnowflakeIO</span><span class=o>.</span><span class=na>UserDataMapper</span><span class=o><</span><span class=n>Long</span><span class=o>>)</span> <span class=n>recordLine</span> <span class=o>-></span> <span class=k>new</span> <span class=n>String</span><span class=o>[]</span> <span class=o>{</span><span class=n>recordLine</span><span class=o>.</span><span class=na>toString</span><span class=o>()};</span>
<span class=o>}</span></code></pre></div></div></p><h3 id=additional-write-options>Additional write options</h3><h4 id=transformation-query>Transformation query</h4><p>The <code>.withQueryTransformation()</code> option for the <code>write()</code> operation accepts a SQL query as a String value, which will be performed while transfering data staged in CSV files directly to the target Snowflake table. For information about the transformation SQL syntax, see the <a href=https://docs.snowfl [...]
<span class=n>data</span><span class=o>.</span><span class=na>apply</span><span class=o>(</span>
<span class=n>SnowflakeIO</span><span class=o>.<~></span><span class=n>write</span><span class=o>()</span>
<span class=o>.</span><span class=na>withDataSourceConfiguration</span><span class=o>(</span><span class=n>dc</span><span class=o>)</span>
- <span class=o>.</span><span class=na>to</span><span class=o>(</span><span class=s>"MY_TABLE"</span><span class=o>)</span>
+ <span class=o>.</span><span class=na>toTable</span><span class=o>(</span><span class=s>"MY_TABLE"</span><span class=o>)</span>
<span class=o>.</span><span class=na>withStagingBucketName</span><span class=o>(</span><span class=s>"BUCKET NAME"</span><span class=o>)</span>
<span class=o>.</span><span class=na>withStorageIntegrationName</span><span class=o>(</span><span class=s>"STORAGE INTEGRATION NAME"</span><span class=o>)</span>
<span class=o>.</span><span class=na>withUserDataMapper</span><span class=o>(</span><span class=n>mapper</span><span class=o>)</span>
@@ -41,7 +108,7 @@ SnowflakeIO uses COPY statements behind the scenes to write (using <a href=https
<span class=o>)</span></code></pre></div></div></p><h4 id=write-disposition>Write disposition</h4><p>Define the write behaviour based on the table where data will be written to by specifying the <code>.withWriteDisposition(...)</code> option for the <code>write()</code> operation. The following values are supported:</p><ul><li><p>APPEND - Default behaviour. Written data is added to the existing rows in the table,</p></li><li><p>EMPTY - The target table must be empty; otherwise, the write [...]
<span class=n>SnowflakeIO</span><span class=o>.<~></span><span class=n>write</span><span class=o>()</span>
<span class=o>.</span><span class=na>withDataSourceConfiguration</span><span class=o>(</span><span class=n>dc</span><span class=o>)</span>
- <span class=o>.</span><span class=na>to</span><span class=o>(</span><span class=s>"MY_TABLE"</span><span class=o>)</span>
+ <span class=o>.</span><span class=na>toTable</span><span class=o>(</span><span class=s>"MY_TABLE"</span><span class=o>)</span>
<span class=o>.</span><span class=na>withStagingBucketName</span><span class=o>(</span><span class=s>"BUCKET NAME"</span><span class=o>)</span>
<span class=o>.</span><span class=na>withStorageIntegrationName</span><span class=o>(</span><span class=s>"STORAGE INTEGRATION NAME"</span><span class=o>)</span>
<span class=o>.</span><span class=na>withUserDataMapper</span><span class=o>(</span><span class=n>mapper</span><span class=o>)</span>
@@ -49,7 +116,7 @@ SnowflakeIO uses COPY statements behind the scenes to write (using <a href=https
<span class=o>)</span></code></pre></div></div></p><h4 id=create-disposition>Create disposition</h4><p>The <code>.withCreateDisposition()</code> option defines the behavior of the write operation if the target table does not exist . The following values are supported:</p><ul><li><p>CREATE_IF_NEEDED - default behaviour. The write operation checks whether the specified target table exists; if it does not, the write operation attempts to create the table Specify the schema for the target ta [...]
<span class=n>SnowflakeIO</span><span class=o>.<~></span><span class=n>write</span><span class=o>()</span>
<span class=o>.</span><span class=na>withDataSourceConfiguration</span><span class=o>(</span><span class=n>dc</span><span class=o>)</span>
- <span class=o>.</span><span class=na>to</span><span class=o>(</span><span class=s>"MY_TABLE"</span><span class=o>)</span>
+ <span class=o>.</span><span class=na>toTable</span><span class=o>(</span><span class=s>"MY_TABLE"</span><span class=o>)</span>
<span class=o>.</span><span class=na>withStagingBucketName</span><span class=o>(</span><span class=s>"BUCKET NAME"</span><span class=o>)</span>
<span class=o>.</span><span class=na>withStorageIntegrationName</span><span class=o>(</span><span class=s>"STORAGE INTEGRATION NAME"</span><span class=o>)</span>
<span class=o>.</span><span class=na>withUserDataMapper</span><span class=o>(</span><span class=n>mapper</span><span class=o>)</span>
@@ -64,12 +131,12 @@ A table schema is a list of <code>SFColumn</code> objects with name and type cor
<span class=n>data</span><span class=o>.</span><span class=na>apply</span><span class=o>(</span>
<span class=n>SnowflakeIO</span><span class=o>.<~></span><span class=n>write</span><span class=o>()</span>
<span class=o>.</span><span class=na>withDataSourceConfiguration</span><span class=o>(</span><span class=n>dc</span><span class=o>)</span>
- <span class=o>.</span><span class=na>to</span><span class=o>(</span><span class=s>"MY_TABLE"</span><span class=o>)</span>
+ <span class=o>.</span><span class=na>toTable</span><span class=o>(</span><span class=s>"MY_TABLE"</span><span class=o>)</span>
<span class=o>.</span><span class=na>withStagingBucketName</span><span class=o>(</span><span class=s>"BUCKET NAME"</span><span class=o>)</span>
<span class=o>.</span><span class=na>withStorageIntegrationName</span><span class=o>(</span><span class=s>"STORAGE INTEGRATION NAME"</span><span class=o>)</span>
<span class=o>.</span><span class=na>withUserDataMapper</span><span class=o>(</span><span class=n>mapper</span><span class=o>)</span>
<span class=o>.</span><span class=na>withTableSchema</span><span class=o>(</span><span class=n>tableSchema</span><span class=o>)</span>
-<span class=o>)</span></code></pre></div></div></p><h2 id=reading-from-snowflake>Reading from Snowflake</h2><p>One of the functions of SnowflakeIO is reading Snowflake tables - either full tables via table name or custom data via query. Output of the read transform is a <a href=https://beam.apache.org/releases/javadoc/2.17.0/org/apache/beam/sdk/values/PCollection.html>PCollection</a> of user-defined data type.</p><h3 id=general-usage-1>General usage</h3><p>The basic <code>.read()</code> [...]
+<span class=o>)</span></code></pre></div></div></p><h2 id=reading-from-snowflake>Reading from Snowflake</h2><p>One of the functions of SnowflakeIO is reading Snowflake tables - either full tables via table name or custom data via query. Output of the read transform is a <a href=https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/values/PCollection.html>PCollection</a> of user-defined data type.</p><h3 id=general-usage-1>General usage</h3><p>The basic <code>.read()</code> [...]
<span class=n>SnowflakeIO</span><span class=o>.<</span><span class=n>USER_DATA_TYPE</span><span class=o>></span><span class=n>read</span><span class=o>()</span>
<span class=o>.</span><span class=na>withDataSourceConfiguration</span><span class=o>(</span><span class=n>dc</span><span class=o>)</span>
<span class=o>.</span><span class=na>fromTable</span><span class=o>(</span><span class=s>"MY_TABLE"</span><span class=o>)</span> <span class=c1>// or .fromQuery("QUERY")
@@ -81,8 +148,8 @@ A table schema is a list of <code>SFColumn</code> objects with name and type cor
TYPE = EXTERNAL_STAGE
STORAGE_PROVIDER = GCS
ENABLED = TRUE
-STORAGE_ALLOWED_LOCATIONS = ('gcs://bucket/');</code></pre>Then:<pre><code>.withStorageIntegrationName(test_integration)</code></pre></p></li><li><p><code>.withCsvMapper(mapper)</code></p><ul><li>Accepts a <a href=https://beam.apache.org/documentation/io/built-in/snowflake/#csvmapper>CSVMapper</a> instance for mapping String[] to USER_DATA_TYPE.</li></ul></li><li><p><code>.withCoder(coder)</code></p><ul><li>Accepts the <a href=https://beam.apache.org/releases/javadoc/2.0.0/org/ap [...]
-SnowflakeIO uses COPY statements behind the scenes to read (using <a href=https://docs.snowflake.net/manuals/sql-reference/sql/copy-into-location.html>COPY to location</a>) files staged in cloud storage.StagingBucketName will be used as a temporary location for storing CSV files. Those temporary directories will be named <code>sf_copy_csv_DATE_TIME_RANDOMSUFFIX</code> and they will be removed automatically once Read operation finishes.</p><h3 id=csvmapper>CSVMapper</h3><p>SnowflakeIO use [...]
+STORAGE_ALLOWED_LOCATIONS = ('gcs://bucket/');</code></pre>Then:<pre><code>.withStorageIntegrationName(test_integration)</code></pre></p></li><li><p><code>.withCsvMapper(mapper)</code></p><ul><li>Accepts a <a href=https://beam.apache.org/documentation/io/built-in/snowflake/#csvmapper>CSVMapper</a> instance for mapping String[] to USER_DATA_TYPE.</li></ul></li><li><p><code>.withCoder(coder)</code></p><ul><li>Accepts the <a href=https://beam.apache.org/releases/javadoc/current/org/ [...]
+SnowflakeIO uses COPY statements behind the scenes to read (using <a href=https://docs.snowflake.net/manuals/sql-reference/sql/copy-into-location.html>COPY to location</a>) files staged in cloud storage.StagingBucketName will be used as a temporary location for storing CSV files. Those temporary directories will be named <code>sf_copy_csv_DATE_TIME_RANDOMSUFFIX</code> and they will be removed automatically once Read operation finishes.</p><h3 id=csvmapper>CSVMapper</h3><p>SnowflakeIO use [...]
<span class=k>return</span> <span class=o>(</span><span class=n>SnowflakeIO</span><span class=o>.</span><span class=na>CsvMapper</span><span class=o><</span><span class=n>GenericRecord</span><span class=o>>)</span>
<span class=n>parts</span> <span class=o>-></span> <span class=o>{</span>
<span class=k>return</span> <span class=k>new</span> <span class=n>GenericRecordBuilder</span><span class=o>(</span><span class=n>PARQUET_SCHEMA</span><span class=o>)</span>
@@ -91,7 +158,74 @@ SnowflakeIO uses COPY statements behind the scenes to read (using <a href=https:
<span class=o>[...]</span>
<span class=o>.</span><span class=na>build</span><span class=o>();</span>
<span class=o>};</span>
-<span class=o>}</span></code></pre></div></div></p></div></div><footer class=footer><div class=footer__contained><div class=footer__cols><div class=footer__cols__col><div class=footer__cols__col__logo><img src=/images/beam_logo_circle.svg class=footer__logo alt="Beam logo"></div><div class=footer__cols__col__logo><img src=/images/apache_logo_circle.svg class=footer__logo alt="Apache logo"></div></div><div class="footer__cols__col footer__cols__col--md"><div class=footer__cols__col__title [...]
+<span class=o>}</span></code></pre></div></div></p><h2 id=using-snowflakeio-in-python-sdk>Using SnowflakeIO in Python SDK</h2><h3 id=intro>Intro</h3><p>Snowflake cross-language implementation is supporting both reading and writing operations for Python programming language, thanks to
+cross-language which is part of <a href=https://beam.apache.org/roadmap/portability/>Portability Framework Roadmap</a> which aims to provide full interoperability
+across the Beam ecosystem. From a developer perspective it means the possibility of combining transforms written in different languages(Java/Python/Go).</p><p>For more information about cross-language please see <a href=https://beam.apache.org/roadmap/connectors-multi-sdk/>multi sdk efforts</a>
+and <a href=https://beam.apache.org/roadmap/connectors-multi-sdk/#cross-language-transforms-api-and-expansion-service>Cross-language transforms API and expansion service</a> articles.</p><h3 id=reading-from-snowflake-1>Reading from Snowflake</h3><p>One of the functions of SnowflakeIO is reading Snowflake tables - either full tables via table name or custom data via query. Output of the read transform is a <a href=https://beam.apache.org/releases/pydoc/current/apache_beam.pvalue.html#apac [...]
+
+<span class=k>with</span> <span class=n>TestPipeline</span><span class=p>(</span><span class=n>options</span><span class=o>=</span><span class=n>PipelineOptions</span><span class=p>(</span><span class=n>OPTIONS</span><span class=p>))</span> <span class=k>as</span> <span class=n>p</span><span class=p>:</span>
+ <span class=p>(</span><span class=n>p</span>
+ <span class=o>|</span> <span class=n>ReadFromSnowflake</span><span class=p>(</span><span class=o>...</span><span class=p>)</span>
+ <span class=o>|</span> <span class=o><</span><span class=n>FURTHER</span> <span class=n>TRANSFORMS</span><span class=o>></span><span class=p>)</span></code></pre></div></div><h4 id=required-parameters>Required parameters</h4><ul><li><p><code>server_name</code> Full Snowflake server name with an account, zone, and domain.</p></li><li><p><code>schema</code> Name of the Snowflake schema in the database to use.</p></li><li><p><code>database</code> Name of the Snowflake database [...]
+Example:<div class=language-py><div class=highlight><pre class=chroma><code class=language-py data-lang=py><span class=k>def</span> <span class=nf>csv_mapper</span><span class=p>(</span><span class=n>strings_array</span><span class=p>):</span>
+ <span class=k>return</span> <span class=n>User</span><span class=p>(</span><span class=n>strings_array</span><span class=p>[</span><span class=mi>0</span><span class=p>],</span> <span class=nb>int</span><span class=p>(</span><span class=n>strings_array</span><span class=p>[</span><span class=mi>1</span><span class=p>])))</span></code></pre></div></div></p></li><li><p><code>table</code> or <code>query</code> Specifies a Snowflake table name or custom SQL query</p></li></ul><h4 id=auth [...]
+
+<span class=k>with</span> <span class=n>TestPipeline</span><span class=p>(</span><span class=n>options</span><span class=o>=</span><span class=n>PipelineOptions</span><span class=p>(</span><span class=n>OPTIONS</span><span class=p>))</span> <span class=k>as</span> <span class=n>p</span><span class=p>:</span>
+ <span class=p>(</span><span class=n>p</span>
+ <span class=o>|</span> <span class=o><</span><span class=n>SOURCE</span> <span class=n>OF</span> <span class=n>DATA</span><span class=o>></span>
+ <span class=o>|</span> <span class=n>WriteToSnowflake</span><span class=p>(</span>
+ <span class=n>server_name</span><span class=o>=<</span><span class=n>SNOWFLAKE</span> <span class=n>SERVER</span> <span class=n>NAME</span><span class=o>></span><span class=p>,</span>
+ <span class=n>username</span><span class=o>=<</span><span class=n>SNOWFLAKE</span> <span class=n>USERNAME</span><span class=o>></span><span class=p>,</span>
+ <span class=n>password</span><span class=o>=<</span><span class=n>SNOWFLAKE</span> <span class=n>PASSWORD</span><span class=o>></span><span class=p>,</span>
+ <span class=n>o_auth_token</span><span class=o>=<</span><span class=n>OAUTH</span> <span class=n>TOKEN</span><span class=o>></span><span class=p>,</span>
+ <span class=n>private_key_path</span><span class=o>=<</span><span class=n>PATH</span> <span class=n>TO</span> <span class=n>P8</span> <span class=n>FILE</span><span class=o>></span><span class=p>,</span>
+ <span class=n>raw_private_key</span><span class=o>=<</span><span class=n>PRIVATE_KEY</span><span class=o>></span>
+ <span class=n>private_key_passphrase</span><span class=o>=<</span><span class=n>PASSWORD</span> <span class=n>FOR</span> <span class=n>KEY</span><span class=o>></span><span class=p>,</span>
+ <span class=n>schema</span><span class=o>=<</span><span class=n>SNOWFLAKE</span> <span class=n>SCHEMA</span><span class=o>></span><span class=p>,</span>
+ <span class=n>database</span><span class=o>=<</span><span class=n>SNOWFLAKE</span> <span class=n>DATABASE</span><span class=o>></span><span class=p>,</span>
+ <span class=n>staging_bucket_name</span><span class=o>=<</span><span class=n>GCS</span> <span class=n>BUCKET</span> <span class=n>NAME</span><span class=o>></span><span class=p>,</span>
+ <span class=n>storage_integration_name</span><span class=o>=<</span><span class=n>SNOWFLAKE</span> <span class=n>STORAGE</span> <span class=n>INTEGRATION</span> <span class=n>NAME</span><span class=o>></span><span class=p>,</span>
+ <span class=n>create_disposition</span><span class=o>=<</span><span class=n>CREATE</span> <span class=n>DISPOSITION</span><span class=o>></span><span class=p>,</span>
+ <span class=n>write_disposition</span><span class=o>=<</span><span class=n>WRITE</span> <span class=n>DISPOSITION</span><span class=o>></span><span class=p>,</span>
+ <span class=n>table_schema</span><span class=o>=<</span><span class=n>SNOWFLAKE</span> <span class=n>TABLE</span> <span class=n>SCHEMA</span><span class=o>></span><span class=p>,</span>
+ <span class=n>user_data_mapper</span><span class=o>=<</span><span class=n>USER</span> <span class=n>DATA</span> <span class=n>MAPPER</span> <span class=n>FUNCTION</span><span class=o>></span><span class=p>,</span>
+ <span class=n>table</span><span class=o>=<</span><span class=n>SNOWFLAKE</span> <span class=n>TABLE</span><span class=o>></span><span class=p>,</span>
+ <span class=n>query</span><span class=o>=<</span><span class=n>IF</span> <span class=n>NOT</span> <span class=n>TABLE</span> <span class=n>THEN</span> <span class=n>QUERY</span><span class=o>></span><span class=p>,</span>
+ <span class=n>role</span><span class=o>=<</span><span class=n>SNOWFLAKE</span> <span class=n>ROLE</span><span class=o>></span><span class=p>,</span>
+ <span class=n>warehouse</span><span class=o>=<</span><span class=n>SNOWFLAKE</span> <span class=n>WAREHOUSE</span><span class=o>></span><span class=p>,</span>
+ <span class=n>expansion_service</span><span class=o>=<</span><span class=n>EXPANSION</span> <span class=n>SERVICE</span> <span class=n>ADDRESS</span><span class=o>></span><span class=p>))</span></code></pre></div></div><h4 id=required-parameters-1>Required parameters</h4><ul><li><p><code>server_name</code> Full Snowflake server name with account, zone and domain.</p></li><li><p><code>schema</code> Name of the Snowflake schema in the database to use.</p></li><li><p><code> [...]
+Example:<div class=language-py><div class=highlight><pre class=chroma><code class=language-py data-lang=py><span class=k>def</span> <span class=nf>user_data_mapper</span><span class=p>(</span><span class=n>user</span><span class=p>):</span>
+ <span class=k>return</span> <span class=p>[</span><span class=n>user</span><span class=o>.</span><span class=n>name</span><span class=p>,</span> <span class=nb>str</span><span class=p>(</span><span class=n>user</span><span class=o>.</span><span class=n>age</span><span class=p>)]</span></code></pre></div></div></p></li><li><p><code>table</code> or <code>query</code> Specifies a Snowflake table name or custom SQL query</p></li></ul><h4 id=authentication-parameters-1>Authentication para [...]
+ <span class=p>{</span>
+ <span class=s2>"dataType"</span><span class=p>:{</span><span class=s2>"type"</span><span class=p>:</span><span class=s2>"<COLUMN DATA TYPE>"</span><span class=p>},</span>
+ <span class=s2>"name"</span><span class=p>:</span><span class=s2>"<COLUMN NAME> "</span><span class=p>,</span>
+ <span class=s2>"nullable"</span><span class=p>:</span> <span class=o><</span><span class=n>NULLABLE</span><span class=o>></span>
+ <span class=p>},</span>
+ <span class=o>...</span>
+ <span class=p>]}</span></code></pre></div></div>All supported data types:<pre><code>{"type":"date"},
+{"type":"datetime"},
+{"type":"time"},
+{"type":"timestamp"},
+{"type":"timestamp_ltz"},
+{"type":"timestamp_ntz"},
+{"type":"timestamp_tz"},
+{"type":"boolean"},
+{"type":"decimal","precision":38,"scale":1},
+{"type":"double"},
+{"type":"float"},
+{"type":"integer","precision":38,"scale":0},
+{"type":"number","precision":38,"scale":1},
+{"type":"numeric","precision":38,"scale":2},
+{"type":"real"},
+{"type":"array"},
+{"type":"object"},
+{"type":"variant"},
+{"type":"binary","size":null},
+{"type":"char","length":1},
+{"type":"string","length":null},
+{"type":"text","length":null},
+{"type":"varbinary","size":null},
+{"type":"varchar","length":100}]</code></pre>You can read about Snowflake data types at <a href=https://docs.snowflake.com/en/sql-reference/data-types.html>Snowflake data types</a>.</p></li><li><p><code>expansion_service</code> Specifies URL of expansion service.</p></li></ul></div></div><footer class=footer><div class=footer__contained><div class=footer__cols><div class=footer__cols__col><div class=footer__cols__col__logo><img src=/images/beam_logo_circle.svg cla [...]
<a href=http://www.apache.org>The Apache Software Foundation</a>
| <a href=/privacy_policy>Privacy Policy</a>
| <a href=/feed.xml>RSS Feed</a><br><br>Apache Beam, Apache, Beam, the Beam logo, and the Apache feather logo are either registered trademarks or trademarks of The Apache Software Foundation. All other products or name brands are trademarks of their respective holders, including The Apache Software Foundation.</div></footer></body></html>
\ No newline at end of file
diff --git a/website/generated-content/sitemap.xml b/website/generated-content/sitemap.xml
index f65f622..c498308 100644
--- a/website/generated-content/sitemap.xml
+++ b/website/generated-content/sitemap.xml
@@ -1 +1 @@
-<?xml version="1.0" encoding="utf-8" standalone="yes"?><urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:xhtml="http://www.w3.org/1999/xhtml"><url><loc>/blog/beam-2.24.0/</loc><lastmod>2020-09-18T12:38:38-07:00</lastmod></url><url><loc>/categories/blog/</loc><lastmod>2020-09-18T12:38:38-07:00</lastmod></url><url><loc>/blog/</loc><lastmod>2020-09-18T12:38:38-07:00</lastmod></url><url><loc>/categories/</loc><lastmod>2020-09-18T12:38:38-07:00</lastmod></url><url><loc>/blog/p [...]
\ No newline at end of file
+<?xml version="1.0" encoding="utf-8" standalone="yes"?><urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:xhtml="http://www.w3.org/1999/xhtml"><url><loc>/blog/beam-2.24.0/</loc><lastmod>2020-09-18T12:38:38-07:00</lastmod></url><url><loc>/categories/blog/</loc><lastmod>2020-09-18T12:38:38-07:00</lastmod></url><url><loc>/blog/</loc><lastmod>2020-09-18T12:38:38-07:00</lastmod></url><url><loc>/categories/</loc><lastmod>2020-09-18T12:38:38-07:00</lastmod></url><url><loc>/blog/p [...]
\ No newline at end of file