You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by gi...@apache.org on 2023/02/09 21:20:35 UTC
[beam] branch dependabot/go_modules/sdks/cloud.google.com/go/spanner-1.44.0 updated (051da1d5cea -> 81fd19270e4)
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a change to branch dependabot/go_modules/sdks/cloud.google.com/go/spanner-1.44.0
in repository https://gitbox.apache.org/repos/asf/beam.git
discard 051da1d5cea Bump cloud.google.com/go/spanner from 1.43.0 to 1.44.0 in /sdks
add d50924d2ae9 [Spark Dataset runner] Fix collection encoder bug that may lead to corrupted data due to naive usage of unsafe storage (fixes #25296). (#25297)
add 72781cb089a [Spark Dataset runner] Break linage of dataset to reduce Spark planning overhead in case of large complex query plans (relates to #24710 and #23845) (#25187)
add 86480f33397 Fix flaky test due to create bigquery dataset conflict (#25266)
add b3be8b65aff 24472 Implement FileWriteSchemaTransformProvider (#24806)
add 0441d9d5823 Embed ML video to docs (#25302)
add d68b02804a5 skip automated expansion test (#25304)
add ed699f4504f Add banner highlighting beam ml (#25306)
add b1c9d8aec07 Optimize to use cached output receiver instead of creating one on DoFn invocation #21250 (#25245)
add c65e777d6c6 Optimize PGBK table to only update cache when there is a large enough size change. #21250 (#25219)
add 679d30256c6 Swap setting a context from being on the hot path when we emit elements to only be done during bundle creation and teardown #21250 (#25291)
add 4e6a3e41aba Basic SchemaTransform implementation for SQLTransform. (#25177)
add 16cb63be7e0 issue24170 google colab link added (#24820)
add aee2c844ad8 Clarify llm download/loading instructions (#25145)
add bb582d8db12 Allow for setMaxRetryJobs in BigQueryIO to be configurable (#25224)
add fd4ea9b6bce delete file used in internal testing (#25339)
add bbe27e96de0 Don't discard output column names
add 9178456b38f Merge pull request #25290: Don't discard output column names
add cd8eb0c9c68 Fix output timestamp for multi output receiver in FnApiDoFnRunner #25344 (#25349)
add dc632029f0c Change UnboundedScheduledExecutorService to avoid creating threads when (#25300)
add d20d0b01c3c Upgrading spring-expression to latest patch version (#25348)
add 2a350761d09 Better batching for higher fixed costs.
add d6d88808aa4 Add an option to get the old behavior.
add 37112d65578 lint
add 46a5e77232d Merge pull request #25351 Better batching for higher fixed costs.
add bf5114bb2fc [BEAM-12164] Enforced only positive state transitions from CREATED -> SCHEDULED -> RUNNING -> FINISHED for the change stream metadata table. (#25311)
add a9e80d2981c Added Role-based access control integration tests for Spanner Change Streams (#25246)
add 88da381cc33 Fail the pipeline when a mismatched Python or Beam version is detected. (#25313)
add 063eea63345 [Spark runner] Removal of Spark 2 runner support (closes #25259) (#25263)
add 0bec597aabe Add TensorRT runinference example for Text Classification (#25226)
add b90ed151467 Bump cryptography from 39.0.0 to 39.0.1 in /sdks/python/container/py39 (#25377)
add e84a3e8e001 Playground Frontend Test workflow (#24728) (#25254)
add 35af801a3d6 Simpify composite literal in metrics_test.go (#25384)
add 338ddec28c8 Replace use of deprecated strings.Title function (#25385)
add e967d6ab74d Minor updates to typescript generation.
add bf09a5ed4a5 Merge pull request #25260 Minor updates to typescript generation.
add 287ddf6aa79 Explicitly set mongo container version for testing (#25369)
add 69ddf44e67d Update upper bound for numpy. (#24725)
add 2965ef2095d Support batching as config in RunInference (sklearn and pytorch) (#25370)
add 9fcb3a5b48a Add a note on increased lower bounds. (#25389)
add ea1625ab5f3 Support samza portable UDF metrics. (#25265)
add 513f4b003b1 Adding support for @SchemaFieldDescription annotation that allows ann… (#25268)
add d36f34df570 Adding SpannerIO.readChangeStreams support for SchemaTransform (#24999)
add 2b15208feb8 Bump google.golang.org/api from 0.108.0 to 0.109.0 in /sdks (#25250)
add 47353efb9d5 Add min and max batch size args to model handler (#25395)
add 840ea6e8bc0 Beam/sdks/io/gcp/java/healcare/hl7v2 io read (#25056)
add e0a36a28c11 Remove reuse of GenericRecord instance when reading Avro from BigQuery (#25320)
add e659c83c60d fixed reading env variable (#25362)
add cbac3e093a4 fix doc on bq sxtorage (#25353)
add 64e40d2c018 [Playground] Fix Test_getRunOrTestCmd on Go 1.20 (#25379)
add 9e067c65392 Fix typo - metdata -> metadata (#25399)
add 41ea9e5268f [prism] Initial commit for READMEs and go.mod (#25404)
add e0749855f99 [prism] Add urns package (#25405)
add 81fd19270e4 Bump cloud.google.com/go/spanner from 1.43.0 to 1.44.0 in /sdks
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (051da1d5cea)
\
N -- N -- N refs/heads/dependabot/go_modules/sdks/cloud.google.com/go/spanner-1.44.0 (81fd19270e4)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.github/workflows/playground_frontend_test.yml | 91 ++++
...ommit_CrossLanguageValidatesRunner_Spark.groovy | 51 ---
.../job_PostCommit_Java_Examples_Spark.groovy | 1 -
...PortableValidatesRunner_Spark2_Streaming.groovy | 43 --
...Java_PortableValidatesRunner_Spark_Batch.groovy | 2 -
...ob_PostCommit_Java_ValidatesRunner_Spark.groovy | 1 -
...ValidatesRunner_SparkStructuredStreaming.groovy | 1 -
.test-infra/tools/stale_bq_datasets_cleaner.sh | 2 +-
CHANGES.md | 8 +-
build.gradle.kts | 6 -
.../org/apache/beam/gradle/BeamModulePlugin.groovy | 5 +-
playground/backend/containers/java/Dockerfile | 2 +-
playground/backend/go.mod | 2 +-
.../code_processing/code_processing_test.go | 13 +-
playground/frontend/README.md | 11 +-
.../frontend/assets/{github.svg => colab.svg} | 10 +-
playground/frontend/assets/github.svg | 4 +-
playground/frontend/assets/translations/en.yaml | 3 +
playground/frontend/build.gradle | 5 +-
.../lib/components/link_button/colab_button.dart | 31 +-
.../lib/components/link_button/dataset_button.dart | 36 +-
.../lib/components/link_button/github_button.dart | 29 +-
.../lib/components/link_button/link_button.dart | 63 +++
playground/frontend/lib/config.example.dart | 15 +-
playground/frontend/lib/l10n/app_en.arb | 4 -
.../description_popover/description_popover.dart | 30 +-
.../examples/components/example_actions.dart | 52 +++
.../widgets/embedded_actions.dart | 58 ++-
playground/frontend/lib/src/assets/assets.gen.dart | 2 +
.../lib/playground_components.dart | 1 +
.../lib/src/models/example.dart | 6 +-
.../lib/src/models/example_base.dart | 6 +-
.../example_client/grpc_example_client.dart | 5 +-
playground/infrastructure/grpc_client.py | 3 +-
.../src/main/scripts/build_release_candidate.sh | 2 +
.../runtime/SamzaMetricsBundleProgressHandler.java | 10 +-
runners/spark/2/build.gradle | 33 --
runners/spark/2/job-server/build.gradle | 31 --
runners/spark/2/job-server/container/build.gradle | 27 --
.../spark/structuredstreaming/Constants.java | 25 --
.../SparkStructuredStreamingPipelineOptions.java | 42 --
.../SparkStructuredStreamingPipelineResult.java | 134 ------
.../SparkStructuredStreamingRunner.java | 228 ----------
.../SparkStructuredStreamingRunnerRegistrar.java | 54 ---
.../aggregators/AggregatorsAccumulator.java | 73 ---
.../aggregators/NamedAggregators.java | 113 -----
.../aggregators/NamedAggregatorsAccumulator.java | 63 ---
.../aggregators/package-info.java | 20 -
.../structuredstreaming/examples/WordCount.java | 132 ------
.../metrics/AggregatorMetric.java | 74 ---
.../metrics/AggregatorMetricSource.java | 49 --
.../structuredstreaming/metrics/BeamMetricSet.java | 60 ---
.../metrics/MetricsAccumulator.java | 76 ----
.../MetricsContainerStepMapAccumulator.java | 65 ---
.../metrics/SparkBeamMetric.java | 107 -----
.../metrics/SparkBeamMetricSource.java | 48 --
.../metrics/SparkMetricsContainerStepMap.java | 43 --
.../metrics/WithMetricsSupport.java | 91 ----
.../structuredstreaming/metrics/package-info.java | 20 -
.../metrics/sink/CodahaleCsvSink.java | 86 ----
.../metrics/sink/CodahaleGraphiteSink.java | 89 ----
.../metrics/sink/package-info.java | 20 -
.../spark/structuredstreaming/package-info.java | 20 -
.../translation/AbstractTranslationContext.java | 235 ----------
.../translation/PipelineTranslator.java | 202 ---------
.../translation/SparkSessionFactory.java | 71 ---
.../translation/SparkTransformOverrides.java | 56 ---
.../translation/TranslationContext.java | 34 --
.../translation/batch/AggregatorCombiner.java | 270 -----------
.../batch/CombinePerKeyTranslatorBatch.java | 114 -----
.../CreatePCollectionViewTranslatorBatch.java | 60 ---
.../translation/batch/DatasetSourceBatch.java | 160 -------
.../translation/batch/DoFnFunction.java | 164 -------
.../translation/batch/DoFnRunnerWithMetrics.java | 104 -----
.../translation/batch/FlattenTranslatorBatch.java | 65 ---
.../batch/GroupByKeyTranslatorBatch.java | 90 ----
.../translation/batch/ImpulseTranslatorBatch.java | 49 --
.../translation/batch/ParDoTranslatorBatch.java | 257 -----------
.../translation/batch/PipelineTranslatorBatch.java | 101 -----
.../translation/batch/ProcessContext.java | 138 ------
.../batch/ReadSourceTranslatorBatch.java | 88 ----
.../batch/ReshuffleTranslatorBatch.java | 30 --
.../batch/WindowAssignTranslatorBatch.java | 61 ---
.../GroupAlsoByWindowViaOutputBufferFn.java | 173 --------
.../batch/functions/NoOpStepContext.java | 36 --
.../batch/functions/SparkSideInputReader.java | 186 --------
.../translation/batch/functions/package-info.java | 20 -
.../translation/batch/package-info.java | 20 -
.../translation/helpers/CoderHelpers.java | 63 ---
.../translation/helpers/EncoderFactory.java | 49 --
.../translation/helpers/EncoderHelpers.java | 71 ---
.../translation/helpers/MultiOutputCoder.java | 84 ----
.../translation/helpers/RowHelpers.java | 75 ----
.../translation/helpers/SchemaHelpers.java | 39 --
.../translation/helpers/SideInputBroadcast.java | 49 --
.../translation/helpers/WindowingHelpers.java | 82 ----
.../translation/helpers/package-info.java | 20 -
.../translation/package-info.java | 20 -
.../streaming/DatasetSourceStreaming.java | 260 -----------
.../streaming/PipelineTranslatorStreaming.java | 93 ----
.../streaming/ReadSourceTranslatorStreaming.java | 87 ----
.../translation/streaming/package-info.java | 20 -
.../translation/utils/CachedSideInputReader.java | 93 ----
.../translation/utils/SideInputStorage.java | 107 -----
.../translation/utils/package-info.java | 20 -
.../structuredstreaming/SparkSessionRule.java | 88 ----
...parkStructuredStreamingRunnerRegistrarTest.java | 70 ---
.../StructuredStreamingPipelineStateTest.java | 225 ----------
.../aggregators/metrics/sink/InMemoryMetrics.java | 80 ----
.../metrics/sink/SparkMetricsSinkTest.java | 73 ---
.../metrics/SparkBeamMetricTest.java | 59 ---
.../translation/batch/CombineTest.java | 186 --------
.../translation/batch/ComplexSourceTest.java | 86 ----
.../translation/batch/FlattenTest.java | 59 ---
.../translation/batch/GroupByKeyTest.java | 124 ------
.../translation/batch/ParDoTest.java | 153 -------
.../translation/batch/SimpleSourceTest.java | 53 ---
.../translation/batch/WindowAssignTest.java | 69 ---
.../translation/helpers/EncoderHelpersTest.java | 98 ----
.../translation/streaming/SimpleSourceTest.java | 57 ---
.../utils/SerializationDebugger.java | 115 -----
.../structuredstreaming/utils/package-info.java | 20 -
.../translation/EvaluationContext.java | 10 +-
.../translation/PipelineTranslator.java | 97 +++-
.../translation/TransformTranslator.java | 13 +-
.../batch/CombineGloballyTranslatorBatch.java | 4 +
.../batch/CombineGroupedValuesTranslatorBatch.java | 4 +
.../batch/CombinePerKeyTranslatorBatch.java | 4 +
.../translation/batch/FlattenTranslatorBatch.java | 4 +
.../batch/GroupByKeyTranslatorBatch.java | 5 +-
.../translation/batch/ImpulseTranslatorBatch.java | 4 +
.../translation/batch/ParDoTranslatorBatch.java | 6 +-
.../batch/ReadSourceTranslatorBatch.java | 7 +-
.../batch/ReshuffleTranslatorBatch.java | 8 +
.../batch/WindowAssignTranslatorBatch.java | 4 +
.../translation/helpers/EncoderHelpers.java | 19 +-
runners/spark/spark_runner.gradle | 16 +-
.../spark/translation/SparkContextFactory.java | 8 +-
.../SparkStreamingPortablePipelineTranslator.java | 4 +-
.../spark/translation/TransformTranslator.java | 11 +-
.../streaming/StreamingTransformTranslator.java | 4 +-
.../beam/runners/spark/util/SparkCompat.java | 117 -----
sdks/go.mod | 16 +-
sdks/go.sum | 8 +-
sdks/go/cmd/specialize/main.go | 5 +-
.../runners/dataflow/dataflowlib/metrics_test.go | 2 +-
sdks/go/pkg/beam/runners/prism/README.md | 190 ++++++++
sdks/go/pkg/beam/runners/prism/internal/README.md | 62 +++
.../pkg/beam/runners/prism/internal/urns/urns.go | 128 ++++++
.../prism/internal/urns/urns_test.go} | 29 +-
sdks/go/pkg/beam/util/shimx/generate.go | 5 +-
.../sdk/schemas/FieldValueTypeInformation.java | 18 +
.../annotations/SchemaFieldDescription.java | 62 +++
.../sdk/schemas/utils/StaticSchemaInference.java | 11 +-
.../beam/sdk/transforms/DoFnOutputReceivers.java | 2 +-
.../util/UnboundedScheduledExecutorService.java | 28 +-
.../beam/sdk/schemas/AutoValueSchemaTest.java | 32 ++
.../beam/sdk/schemas/JavaBeanSchemaTest.java | 11 +
.../beam/sdk/schemas/JavaFieldSchemaTest.java | 9 +
.../beam/sdk/schemas/utils/TestJavaBeans.java | 23 +
.../apache/beam/sdk/schemas/utils/TestPOJOs.java | 39 ++
.../UnboundedScheduledExecutorServiceTest.java | 54 +++
.../extensions/sql/expansion-service/build.gradle | 1 +
.../SqlTransformSchemaTransformProvider.java | 232 ++++++++++
.../beam/sdk/extensions/sql/SqlTransform.java | 3 +-
.../extensions/sql/impl/planner/BeamRuleSets.java | 4 -
.../sql/impl/rel/AbstractBeamCalcRel.java | 10 +
.../sql/impl/rule/LogicalCalcMergeRule.java | 42 --
.../extensions/sql/BeamSqlMultipleSchemasTest.java | 21 +
.../sql/impl/rule/IOPushDownRuleTest.java | 2 -
...tTableProviderWithFilterAndProjectPushDown.java | 2 -
.../test/TestTableProviderWithFilterPushDown.java | 2 -
.../test/TestTableProviderWithProjectPushDown.java | 2 -
.../sql/zetasql/ZetaSQLQueryPlanner.java | 15 +-
.../jmh/PrecombineGroupingTableBenchmark.java | 87 ++--
.../java/org/apache/beam/fn/harness/Caches.java | 43 +-
.../apache/beam/fn/harness/FnApiDoFnRunner.java | 309 ++++++++++++-
.../beam/fn/harness/PrecombineGroupingTable.java | 10 +-
.../fn/harness/control/ExecutionStateSampler.java | 109 ++++-
.../fn/harness/control/ProcessBundleHandler.java | 57 +--
.../harness/data/PCollectionConsumerRegistry.java | 74 +--
.../harness/data/PTransformFunctionRegistry.java | 26 +-
.../fn/harness/PrecombineGroupingTableTest.java | 33 +-
.../harness/control/ExecutionStateSamplerTest.java | 144 ++++++
.../harness/control/ProcessBundleHandlerTest.java | 11 +-
.../data/PCollectionConsumerRegistryTest.java | 145 +++---
.../data/PTransformFunctionRegistryTest.java | 81 ++--
.../beam/sdk/io/common/SchemaAwareJavaBeans.java | 494 +++++++++++++++++++++
.../sdk/io/common/SchemaAwareJavaBeansTest.java | 175 ++++++++
.../OWNERS | 3 +
.../build.gradle | 20 +-
.../AvroWriteSchemaTransformFormatProvider.java | 88 ++++
.../CsvWriteSchemaTransformFormatProvider.java | 29 +-
.../FileWriteSchemaTransformConfiguration.java | 66 +--
.../FileWriteSchemaTransformFormatProvider.java | 16 +-
.../FileWriteSchemaTransformFormatProviders.java | 148 ++++++
.../FileWriteSchemaTransformProvider.java | 191 ++++++++
.../JsonWriteSchemaTransformFormatProvider.java | 93 ++++
.../ParquetWriteSchemaTransformFormatProvider.java | 117 +++++
.../io/fileschematransform/XmlDateTimeAdapter.java | 34 +-
.../sdk/io/fileschematransform/XmlRowAdapter.java | 93 ++++
.../sdk/io/fileschematransform/XmlRowValue.java | 268 +++++++++++
.../XmlWriteSchemaTransformFormatProvider.java | 115 +++++
.../sdk/io/fileschematransform/package-info.java | 0
...FileWriteSchemaTransformFormatProviderTest.java | 104 +++++
...FileWriteSchemaTransformFormatProviderTest.java | 434 ++++++++++++++++++
...WriteSchemaTransformFormatProviderTestData.java | 231 ++++++++++
...ileWriteSchemaTransformFormatProvidersTest.java | 17 +-
.../FileWriteSchemaTransformProviderTest.java | 127 ++++++
...FileWriteSchemaTransformFormatProviderTest.java | 104 +++++
...FileWriteSchemaTransformFormatProviderTest.java | 112 +++++
.../XmlDateTimeAdapterTest.java} | 32 +-
...FileWriteSchemaTransformFormatProviderTest.java | 120 +++++
.../io/fileschematransform/XmlRowAdapterTest.java | 190 ++++++++
.../io/fileschematransform/XmlRowValueTest.java | 391 ++++++++++++++++
.../FileWriteSchemaTransformFormatProviders.java | 124 ------
.../beam/sdk/io/gcp/bigquery/BigQueryIO.java | 12 +-
.../io/gcp/bigquery/BigQueryStorageAvroReader.java | 10 +-
.../apache/beam/sdk/io/gcp/healthcare/HL7v2IO.java | 272 +++++++++---
.../beam/sdk/io/gcp/healthcare/HL7v2Message.java | 26 ++
.../sdk/io/gcp/healthcare/HL7v2ReadParameter.java | 85 ++++
.../sdk/io/gcp/healthcare/HL7v2ReadResponse.java | 96 ++++
.../io/gcp/healthcare/HL7v2ReadResponseCoder.java | 58 +++
.../beam/sdk/io/gcp/spanner/ReadSpannerSchema.java | 37 +-
.../beam/sdk/io/gcp/spanner/SpannerSchema.java | 8 +-
...erChangestreamsReadSchemaTransformProvider.java | 394 ++++++++++++++++
.../action/DetectNewPartitionsAction.java | 2 +-
.../action/QueryChangeStreamAction.java | 2 +-
.../changestreams/dao/PartitionMetadataDao.java | 85 +++-
.../io/gcp/bigquery/BigQueryIOStorageReadTest.java | 50 +++
.../sdk/io/gcp/bigquery/BigQueryIOWriteTest.java | 14 +
.../sdk/io/gcp/healthcare/HL7v2IOReadWriteIT.java | 84 ++++
.../beam/sdk/io/gcp/healthcare/HL7v2IOTest.java | 41 +-
.../dao/PartitionMetadataDaoTest.java | 108 ++++-
.../it/ChangeStreamTestPipelineOptions.java | 8 +-
.../changestreams/it/IntegrationTestEnv.java | 62 ++-
.../changestreams/it/SpannerChangeStreamIT.java | 40 +-
.../it/SpannerChangeStreamsSchemaTransformIT.java | 262 +++++++++++
sdks/java/io/kafka/build.gradle | 2 +-
sdks/java/testing/watermarks/build.gradle | 19 -
sdks/python/apache_beam/coders/row_coder_test.py | 3 +-
.../inference/large_language_modeling/main.py | 4 +-
.../inference/pytorch_image_classification.py | 10 +-
.../inference/run_inference_side_inputs.py | 6 +-
.../inference/tensorrt_text_classification.py | 126 ++++++
.../io/gcp/big_query_query_to_table_it_test.py | 6 +-
.../apache_beam/io/gcp/bigquery_file_loads_test.py | 8 +-
.../apache_beam/io/gcp/bigquery_read_it_test.py | 10 +-
sdks/python/apache_beam/io/gcp/bigquery_test.py | 14 +-
.../apache_beam/io/gcp/bigquery_write_it_test.py | 8 +-
sdks/python/apache_beam/io/gcp/tests/utils.py | 6 +-
sdks/python/apache_beam/ml/inference/base.py | 10 +-
sdks/python/apache_beam/ml/inference/base_test.py | 6 +-
.../apache_beam/ml/inference/pytorch_inference.py | 24 +-
.../ml/inference/pytorch_inference_test.py | 92 ++++
.../apache_beam/ml/inference/sklearn_inference.py | 24 +-
.../ml/inference/sklearn_inference_test.py | 68 +++
.../apache_beam/ml/inference/tensorrt_inference.py | 11 +-
.../python/apache_beam/options/pipeline_options.py | 5 +-
sdks/python/apache_beam/pipeline_test.py | 16 +
.../runners/portability/spark_runner.py | 4 +-
.../portability/spark_uber_jar_job_server.py | 4 +-
.../apache_beam/runners/worker/bundle_processor.py | 23 +
sdks/python/apache_beam/transforms/environments.py | 9 +-
sdks/python/apache_beam/transforms/util.py | 40 +-
sdks/python/apache_beam/transforms/util_test.py | 18 +-
sdks/python/build-requirements.txt | 2 +-
.../container/py310/base_image_requirements.txt | 2 +-
.../container/py38/base_image_requirements.txt | 2 +-
.../container/py39/base_image_requirements.txt | 4 +-
sdks/python/setup.py | 15 +-
sdks/python/test-suites/direct/common.gradle | 3 +-
sdks/python/test-suites/tox/py38/build.gradle | 26 +-
sdks/python/tox.ini | 15 +-
sdks/typescript/package.json | 1 +
settings.gradle.kts | 5 +-
website/www/site/assets/scss/_banner.sass | 43 ++
website/www/site/assets/scss/_hero.scss | 3 +-
website/www/site/assets/scss/_navbar-desktop.scss | 3 +-
website/www/site/assets/scss/main.scss | 1 +
.../documentation/io/built-in/google-bigquery.md | 4 +-
.../en/documentation/ml/large-language-modeling.md | 26 +-
.../site/content/en/documentation/ml/overview.md | 5 +-
.../en/documentation/ml/tensorrt-runinference.md | 150 +++++++
.../site/content/en/documentation/runners/spark.md | 2 +-
.../get-started/resources/videos-and-podcasts.md | 7 +
website/www/site/layouts/partials/header.html | 6 +
.../partials/section-menu/en/documentation.html | 1 +
website/www/site/static/images/banner_desktop.jpg | Bin 0 -> 111671 bytes
website/www/site/static/images/banner_mobile.jpg | Bin 0 -> 59518 bytes
290 files changed, 8417 insertions(+), 8793 deletions(-)
create mode 100644 .github/workflows/playground_frontend_test.yml
delete mode 100644 .test-infra/jenkins/job_PostCommit_CrossLanguageValidatesRunner_Spark.groovy
delete mode 100644 .test-infra/jenkins/job_PostCommit_Java_PortableValidatesRunner_Spark2_Streaming.groovy
copy playground/frontend/assets/{github.svg => colab.svg} (53%)
rename runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/helpers/KVHelpers.java => playground/frontend/lib/components/link_button/colab_button.dart (58%)
rename runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/utils/ScalaInterop.java => playground/frontend/lib/components/link_button/dataset_button.dart (50%)
rename runners/spark/2/src/test/java/org/apache/beam/runners/spark/structuredstreaming/aggregators/metrics/sink/InMemoryMetricsSinkRule.java => playground/frontend/lib/components/link_button/github_button.dart (59%)
create mode 100644 playground/frontend/lib/components/link_button/link_button.dart
create mode 100644 playground/frontend/lib/modules/examples/components/example_actions.dart
delete mode 100644 runners/spark/2/build.gradle
delete mode 100644 runners/spark/2/job-server/build.gradle
delete mode 100644 runners/spark/2/job-server/container/build.gradle
delete mode 100644 runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/Constants.java
delete mode 100644 runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/SparkStructuredStreamingPipelineOptions.java
delete mode 100644 runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/SparkStructuredStreamingPipelineResult.java
delete mode 100644 runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/SparkStructuredStreamingRunner.java
delete mode 100644 runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/SparkStructuredStreamingRunnerRegistrar.java
delete mode 100644 runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/aggregators/AggregatorsAccumulator.java
delete mode 100644 runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/aggregators/NamedAggregators.java
delete mode 100644 runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/aggregators/NamedAggregatorsAccumulator.java
delete mode 100644 runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/aggregators/package-info.java
delete mode 100644 runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/examples/WordCount.java
delete mode 100644 runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/metrics/AggregatorMetric.java
delete mode 100644 runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/metrics/AggregatorMetricSource.java
delete mode 100644 runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/metrics/BeamMetricSet.java
delete mode 100644 runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/metrics/MetricsAccumulator.java
delete mode 100644 runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/metrics/MetricsContainerStepMapAccumulator.java
delete mode 100644 runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/metrics/SparkBeamMetric.java
delete mode 100644 runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/metrics/SparkBeamMetricSource.java
delete mode 100644 runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/metrics/SparkMetricsContainerStepMap.java
delete mode 100644 runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/metrics/WithMetricsSupport.java
delete mode 100644 runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/metrics/package-info.java
delete mode 100644 runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/metrics/sink/CodahaleCsvSink.java
delete mode 100644 runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/metrics/sink/CodahaleGraphiteSink.java
delete mode 100644 runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/metrics/sink/package-info.java
delete mode 100644 runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/package-info.java
delete mode 100644 runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/AbstractTranslationContext.java
delete mode 100644 runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/PipelineTranslator.java
delete mode 100644 runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/SparkSessionFactory.java
delete mode 100644 runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/SparkTransformOverrides.java
delete mode 100644 runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/TranslationContext.java
delete mode 100644 runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/AggregatorCombiner.java
delete mode 100644 runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/CombinePerKeyTranslatorBatch.java
delete mode 100644 runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/CreatePCollectionViewTranslatorBatch.java
delete mode 100644 runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/DatasetSourceBatch.java
delete mode 100644 runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/DoFnFunction.java
delete mode 100644 runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/DoFnRunnerWithMetrics.java
delete mode 100644 runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/FlattenTranslatorBatch.java
delete mode 100644 runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/GroupByKeyTranslatorBatch.java
delete mode 100644 runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/ImpulseTranslatorBatch.java
delete mode 100644 runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/ParDoTranslatorBatch.java
delete mode 100644 runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/PipelineTranslatorBatch.java
delete mode 100644 runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/ProcessContext.java
delete mode 100644 runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/ReadSourceTranslatorBatch.java
delete mode 100644 runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/ReshuffleTranslatorBatch.java
delete mode 100644 runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/WindowAssignTranslatorBatch.java
delete mode 100644 runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/functions/GroupAlsoByWindowViaOutputBufferFn.java
delete mode 100644 runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/functions/NoOpStepContext.java
delete mode 100644 runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/functions/SparkSideInputReader.java
delete mode 100644 runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/functions/package-info.java
delete mode 100644 runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/package-info.java
delete mode 100644 runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/helpers/CoderHelpers.java
delete mode 100644 runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/helpers/EncoderFactory.java
delete mode 100644 runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/helpers/EncoderHelpers.java
delete mode 100644 runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/helpers/MultiOutputCoder.java
delete mode 100644 runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/helpers/RowHelpers.java
delete mode 100644 runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/helpers/SchemaHelpers.java
delete mode 100644 runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/helpers/SideInputBroadcast.java
delete mode 100644 runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/helpers/WindowingHelpers.java
delete mode 100644 runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/helpers/package-info.java
delete mode 100644 runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/package-info.java
delete mode 100644 runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/streaming/DatasetSourceStreaming.java
delete mode 100644 runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/streaming/PipelineTranslatorStreaming.java
delete mode 100644 runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/streaming/ReadSourceTranslatorStreaming.java
delete mode 100644 runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/streaming/package-info.java
delete mode 100644 runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/utils/CachedSideInputReader.java
delete mode 100644 runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/utils/SideInputStorage.java
delete mode 100644 runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/utils/package-info.java
delete mode 100644 runners/spark/2/src/test/java/org/apache/beam/runners/spark/structuredstreaming/SparkSessionRule.java
delete mode 100644 runners/spark/2/src/test/java/org/apache/beam/runners/spark/structuredstreaming/SparkStructuredStreamingRunnerRegistrarTest.java
delete mode 100644 runners/spark/2/src/test/java/org/apache/beam/runners/spark/structuredstreaming/StructuredStreamingPipelineStateTest.java
delete mode 100644 runners/spark/2/src/test/java/org/apache/beam/runners/spark/structuredstreaming/aggregators/metrics/sink/InMemoryMetrics.java
delete mode 100644 runners/spark/2/src/test/java/org/apache/beam/runners/spark/structuredstreaming/aggregators/metrics/sink/SparkMetricsSinkTest.java
delete mode 100644 runners/spark/2/src/test/java/org/apache/beam/runners/spark/structuredstreaming/metrics/SparkBeamMetricTest.java
delete mode 100644 runners/spark/2/src/test/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/CombineTest.java
delete mode 100644 runners/spark/2/src/test/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/ComplexSourceTest.java
delete mode 100644 runners/spark/2/src/test/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/FlattenTest.java
delete mode 100644 runners/spark/2/src/test/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/GroupByKeyTest.java
delete mode 100644 runners/spark/2/src/test/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/ParDoTest.java
delete mode 100644 runners/spark/2/src/test/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/SimpleSourceTest.java
delete mode 100644 runners/spark/2/src/test/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/WindowAssignTest.java
delete mode 100644 runners/spark/2/src/test/java/org/apache/beam/runners/spark/structuredstreaming/translation/helpers/EncoderHelpersTest.java
delete mode 100644 runners/spark/2/src/test/java/org/apache/beam/runners/spark/structuredstreaming/translation/streaming/SimpleSourceTest.java
delete mode 100644 runners/spark/2/src/test/java/org/apache/beam/runners/spark/structuredstreaming/utils/SerializationDebugger.java
delete mode 100644 runners/spark/2/src/test/java/org/apache/beam/runners/spark/structuredstreaming/utils/package-info.java
delete mode 100644 runners/spark/src/main/java/org/apache/beam/runners/spark/util/SparkCompat.java
create mode 100644 sdks/go/pkg/beam/runners/prism/README.md
create mode 100644 sdks/go/pkg/beam/runners/prism/internal/README.md
create mode 100644 sdks/go/pkg/beam/runners/prism/internal/urns/urns.go
rename sdks/go/pkg/beam/{core/runtime/xlangx/expand_test.go => runners/prism/internal/urns/urns_test.go} (56%)
create mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/annotations/SchemaFieldDescription.java
create mode 100644 sdks/java/extensions/sql/expansion-service/src/main/java/org/apache/beam/sdk/extensions/sql/expansion/SqlTransformSchemaTransformProvider.java
delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/LogicalCalcMergeRule.java
create mode 100644 sdks/java/io/common/src/test/java/org/apache/beam/sdk/io/common/SchemaAwareJavaBeans.java
create mode 100644 sdks/java/io/common/src/test/java/org/apache/beam/sdk/io/common/SchemaAwareJavaBeansTest.java
rename sdks/java/io/{fileschematransform => file-schema-transform}/OWNERS (55%)
rename sdks/java/io/{fileschematransform => file-schema-transform}/build.gradle (70%)
create mode 100644 sdks/java/io/file-schema-transform/src/main/java/org/apache/beam/sdk/io/fileschematransform/AvroWriteSchemaTransformFormatProvider.java
rename runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/TransformTranslator.java => sdks/java/io/file-schema-transform/src/main/java/org/apache/beam/sdk/io/fileschematransform/CsvWriteSchemaTransformFormatProvider.java (50%)
rename sdks/java/io/{fileschematransform => file-schema-transform}/src/main/java/org/apache/beam/sdk/io/fileschematransform/FileWriteSchemaTransformConfiguration.java (77%)
rename sdks/java/io/{fileschematransform => file-schema-transform}/src/main/java/org/apache/beam/sdk/io/fileschematransform/FileWriteSchemaTransformFormatProvider.java (66%)
create mode 100644 sdks/java/io/file-schema-transform/src/main/java/org/apache/beam/sdk/io/fileschematransform/FileWriteSchemaTransformFormatProviders.java
create mode 100644 sdks/java/io/file-schema-transform/src/main/java/org/apache/beam/sdk/io/fileschematransform/FileWriteSchemaTransformProvider.java
create mode 100644 sdks/java/io/file-schema-transform/src/main/java/org/apache/beam/sdk/io/fileschematransform/JsonWriteSchemaTransformFormatProvider.java
create mode 100644 sdks/java/io/file-schema-transform/src/main/java/org/apache/beam/sdk/io/fileschematransform/ParquetWriteSchemaTransformFormatProvider.java
rename runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/metrics/CompositeSource.java => sdks/java/io/file-schema-transform/src/main/java/org/apache/beam/sdk/io/fileschematransform/XmlDateTimeAdapter.java (50%)
create mode 100644 sdks/java/io/file-schema-transform/src/main/java/org/apache/beam/sdk/io/fileschematransform/XmlRowAdapter.java
create mode 100644 sdks/java/io/file-schema-transform/src/main/java/org/apache/beam/sdk/io/fileschematransform/XmlRowValue.java
create mode 100644 sdks/java/io/file-schema-transform/src/main/java/org/apache/beam/sdk/io/fileschematransform/XmlWriteSchemaTransformFormatProvider.java
rename sdks/java/io/{fileschematransform => file-schema-transform}/src/main/java/org/apache/beam/sdk/io/fileschematransform/package-info.java (100%)
create mode 100644 sdks/java/io/file-schema-transform/src/test/java/org/apache/beam/sdk/io/fileschematransform/AvroFileWriteSchemaTransformFormatProviderTest.java
create mode 100644 sdks/java/io/file-schema-transform/src/test/java/org/apache/beam/sdk/io/fileschematransform/FileWriteSchemaTransformFormatProviderTest.java
create mode 100644 sdks/java/io/file-schema-transform/src/test/java/org/apache/beam/sdk/io/fileschematransform/FileWriteSchemaTransformFormatProviderTestData.java
copy sdks/java/io/{fileschematransform => file-schema-transform}/src/test/java/org/apache/beam/sdk/io/fileschematransform/FileWriteSchemaTransformFormatProvidersTest.java (64%)
create mode 100644 sdks/java/io/file-schema-transform/src/test/java/org/apache/beam/sdk/io/fileschematransform/FileWriteSchemaTransformProviderTest.java
create mode 100644 sdks/java/io/file-schema-transform/src/test/java/org/apache/beam/sdk/io/fileschematransform/JsonFileWriteSchemaTransformFormatProviderTest.java
create mode 100644 sdks/java/io/file-schema-transform/src/test/java/org/apache/beam/sdk/io/fileschematransform/ParquetFileWriteSchemaTransformFormatProviderTest.java
rename sdks/java/io/{fileschematransform/src/test/java/org/apache/beam/sdk/io/fileschematransform/FileWriteSchemaTransformFormatProvidersTest.java => file-schema-transform/src/test/java/org/apache/beam/sdk/io/fileschematransform/XmlDateTimeAdapterTest.java} (56%)
create mode 100644 sdks/java/io/file-schema-transform/src/test/java/org/apache/beam/sdk/io/fileschematransform/XmlFileWriteSchemaTransformFormatProviderTest.java
create mode 100644 sdks/java/io/file-schema-transform/src/test/java/org/apache/beam/sdk/io/fileschematransform/XmlRowAdapterTest.java
create mode 100644 sdks/java/io/file-schema-transform/src/test/java/org/apache/beam/sdk/io/fileschematransform/XmlRowValueTest.java
delete mode 100644 sdks/java/io/fileschematransform/src/main/java/org/apache/beam/sdk/io/fileschematransform/FileWriteSchemaTransformFormatProviders.java
create mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/healthcare/HL7v2ReadParameter.java
create mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/healthcare/HL7v2ReadResponse.java
create mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/healthcare/HL7v2ReadResponseCoder.java
create mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/SpannerChangestreamsReadSchemaTransformProvider.java
create mode 100644 sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/it/SpannerChangeStreamsSchemaTransformIT.java
create mode 100644 sdks/python/apache_beam/examples/inference/tensorrt_text_classification.py
create mode 100644 website/www/site/assets/scss/_banner.sass
create mode 100644 website/www/site/content/en/documentation/ml/tensorrt-runinference.md
create mode 100644 website/www/site/static/images/banner_desktop.jpg
create mode 100644 website/www/site/static/images/banner_mobile.jpg