You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/05/23 02:42:34 UTC
[GitHub] [beam] epicfaace opened a new pull request #11802: [BEAM-9916] Update I/O documentation links and create more complete I/O matrix
epicfaace opened a new pull request #11802:
URL: https://github.com/apache/beam/pull/11802
Update I/O documentation links to point to the appropriate pydoc / javadoc / godoc pages. Also, create a more complete I/O matrix that has all the existing I/O's used in all three languages, with a tab to switch between languages, too. R: @pabloem
The page now looks like this:
Additionally, I've also added links to the built-in I/O connector guides on the side bar, because previously they didn't exist:
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)
XLang | --- | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/)
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] datancoffee commented on pull request #11802: [BEAM-9916] Update I/O documentation links and create more complete I/O matrix
Posted by GitBox <gi...@apache.org>.
datancoffee commented on pull request #11802:
URL: https://github.com/apache/beam/pull/11802#issuecomment-634267799
Can we also add a section on "Applications" for application-specific connectors?
---
Add a section "Applications" after the "Database" section but before "Miscellaneous"
FhirIO provides an API for reading and writing resources to <a href="https://cloud.google.com/healthcare/docs/concepts/fhir">Google Cloud Healthcare Fhir API. </a>
https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/healthcare/FhirIO.java
HL7v2IO provides an API for reading from and writing to <a
href="https://cloud.google.com/healthcare/docs/concepts/hl7v2">Google Cloud Healthcare HL7v2 API. </a>
https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/healthcare/HL7v2IO.java
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] datancoffee commented on pull request #11802: [BEAM-9916] Update I/O documentation links and create more complete I/O matrix
Posted by GitBox <gi...@apache.org>.
datancoffee commented on pull request #11802:
URL: https://github.com/apache/beam/pull/11802#issuecomment-634437572
> FhirIO and HL7v2IO are not yet part of a release (they will be in 2.22.0 and 2.21.0 respectively), so we can skip that update for now. @jaketf will add them once their corresponding releases are out.
Sounds good. Thanks, Pablo!
New page looks much more readable than before (and can be extended with new sections)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev removed a comment on pull request #11802: [BEAM-9916] Update I/O documentation links and create more complete I/O matrix
Posted by GitBox <gi...@apache.org>.
aromanenko-dev removed a comment on pull request #11802:
URL: https://github.com/apache/beam/pull/11802#issuecomment-634093933
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] garrettjonesgoogle commented on a change in pull request #11802: [BEAM-9916] Update I/O documentation links and create more complete I/O matrix
Posted by GitBox <gi...@apache.org>.
garrettjonesgoogle commented on a change in pull request #11802:
URL: https://github.com/apache/beam/pull/11802#discussion_r431403908
##########
File path: website/www/site/data/io_matrix.yaml
##########
@@ -0,0 +1,377 @@
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+categories:
+ - name: File-based
+ description: These I/O connectors involve working with files.
+ rows:
+ - transform: FileIO
+ description: "General-purpose transforms for working with files: listing files (matching), reading and writing."
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.FileIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/FileIO.html
+ - language: py
+ name: apache_beam.io.FileIO
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.fileio.html
+ - transform: AvroIO
+ description: PTransforms for reading from and writing to [Avro](https://avro.apache.org/) files.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.AvroIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/AvroIO.html
+ - language: py
+ name: apache_beam.io.avroio
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.avroio.html
+ - language: go
+ name: github.com/apache/beam/sdks/go/pkg/beam/io/avroio
+ url: https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam/io/avroio
+ - transform: TextIO
+ description: PTransforms for reading and writing text files.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.TextIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/TextIO.html
+ - language: py
+ name: apache_beam.io.textio
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.textio.html
+ - language: go
+ name: github.com/apache/beam/sdks/go/pkg/beam/io/textio
+ url: https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam/io/textio
+ - transform: TFRecordIO
+ description: PTransforms for reading and writing [TensorFlow TFRecord](https://www.tensorflow.org/tutorials/load_data/tfrecord) files.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.TFRecordIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/TFRecordIO.html
+ - language: py
+ name: apache_beam.io.tfrecordio
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.tfrecordio.html
+ - transform: XmlIO
+ description: Transforms for reading and writing XML files using [JAXB](https://www.oracle.com/technical-resources/articles/javase/jaxb.html) mappers.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.xml.XmlIO
+ url: https://beam.apache.org/releases/javadoc/2.3.0/org/apache/beam/sdk/io/xml/XmlIO.html
+ - transform: TikaIO
+ description: Transforms for parsing arbitrary files using [Apache Tika](https://tika.apache.org/).
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.tika.TikaIO
+ url: https://beam.apache.org/releases/javadoc/2.3.0/org/apache/beam/sdk/io/tika/TikaIO.html
+ - transform: ParquetIO
+ description: IO for reading from and writing to [Parquet](https://parquet.apache.org/) files.
+ docs: /documentation/io/built-in/parquet/
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.parquet.ParquetIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/parquet/ParquetIO.html
+ - language: py
+ name: apache_beam.io.parquetio
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.parquetio.html
+ - transform: ThriftIO
+ description: PTransforms for reading and writing files containing [Thrift](https://thrift.apache.org/)-encoded data.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.thrift.ThriftIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/thrift/ThriftIO.html
+ - transform: VcfIO
+ description: A source for reading from [VCF files](https://samtools.github.io/hts-specs/VCFv4.2.pdf) (version 4.x).
+ implementations:
+ - language: py
+ name: apache_beam.io.vcfio
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.vcfio.html
+ - transform: S3IO
+ description: A source for reading from and writing to [Amazon S3](https://aws.amazon.com/s3/).
+ implementations:
+ - language: py
+ name: apache_beam.io.aws.s3io
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.aws.s3io.html
+ - transform: GcsIO
+ description: A source for reading from and writing to [Google Cloud Storage](https://cloud.google.com/storage).
+ implementations:
+ - language: py
+ name: apache_beam.io.gcp.gcsio
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.gcp.gcsio.html
+ - name: FileSystem
+ description: Beam provides a File system interface that defines APIs for writing file systems agnostic code. Several I/O connectors are implemented as a FileSystem implementation.
+ rows:
+ - transform: HadoopFileSystem
+ description: "`FileSystem` implementation for accessing [Hadoop](https://hadoop.apache.org/) Distributed File System files."
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.hdfs.HadoopFileSystemRegistrar
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/hdfs/HadoopFileSystemRegistrar.html
+ - language: py
+ name: apache_beam.io.hadoopfilesystem
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.hadoopfilesystem.html
+ - transform: GcsFileSystem
+ description: "`FileSystem` implementation for [Google Cloud Storage](https://cloud.google.com/storage)."
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.extensions.gcp.storage.GcsFileSystemRegistrar
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/extensions/gcp/storage/GcsFileSystemRegistrar.html
+ - language: py
+ name: apache_beam.io.gcp.gcsfilesystem
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.gcp.gcsfilesystem.html
+ - language: go
+ name: github.com/apache/beam/sdks/go/pkg/beam/io/filesystem/gcs
+ url: https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam/io/filesystem/gcs
+ - transform: LocalFileSystem
+ description: "`FileSystem` implementation for accessing files on disk."
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.LocalFileSystemRegistrar
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/LocalFileSystemRegistrar.html
+ - language: py
+ name: apache_beam.io.localfilesystem
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.localfilesystem.html
+ - language: go
+ name: github.com/apache/beam/sdks/go/pkg/beam/io/filesystem/local
+ url: https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam/io/filesystem/local
+ - transform: S3FileSystem
+ description: "`FileSystem` implementation for [Amazon S3](https://aws.amazon.com/s3/)."
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.aws.s3.S3FileSystemRegistrar
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/hdfs/package-summary.html
+ - transform: In-memory
+ description: "`FileSystem` implementation in memory; useful for testing."
+ implementations:
+ - language: go
+ name: github.com/apache/beam/sdks/go/pkg/beam/io/filesystem/memfs
+ url: https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam/io/filesystem/memfs
+ - name: Messaging
+ description: These I/O connectors typically involve working with unbounded sources that come from messaging sources.
+ rows:
+ - transform: KinesisIO
+ description: PTransforms for reading from and writing to [Kinesis](https://aws.amazon.com/kinesis/) streams.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.kinesis.KinesisIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/kinesis/KinesisIO.html
+ - transform: AmqpIO
+ description: AMQP 1.0 protocol using the Apache QPid Proton-J library
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.amqp.AmqpIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/amqp/AmqpIO.html
+ - transform: KafkaIO
+ description: Read and Write PTransforms for [Apache Kafka](https://kafka.apache.org/).
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.kafka.KafkaIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/kafka/KafkaIO.html
+ - language: py
+ name: apache_beam.io.external.kafka
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.external.kafka.html
+ - transform: PubSubIO
+ description: Read and Write PTransforms for [Google Cloud Pub/Sub](https://cloud.google.com/pubsub) streams.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.gcp.pubsub.PubsubIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.html
+ - language: py
+ name: apache_beam.io.gcp.pubsub
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.gcp.pubsub.html
+ - language: py
+ name: apache_beam.io.external.gcp.pubsub
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.external.gcp.pubsub.html
+ - language: go
+ name: github.com/apache/beam/sdks/go/pkg/beam/io/pubsubio
+ url: https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam/io/pubsubio
+ - transform: JmsIO
+ description: An unbounded source for [JMS](https://www.oracle.com/java/technologies/java-message-service.html) destinations (queues or topics).
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.jms.JmsIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/jms/JmsIO.html
+ - transform: MqttIO
+ description: An unbounded source for [MQTT](https://mqtt.org/) broker.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.mqtt.MqttIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/mqtt/MqttIO.html
+ - transform: RabbitMqIO
+ description: A IO to publish or consume messages with a RabbitMQ broker.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.rabbitmq.RabbitMqIO
+ url: https://github.com/apache/beam/blob/master/sdks/java/io/rabbitmq/src/main/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIO.java
Review comment:
The PR #7197 essentially just switched the exporting of javadocs from an include list to an exclude list, and preserved the bit for all components that existed at that point in time. RabbitMqIO and KuduIO didn't have javadocs exported before my change either. Thus, the explanation for their lack of javadocs would be due to something before me.
My suspicion is that since the exporting of javadocs was an include list before my change, it was easy to miss components, and those two were probably missed accidentally.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on a change in pull request #11802: [BEAM-9916] Update I/O documentation links and create more complete I/O matrix
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on a change in pull request #11802:
URL: https://github.com/apache/beam/pull/11802#discussion_r431233203
##########
File path: website/www/site/data/io_matrix.yaml
##########
@@ -0,0 +1,377 @@
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+categories:
+ - name: File-based
+ description: These I/O connectors involve working with files.
+ rows:
+ - transform: FileIO
+ description: "General-purpose transforms for working with files: listing files (matching), reading and writing."
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.FileIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/FileIO.html
+ - language: py
+ name: apache_beam.io.FileIO
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.fileio.html
+ - transform: AvroIO
+ description: PTransforms for reading from and writing to [Avro](https://avro.apache.org/) files.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.AvroIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/AvroIO.html
+ - language: py
+ name: apache_beam.io.avroio
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.avroio.html
+ - language: go
+ name: github.com/apache/beam/sdks/go/pkg/beam/io/avroio
+ url: https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam/io/avroio
+ - transform: TextIO
+ description: PTransforms for reading and writing text files.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.TextIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/TextIO.html
+ - language: py
+ name: apache_beam.io.textio
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.textio.html
+ - language: go
+ name: github.com/apache/beam/sdks/go/pkg/beam/io/textio
+ url: https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam/io/textio
+ - transform: TFRecordIO
+ description: PTransforms for reading and writing [TensorFlow TFRecord](https://www.tensorflow.org/tutorials/load_data/tfrecord) files.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.TFRecordIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/TFRecordIO.html
+ - language: py
+ name: apache_beam.io.tfrecordio
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.tfrecordio.html
+ - transform: XmlIO
+ description: Transforms for reading and writing XML files using [JAXB](https://www.oracle.com/technical-resources/articles/javase/jaxb.html) mappers.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.xml.XmlIO
+ url: https://beam.apache.org/releases/javadoc/2.3.0/org/apache/beam/sdk/io/xml/XmlIO.html
+ - transform: TikaIO
+ description: Transforms for parsing arbitrary files using [Apache Tika](https://tika.apache.org/).
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.tika.TikaIO
+ url: https://beam.apache.org/releases/javadoc/2.3.0/org/apache/beam/sdk/io/tika/TikaIO.html
+ - transform: ParquetIO
+ description: IO for reading from and writing to [Parquet](https://parquet.apache.org/) files.
+ docs: /documentation/io/built-in/parquet/
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.parquet.ParquetIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/parquet/ParquetIO.html
+ - language: py
+ name: apache_beam.io.parquetio
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.parquetio.html
+ - transform: ThriftIO
+ description: PTransforms for reading and writing files containing [Thrift](https://thrift.apache.org/)-encoded data.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.thrift.ThriftIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/thrift/ThriftIO.html
+ - transform: VcfIO
+ description: A source for reading from [VCF files](https://samtools.github.io/hts-specs/VCFv4.2.pdf) (version 4.x).
+ implementations:
+ - language: py
+ name: apache_beam.io.vcfio
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.vcfio.html
+ - transform: S3IO
+ description: A source for reading from and writing to [Amazon S3](https://aws.amazon.com/s3/).
+ implementations:
+ - language: py
+ name: apache_beam.io.aws.s3io
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.aws.s3io.html
+ - transform: GcsIO
+ description: A source for reading from and writing to [Google Cloud Storage](https://cloud.google.com/storage).
+ implementations:
+ - language: py
+ name: apache_beam.io.gcp.gcsio
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.gcp.gcsio.html
+ - name: FileSystem
+ description: Beam provides a File system interface that defines APIs for writing file systems agnostic code. Several I/O connectors are implemented as a FileSystem implementation.
+ rows:
+ - transform: HadoopFileSystem
+ description: "`FileSystem` implementation for accessing [Hadoop](https://hadoop.apache.org/) Distributed File System files."
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.hdfs.HadoopFileSystemRegistrar
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/hdfs/HadoopFileSystemRegistrar.html
+ - language: py
+ name: apache_beam.io.hadoopfilesystem
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.hadoopfilesystem.html
+ - transform: GcsFileSystem
+ description: "`FileSystem` implementation for [Google Cloud Storage](https://cloud.google.com/storage)."
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.extensions.gcp.storage.GcsFileSystemRegistrar
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/extensions/gcp/storage/GcsFileSystemRegistrar.html
+ - language: py
+ name: apache_beam.io.gcp.gcsfilesystem
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.gcp.gcsfilesystem.html
+ - language: go
+ name: github.com/apache/beam/sdks/go/pkg/beam/io/filesystem/gcs
+ url: https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam/io/filesystem/gcs
+ - transform: LocalFileSystem
+ description: "`FileSystem` implementation for accessing files on disk."
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.LocalFileSystemRegistrar
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/LocalFileSystemRegistrar.html
+ - language: py
+ name: apache_beam.io.localfilesystem
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.localfilesystem.html
+ - language: go
+ name: github.com/apache/beam/sdks/go/pkg/beam/io/filesystem/local
+ url: https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam/io/filesystem/local
+ - transform: S3FileSystem
+ description: "`FileSystem` implementation for [Amazon S3](https://aws.amazon.com/s3/)."
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.aws.s3.S3FileSystemRegistrar
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/hdfs/package-summary.html
+ - transform: In-memory
+ description: "`FileSystem` implementation in memory; useful for testing."
+ implementations:
+ - language: go
+ name: github.com/apache/beam/sdks/go/pkg/beam/io/filesystem/memfs
+ url: https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam/io/filesystem/memfs
+ - name: Messaging
+ description: These I/O connectors typically involve working with unbounded sources that come from messaging sources.
+ rows:
+ - transform: KinesisIO
+ description: PTransforms for reading from and writing to [Kinesis](https://aws.amazon.com/kinesis/) streams.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.kinesis.KinesisIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/kinesis/KinesisIO.html
+ - transform: AmqpIO
+ description: AMQP 1.0 protocol using the Apache QPid Proton-J library
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.amqp.AmqpIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/amqp/AmqpIO.html
+ - transform: KafkaIO
+ description: Read and Write PTransforms for [Apache Kafka](https://kafka.apache.org/).
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.kafka.KafkaIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/kafka/KafkaIO.html
+ - language: py
+ name: apache_beam.io.external.kafka
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.external.kafka.html
+ - transform: PubSubIO
+ description: Read and Write PTransforms for [Google Cloud Pub/Sub](https://cloud.google.com/pubsub) streams.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.gcp.pubsub.PubsubIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.html
+ - language: py
+ name: apache_beam.io.gcp.pubsub
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.gcp.pubsub.html
+ - language: py
+ name: apache_beam.io.external.gcp.pubsub
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.external.gcp.pubsub.html
+ - language: go
+ name: github.com/apache/beam/sdks/go/pkg/beam/io/pubsubio
+ url: https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam/io/pubsubio
+ - transform: JmsIO
+ description: An unbounded source for [JMS](https://www.oracle.com/java/technologies/java-message-service.html) destinations (queues or topics).
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.jms.JmsIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/jms/JmsIO.html
+ - transform: MqttIO
+ description: An unbounded source for [MQTT](https://mqtt.org/) broker.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.mqtt.MqttIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/mqtt/MqttIO.html
+ - transform: RabbitMqIO
+ description: A IO to publish or consume messages with a RabbitMQ broker.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.rabbitmq.RabbitMqIO
+ url: https://github.com/apache/beam/blob/master/sdks/java/io/rabbitmq/src/main/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIO.java
Review comment:
@pabloem Hmm, I don't remember why it was done for only these 2 Java IOs... It was introduced by this PR - https://github.com/apache/beam/pull/7197
For now, I don't see we can't enable it back.
Maybe @garrettjonesgoogle or @swegner, who worked on that PR, could add something?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem merged pull request #11802: [BEAM-9916] Update I/O documentation links and create more complete I/O matrix
Posted by GitBox <gi...@apache.org>.
pabloem merged pull request #11802:
URL: https://github.com/apache/beam/pull/11802
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on a change in pull request #11802: [BEAM-9916] Update I/O documentation links and create more complete I/O matrix
Posted by GitBox <gi...@apache.org>.
pabloem commented on a change in pull request #11802:
URL: https://github.com/apache/beam/pull/11802#discussion_r431442795
##########
File path: website/www/site/data/io_matrix.yaml
##########
@@ -0,0 +1,377 @@
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+categories:
+ - name: File-based
+ description: These I/O connectors involve working with files.
+ rows:
+ - transform: FileIO
+ description: "General-purpose transforms for working with files: listing files (matching), reading and writing."
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.FileIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/FileIO.html
+ - language: py
+ name: apache_beam.io.FileIO
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.fileio.html
+ - transform: AvroIO
+ description: PTransforms for reading from and writing to [Avro](https://avro.apache.org/) files.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.AvroIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/AvroIO.html
+ - language: py
+ name: apache_beam.io.avroio
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.avroio.html
+ - language: go
+ name: github.com/apache/beam/sdks/go/pkg/beam/io/avroio
+ url: https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam/io/avroio
+ - transform: TextIO
+ description: PTransforms for reading and writing text files.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.TextIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/TextIO.html
+ - language: py
+ name: apache_beam.io.textio
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.textio.html
+ - language: go
+ name: github.com/apache/beam/sdks/go/pkg/beam/io/textio
+ url: https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam/io/textio
+ - transform: TFRecordIO
+ description: PTransforms for reading and writing [TensorFlow TFRecord](https://www.tensorflow.org/tutorials/load_data/tfrecord) files.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.TFRecordIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/TFRecordIO.html
+ - language: py
+ name: apache_beam.io.tfrecordio
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.tfrecordio.html
+ - transform: XmlIO
+ description: Transforms for reading and writing XML files using [JAXB](https://www.oracle.com/technical-resources/articles/javase/jaxb.html) mappers.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.xml.XmlIO
+ url: https://beam.apache.org/releases/javadoc/2.3.0/org/apache/beam/sdk/io/xml/XmlIO.html
+ - transform: TikaIO
+ description: Transforms for parsing arbitrary files using [Apache Tika](https://tika.apache.org/).
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.tika.TikaIO
+ url: https://beam.apache.org/releases/javadoc/2.3.0/org/apache/beam/sdk/io/tika/TikaIO.html
+ - transform: ParquetIO
+ description: IO for reading from and writing to [Parquet](https://parquet.apache.org/) files.
+ docs: /documentation/io/built-in/parquet/
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.parquet.ParquetIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/parquet/ParquetIO.html
+ - language: py
+ name: apache_beam.io.parquetio
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.parquetio.html
+ - transform: ThriftIO
+ description: PTransforms for reading and writing files containing [Thrift](https://thrift.apache.org/)-encoded data.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.thrift.ThriftIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/thrift/ThriftIO.html
+ - transform: VcfIO
+ description: A source for reading from [VCF files](https://samtools.github.io/hts-specs/VCFv4.2.pdf) (version 4.x).
+ implementations:
+ - language: py
+ name: apache_beam.io.vcfio
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.vcfio.html
+ - transform: S3IO
+ description: A source for reading from and writing to [Amazon S3](https://aws.amazon.com/s3/).
+ implementations:
+ - language: py
+ name: apache_beam.io.aws.s3io
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.aws.s3io.html
+ - transform: GcsIO
+ description: A source for reading from and writing to [Google Cloud Storage](https://cloud.google.com/storage).
+ implementations:
+ - language: py
+ name: apache_beam.io.gcp.gcsio
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.gcp.gcsio.html
+ - name: FileSystem
+ description: Beam provides a File system interface that defines APIs for writing file systems agnostic code. Several I/O connectors are implemented as a FileSystem implementation.
+ rows:
+ - transform: HadoopFileSystem
+ description: "`FileSystem` implementation for accessing [Hadoop](https://hadoop.apache.org/) Distributed File System files."
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.hdfs.HadoopFileSystemRegistrar
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/hdfs/HadoopFileSystemRegistrar.html
+ - language: py
+ name: apache_beam.io.hadoopfilesystem
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.hadoopfilesystem.html
+ - transform: GcsFileSystem
+ description: "`FileSystem` implementation for [Google Cloud Storage](https://cloud.google.com/storage)."
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.extensions.gcp.storage.GcsFileSystemRegistrar
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/extensions/gcp/storage/GcsFileSystemRegistrar.html
+ - language: py
+ name: apache_beam.io.gcp.gcsfilesystem
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.gcp.gcsfilesystem.html
+ - language: go
+ name: github.com/apache/beam/sdks/go/pkg/beam/io/filesystem/gcs
+ url: https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam/io/filesystem/gcs
+ - transform: LocalFileSystem
+ description: "`FileSystem` implementation for accessing files on disk."
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.LocalFileSystemRegistrar
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/LocalFileSystemRegistrar.html
+ - language: py
+ name: apache_beam.io.localfilesystem
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.localfilesystem.html
+ - language: go
+ name: github.com/apache/beam/sdks/go/pkg/beam/io/filesystem/local
+ url: https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam/io/filesystem/local
+ - transform: S3FileSystem
+ description: "`FileSystem` implementation for [Amazon S3](https://aws.amazon.com/s3/)."
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.aws.s3.S3FileSystemRegistrar
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/hdfs/package-summary.html
+ - transform: In-memory
+ description: "`FileSystem` implementation in memory; useful for testing."
+ implementations:
+ - language: go
+ name: github.com/apache/beam/sdks/go/pkg/beam/io/filesystem/memfs
+ url: https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam/io/filesystem/memfs
+ - name: Messaging
+ description: These I/O connectors typically involve working with unbounded sources that come from messaging sources.
+ rows:
+ - transform: KinesisIO
+ description: PTransforms for reading from and writing to [Kinesis](https://aws.amazon.com/kinesis/) streams.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.kinesis.KinesisIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/kinesis/KinesisIO.html
+ - transform: AmqpIO
+ description: AMQP 1.0 protocol using the Apache QPid Proton-J library
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.amqp.AmqpIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/amqp/AmqpIO.html
+ - transform: KafkaIO
+ description: Read and Write PTransforms for [Apache Kafka](https://kafka.apache.org/).
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.kafka.KafkaIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/kafka/KafkaIO.html
+ - language: py
+ name: apache_beam.io.external.kafka
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.external.kafka.html
+ - transform: PubSubIO
+ description: Read and Write PTransforms for [Google Cloud Pub/Sub](https://cloud.google.com/pubsub) streams.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.gcp.pubsub.PubsubIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.html
+ - language: py
+ name: apache_beam.io.gcp.pubsub
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.gcp.pubsub.html
+ - language: py
+ name: apache_beam.io.external.gcp.pubsub
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.external.gcp.pubsub.html
+ - language: go
+ name: github.com/apache/beam/sdks/go/pkg/beam/io/pubsubio
+ url: https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam/io/pubsubio
+ - transform: JmsIO
+ description: An unbounded source for [JMS](https://www.oracle.com/java/technologies/java-message-service.html) destinations (queues or topics).
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.jms.JmsIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/jms/JmsIO.html
+ - transform: MqttIO
+ description: An unbounded source for [MQTT](https://mqtt.org/) broker.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.mqtt.MqttIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/mqtt/MqttIO.html
+ - transform: RabbitMqIO
+ description: A IO to publish or consume messages with a RabbitMQ broker.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.rabbitmq.RabbitMqIO
+ url: https://github.com/apache/beam/blob/master/sdks/java/io/rabbitmq/src/main/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIO.java
Review comment:
I've opened https://github.com/apache/beam/pull/11837, as it does look like both IOs have some amount of javadocs defined in their headers.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on pull request #11802: [BEAM-9916] Update I/O documentation links and create more complete I/O matrix
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on pull request #11802:
URL: https://github.com/apache/beam/pull/11802#issuecomment-634743675
@epicfaace Thanks for working on this, now this doc part looks much better!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on a change in pull request #11802: [BEAM-9916] Update I/O documentation links and create more complete I/O matrix
Posted by GitBox <gi...@apache.org>.
pabloem commented on a change in pull request #11802:
URL: https://github.com/apache/beam/pull/11802#discussion_r430712882
##########
File path: website/www/site/data/io_matrix.yaml
##########
@@ -0,0 +1,377 @@
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+categories:
+ - name: File-based
+ description: These I/O connectors involve working with files.
+ rows:
+ - transform: FileIO
+ description: "General-purpose transforms for working with files: listing files (matching), reading and writing."
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.FileIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/FileIO.html
+ - language: py
+ name: apache_beam.io.FileIO
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.fileio.html
+ - transform: AvroIO
+ description: PTransforms for reading from and writing to [Avro](https://avro.apache.org/) files.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.AvroIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/AvroIO.html
+ - language: py
+ name: apache_beam.io.avroio
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.avroio.html
+ - language: go
+ name: github.com/apache/beam/sdks/go/pkg/beam/io/avroio
+ url: https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam/io/avroio
+ - transform: TextIO
+ description: PTransforms for reading and writing text files.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.TextIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/TextIO.html
+ - language: py
+ name: apache_beam.io.textio
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.textio.html
+ - language: go
+ name: github.com/apache/beam/sdks/go/pkg/beam/io/textio
+ url: https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam/io/textio
+ - transform: TFRecordIO
+ description: PTransforms for reading and writing [TensorFlow TFRecord](https://www.tensorflow.org/tutorials/load_data/tfrecord) files.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.TFRecordIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/TFRecordIO.html
+ - language: py
+ name: apache_beam.io.tfrecordio
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.tfrecordio.html
+ - transform: XmlIO
+ description: Transforms for reading and writing XML files using [JAXB](https://www.oracle.com/technical-resources/articles/javase/jaxb.html) mappers.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.xml.XmlIO
+ url: https://beam.apache.org/releases/javadoc/2.3.0/org/apache/beam/sdk/io/xml/XmlIO.html
+ - transform: TikaIO
+ description: Transforms for parsing arbitrary files using [Apache Tika](https://tika.apache.org/).
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.tika.TikaIO
+ url: https://beam.apache.org/releases/javadoc/2.3.0/org/apache/beam/sdk/io/tika/TikaIO.html
+ - transform: ParquetIO
+ description: IO for reading from and writing to [Parquet](https://parquet.apache.org/) files.
+ docs: /documentation/io/built-in/parquet/
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.parquet.ParquetIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/parquet/ParquetIO.html
+ - language: py
+ name: apache_beam.io.parquetio
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.parquetio.html
+ - transform: ThriftIO
+ description: PTransforms for reading and writing files containing [Thrift](https://thrift.apache.org/)-encoded data.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.thrift.ThriftIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/thrift/ThriftIO.html
+ - transform: VcfIO
+ description: A source for reading from [VCF files](https://samtools.github.io/hts-specs/VCFv4.2.pdf) (version 4.x).
+ implementations:
+ - language: py
+ name: apache_beam.io.vcfio
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.vcfio.html
+ - transform: S3IO
+ description: A source for reading from and writing to [Amazon S3](https://aws.amazon.com/s3/).
+ implementations:
+ - language: py
+ name: apache_beam.io.aws.s3io
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.aws.s3io.html
+ - transform: GcsIO
+ description: A source for reading from and writing to [Google Cloud Storage](https://cloud.google.com/storage).
+ implementations:
+ - language: py
+ name: apache_beam.io.gcp.gcsio
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.gcp.gcsio.html
+ - name: FileSystem
+ description: Beam provides a File system interface that defines APIs for writing file systems agnostic code. Several I/O connectors are implemented as a FileSystem implementation.
+ rows:
+ - transform: HadoopFileSystem
+ description: "`FileSystem` implementation for accessing [Hadoop](https://hadoop.apache.org/) Distributed File System files."
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.hdfs.HadoopFileSystemRegistrar
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/hdfs/HadoopFileSystemRegistrar.html
+ - language: py
+ name: apache_beam.io.hadoopfilesystem
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.hadoopfilesystem.html
+ - transform: GcsFileSystem
+ description: "`FileSystem` implementation for [Google Cloud Storage](https://cloud.google.com/storage)."
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.extensions.gcp.storage.GcsFileSystemRegistrar
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/extensions/gcp/storage/GcsFileSystemRegistrar.html
+ - language: py
+ name: apache_beam.io.gcp.gcsfilesystem
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.gcp.gcsfilesystem.html
+ - language: go
+ name: github.com/apache/beam/sdks/go/pkg/beam/io/filesystem/gcs
+ url: https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam/io/filesystem/gcs
+ - transform: LocalFileSystem
+ description: "`FileSystem` implementation for accessing files on disk."
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.LocalFileSystemRegistrar
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/LocalFileSystemRegistrar.html
+ - language: py
+ name: apache_beam.io.localfilesystem
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.localfilesystem.html
+ - language: go
+ name: github.com/apache/beam/sdks/go/pkg/beam/io/filesystem/local
+ url: https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam/io/filesystem/local
+ - transform: S3FileSystem
+ description: "`FileSystem` implementation for [Amazon S3](https://aws.amazon.com/s3/)."
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.aws.s3.S3FileSystemRegistrar
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/hdfs/package-summary.html
+ - transform: In-memory
+ description: "`FileSystem` implementation in memory; useful for testing."
+ implementations:
+ - language: go
+ name: github.com/apache/beam/sdks/go/pkg/beam/io/filesystem/memfs
+ url: https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam/io/filesystem/memfs
+ - name: Messaging
+ description: These I/O connectors typically involve working with unbounded sources that come from messaging sources.
+ rows:
+ - transform: KinesisIO
+ description: PTransforms for reading from and writing to [Kinesis](https://aws.amazon.com/kinesis/) streams.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.kinesis.KinesisIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/kinesis/KinesisIO.html
+ - transform: AmqpIO
+ description: AMQP 1.0 protocol using the Apache QPid Proton-J library
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.amqp.AmqpIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/amqp/AmqpIO.html
+ - transform: KafkaIO
+ description: Read and Write PTransforms for [Apache Kafka](https://kafka.apache.org/).
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.kafka.KafkaIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/kafka/KafkaIO.html
+ - language: py
+ name: apache_beam.io.external.kafka
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.external.kafka.html
+ - transform: PubSubIO
+ description: Read and Write PTransforms for [Google Cloud Pub/Sub](https://cloud.google.com/pubsub) streams.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.gcp.pubsub.PubsubIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.html
+ - language: py
+ name: apache_beam.io.gcp.pubsub
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.gcp.pubsub.html
+ - language: py
+ name: apache_beam.io.external.gcp.pubsub
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.external.gcp.pubsub.html
+ - language: go
+ name: github.com/apache/beam/sdks/go/pkg/beam/io/pubsubio
+ url: https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam/io/pubsubio
+ - transform: JmsIO
+ description: An unbounded source for [JMS](https://www.oracle.com/java/technologies/java-message-service.html) destinations (queues or topics).
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.jms.JmsIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/jms/JmsIO.html
+ - transform: MqttIO
+ description: An unbounded source for [MQTT](https://mqtt.org/) broker.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.mqtt.MqttIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/mqtt/MqttIO.html
+ - transform: RabbitMqIO
+ description: A IO to publish or consume messages with a RabbitMQ broker.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.rabbitmq.RabbitMqIO
+ url: https://github.com/apache/beam/blob/master/sdks/java/io/rabbitmq/src/main/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIO.java
Review comment:
it seems that javadoc export is deactivated for those IOs (see line 20 of `sdks/java/io/kudu/build.gradle` and `sdks/java/io/rabbitmq/build.gradle`) - @aromanenko-dev do you know if that's expected or should be fixed?
##########
File path: website/www/site/data/io_matrix.yaml
##########
@@ -0,0 +1,377 @@
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+categories:
+ - name: File-based
+ description: These I/O connectors involve working with files.
+ rows:
+ - transform: FileIO
+ description: "General-purpose transforms for working with files: listing files (matching), reading and writing."
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.FileIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/FileIO.html
+ - language: py
+ name: apache_beam.io.FileIO
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.fileio.html
+ - transform: AvroIO
+ description: PTransforms for reading from and writing to [Avro](https://avro.apache.org/) files.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.AvroIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/AvroIO.html
+ - language: py
+ name: apache_beam.io.avroio
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.avroio.html
+ - language: go
+ name: github.com/apache/beam/sdks/go/pkg/beam/io/avroio
+ url: https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam/io/avroio
+ - transform: TextIO
+ description: PTransforms for reading and writing text files.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.TextIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/TextIO.html
+ - language: py
+ name: apache_beam.io.textio
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.textio.html
+ - language: go
+ name: github.com/apache/beam/sdks/go/pkg/beam/io/textio
+ url: https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam/io/textio
+ - transform: TFRecordIO
+ description: PTransforms for reading and writing [TensorFlow TFRecord](https://www.tensorflow.org/tutorials/load_data/tfrecord) files.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.TFRecordIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/TFRecordIO.html
+ - language: py
+ name: apache_beam.io.tfrecordio
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.tfrecordio.html
+ - transform: XmlIO
+ description: Transforms for reading and writing XML files using [JAXB](https://www.oracle.com/technical-resources/articles/javase/jaxb.html) mappers.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.xml.XmlIO
+ url: https://beam.apache.org/releases/javadoc/2.3.0/org/apache/beam/sdk/io/xml/XmlIO.html
+ - transform: TikaIO
+ description: Transforms for parsing arbitrary files using [Apache Tika](https://tika.apache.org/).
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.tika.TikaIO
+ url: https://beam.apache.org/releases/javadoc/2.3.0/org/apache/beam/sdk/io/tika/TikaIO.html
+ - transform: ParquetIO
+ description: IO for reading from and writing to [Parquet](https://parquet.apache.org/) files.
+ docs: /documentation/io/built-in/parquet/
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.parquet.ParquetIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/parquet/ParquetIO.html
+ - language: py
+ name: apache_beam.io.parquetio
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.parquetio.html
+ - transform: ThriftIO
+ description: PTransforms for reading and writing files containing [Thrift](https://thrift.apache.org/)-encoded data.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.thrift.ThriftIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/thrift/ThriftIO.html
+ - transform: VcfIO
+ description: A source for reading from [VCF files](https://samtools.github.io/hts-specs/VCFv4.2.pdf) (version 4.x).
+ implementations:
+ - language: py
+ name: apache_beam.io.vcfio
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.vcfio.html
+ - transform: S3IO
+ description: A source for reading from and writing to [Amazon S3](https://aws.amazon.com/s3/).
+ implementations:
+ - language: py
+ name: apache_beam.io.aws.s3io
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.aws.s3io.html
+ - transform: GcsIO
+ description: A source for reading from and writing to [Google Cloud Storage](https://cloud.google.com/storage).
+ implementations:
+ - language: py
+ name: apache_beam.io.gcp.gcsio
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.gcp.gcsio.html
+ - name: FileSystem
+ description: Beam provides a File system interface that defines APIs for writing file systems agnostic code. Several I/O connectors are implemented as a FileSystem implementation.
+ rows:
+ - transform: HadoopFileSystem
+ description: "`FileSystem` implementation for accessing [Hadoop](https://hadoop.apache.org/) Distributed File System files."
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.hdfs.HadoopFileSystemRegistrar
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/hdfs/HadoopFileSystemRegistrar.html
+ - language: py
+ name: apache_beam.io.hadoopfilesystem
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.hadoopfilesystem.html
+ - transform: GcsFileSystem
+ description: "`FileSystem` implementation for [Google Cloud Storage](https://cloud.google.com/storage)."
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.extensions.gcp.storage.GcsFileSystemRegistrar
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/extensions/gcp/storage/GcsFileSystemRegistrar.html
+ - language: py
+ name: apache_beam.io.gcp.gcsfilesystem
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.gcp.gcsfilesystem.html
+ - language: go
+ name: github.com/apache/beam/sdks/go/pkg/beam/io/filesystem/gcs
+ url: https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam/io/filesystem/gcs
+ - transform: LocalFileSystem
+ description: "`FileSystem` implementation for accessing files on disk."
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.LocalFileSystemRegistrar
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/LocalFileSystemRegistrar.html
+ - language: py
+ name: apache_beam.io.localfilesystem
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.localfilesystem.html
+ - language: go
+ name: github.com/apache/beam/sdks/go/pkg/beam/io/filesystem/local
+ url: https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam/io/filesystem/local
+ - transform: S3FileSystem
+ description: "`FileSystem` implementation for [Amazon S3](https://aws.amazon.com/s3/)."
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.aws.s3.S3FileSystemRegistrar
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/hdfs/package-summary.html
+ - transform: In-memory
+ description: "`FileSystem` implementation in memory; useful for testing."
+ implementations:
+ - language: go
+ name: github.com/apache/beam/sdks/go/pkg/beam/io/filesystem/memfs
+ url: https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam/io/filesystem/memfs
+ - name: Messaging
+ description: These I/O connectors typically involve working with unbounded sources that come from messaging sources.
+ rows:
+ - transform: KinesisIO
+ description: PTransforms for reading from and writing to [Kinesis](https://aws.amazon.com/kinesis/) streams.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.kinesis.KinesisIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/kinesis/KinesisIO.html
+ - transform: AmqpIO
+ description: AMQP 1.0 protocol using the Apache QPid Proton-J library
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.amqp.AmqpIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/amqp/AmqpIO.html
+ - transform: KafkaIO
+ description: Read and Write PTransforms for [Apache Kafka](https://kafka.apache.org/).
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.kafka.KafkaIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/kafka/KafkaIO.html
+ - language: py
+ name: apache_beam.io.external.kafka
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.external.kafka.html
+ - transform: PubSubIO
+ description: Read and Write PTransforms for [Google Cloud Pub/Sub](https://cloud.google.com/pubsub) streams.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.gcp.pubsub.PubsubIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.html
+ - language: py
+ name: apache_beam.io.gcp.pubsub
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.gcp.pubsub.html
+ - language: py
+ name: apache_beam.io.external.gcp.pubsub
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.external.gcp.pubsub.html
+ - language: go
+ name: github.com/apache/beam/sdks/go/pkg/beam/io/pubsubio
+ url: https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam/io/pubsubio
+ - transform: JmsIO
+ description: An unbounded source for [JMS](https://www.oracle.com/java/technologies/java-message-service.html) destinations (queues or topics).
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.jms.JmsIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/jms/JmsIO.html
+ - transform: MqttIO
+ description: An unbounded source for [MQTT](https://mqtt.org/) broker.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.mqtt.MqttIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/mqtt/MqttIO.html
+ - transform: RabbitMqIO
+ description: A IO to publish or consume messages with a RabbitMQ broker.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.rabbitmq.RabbitMqIO
+ url: https://github.com/apache/beam/blob/master/sdks/java/io/rabbitmq/src/main/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIO.java
Review comment:
@epicfaace could you file JIRA issues for these?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on a change in pull request #11802: [BEAM-9916] Update I/O documentation links and create more complete I/O matrix
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on a change in pull request #11802:
URL: https://github.com/apache/beam/pull/11802#discussion_r431233203
##########
File path: website/www/site/data/io_matrix.yaml
##########
@@ -0,0 +1,377 @@
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+categories:
+ - name: File-based
+ description: These I/O connectors involve working with files.
+ rows:
+ - transform: FileIO
+ description: "General-purpose transforms for working with files: listing files (matching), reading and writing."
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.FileIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/FileIO.html
+ - language: py
+ name: apache_beam.io.FileIO
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.fileio.html
+ - transform: AvroIO
+ description: PTransforms for reading from and writing to [Avro](https://avro.apache.org/) files.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.AvroIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/AvroIO.html
+ - language: py
+ name: apache_beam.io.avroio
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.avroio.html
+ - language: go
+ name: github.com/apache/beam/sdks/go/pkg/beam/io/avroio
+ url: https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam/io/avroio
+ - transform: TextIO
+ description: PTransforms for reading and writing text files.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.TextIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/TextIO.html
+ - language: py
+ name: apache_beam.io.textio
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.textio.html
+ - language: go
+ name: github.com/apache/beam/sdks/go/pkg/beam/io/textio
+ url: https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam/io/textio
+ - transform: TFRecordIO
+ description: PTransforms for reading and writing [TensorFlow TFRecord](https://www.tensorflow.org/tutorials/load_data/tfrecord) files.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.TFRecordIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/TFRecordIO.html
+ - language: py
+ name: apache_beam.io.tfrecordio
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.tfrecordio.html
+ - transform: XmlIO
+ description: Transforms for reading and writing XML files using [JAXB](https://www.oracle.com/technical-resources/articles/javase/jaxb.html) mappers.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.xml.XmlIO
+ url: https://beam.apache.org/releases/javadoc/2.3.0/org/apache/beam/sdk/io/xml/XmlIO.html
+ - transform: TikaIO
+ description: Transforms for parsing arbitrary files using [Apache Tika](https://tika.apache.org/).
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.tika.TikaIO
+ url: https://beam.apache.org/releases/javadoc/2.3.0/org/apache/beam/sdk/io/tika/TikaIO.html
+ - transform: ParquetIO
+ description: IO for reading from and writing to [Parquet](https://parquet.apache.org/) files.
+ docs: /documentation/io/built-in/parquet/
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.parquet.ParquetIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/parquet/ParquetIO.html
+ - language: py
+ name: apache_beam.io.parquetio
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.parquetio.html
+ - transform: ThriftIO
+ description: PTransforms for reading and writing files containing [Thrift](https://thrift.apache.org/)-encoded data.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.thrift.ThriftIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/thrift/ThriftIO.html
+ - transform: VcfIO
+ description: A source for reading from [VCF files](https://samtools.github.io/hts-specs/VCFv4.2.pdf) (version 4.x).
+ implementations:
+ - language: py
+ name: apache_beam.io.vcfio
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.vcfio.html
+ - transform: S3IO
+ description: A source for reading from and writing to [Amazon S3](https://aws.amazon.com/s3/).
+ implementations:
+ - language: py
+ name: apache_beam.io.aws.s3io
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.aws.s3io.html
+ - transform: GcsIO
+ description: A source for reading from and writing to [Google Cloud Storage](https://cloud.google.com/storage).
+ implementations:
+ - language: py
+ name: apache_beam.io.gcp.gcsio
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.gcp.gcsio.html
+ - name: FileSystem
+ description: Beam provides a File system interface that defines APIs for writing file systems agnostic code. Several I/O connectors are implemented as a FileSystem implementation.
+ rows:
+ - transform: HadoopFileSystem
+ description: "`FileSystem` implementation for accessing [Hadoop](https://hadoop.apache.org/) Distributed File System files."
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.hdfs.HadoopFileSystemRegistrar
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/hdfs/HadoopFileSystemRegistrar.html
+ - language: py
+ name: apache_beam.io.hadoopfilesystem
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.hadoopfilesystem.html
+ - transform: GcsFileSystem
+ description: "`FileSystem` implementation for [Google Cloud Storage](https://cloud.google.com/storage)."
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.extensions.gcp.storage.GcsFileSystemRegistrar
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/extensions/gcp/storage/GcsFileSystemRegistrar.html
+ - language: py
+ name: apache_beam.io.gcp.gcsfilesystem
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.gcp.gcsfilesystem.html
+ - language: go
+ name: github.com/apache/beam/sdks/go/pkg/beam/io/filesystem/gcs
+ url: https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam/io/filesystem/gcs
+ - transform: LocalFileSystem
+ description: "`FileSystem` implementation for accessing files on disk."
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.LocalFileSystemRegistrar
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/LocalFileSystemRegistrar.html
+ - language: py
+ name: apache_beam.io.localfilesystem
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.localfilesystem.html
+ - language: go
+ name: github.com/apache/beam/sdks/go/pkg/beam/io/filesystem/local
+ url: https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam/io/filesystem/local
+ - transform: S3FileSystem
+ description: "`FileSystem` implementation for [Amazon S3](https://aws.amazon.com/s3/)."
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.aws.s3.S3FileSystemRegistrar
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/hdfs/package-summary.html
+ - transform: In-memory
+ description: "`FileSystem` implementation in memory; useful for testing."
+ implementations:
+ - language: go
+ name: github.com/apache/beam/sdks/go/pkg/beam/io/filesystem/memfs
+ url: https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam/io/filesystem/memfs
+ - name: Messaging
+ description: These I/O connectors typically involve working with unbounded sources that come from messaging sources.
+ rows:
+ - transform: KinesisIO
+ description: PTransforms for reading from and writing to [Kinesis](https://aws.amazon.com/kinesis/) streams.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.kinesis.KinesisIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/kinesis/KinesisIO.html
+ - transform: AmqpIO
+ description: AMQP 1.0 protocol using the Apache QPid Proton-J library
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.amqp.AmqpIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/amqp/AmqpIO.html
+ - transform: KafkaIO
+ description: Read and Write PTransforms for [Apache Kafka](https://kafka.apache.org/).
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.kafka.KafkaIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/kafka/KafkaIO.html
+ - language: py
+ name: apache_beam.io.external.kafka
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.external.kafka.html
+ - transform: PubSubIO
+ description: Read and Write PTransforms for [Google Cloud Pub/Sub](https://cloud.google.com/pubsub) streams.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.gcp.pubsub.PubsubIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.html
+ - language: py
+ name: apache_beam.io.gcp.pubsub
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.gcp.pubsub.html
+ - language: py
+ name: apache_beam.io.external.gcp.pubsub
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.external.gcp.pubsub.html
+ - language: go
+ name: github.com/apache/beam/sdks/go/pkg/beam/io/pubsubio
+ url: https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam/io/pubsubio
+ - transform: JmsIO
+ description: An unbounded source for [JMS](https://www.oracle.com/java/technologies/java-message-service.html) destinations (queues or topics).
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.jms.JmsIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/jms/JmsIO.html
+ - transform: MqttIO
+ description: An unbounded source for [MQTT](https://mqtt.org/) broker.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.mqtt.MqttIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/mqtt/MqttIO.html
+ - transform: RabbitMqIO
+ description: A IO to publish or consume messages with a RabbitMQ broker.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.rabbitmq.RabbitMqIO
+ url: https://github.com/apache/beam/blob/master/sdks/java/io/rabbitmq/src/main/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIO.java
Review comment:
@pabloem Hmm, I don't remember why it was done for only these 2 Java IOs... It was introduced by this PR - https://github.com/apache/beam/pull/7197
For now, I don't see why we can't enable it back.
Maybe @garrettjonesgoogle or @swegner, who worked on that PR, could add something?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] epicfaace commented on a change in pull request #11802: [BEAM-9916] Update I/O documentation links and create more complete I/O matrix
Posted by GitBox <gi...@apache.org>.
epicfaace commented on a change in pull request #11802:
URL: https://github.com/apache/beam/pull/11802#discussion_r430709836
##########
File path: website/www/site/data/io_matrix.yaml
##########
@@ -0,0 +1,377 @@
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+categories:
+ - name: File-based
+ description: These I/O connectors involve working with files.
+ rows:
+ - transform: FileIO
+ description: "General-purpose transforms for working with files: listing files (matching), reading and writing."
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.FileIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/FileIO.html
+ - language: py
+ name: apache_beam.io.FileIO
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.fileio.html
+ - transform: AvroIO
+ description: PTransforms for reading from and writing to [Avro](https://avro.apache.org/) files.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.AvroIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/AvroIO.html
+ - language: py
+ name: apache_beam.io.avroio
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.avroio.html
+ - language: go
+ name: github.com/apache/beam/sdks/go/pkg/beam/io/avroio
+ url: https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam/io/avroio
+ - transform: TextIO
+ description: PTransforms for reading and writing text files.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.TextIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/TextIO.html
+ - language: py
+ name: apache_beam.io.textio
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.textio.html
+ - language: go
+ name: github.com/apache/beam/sdks/go/pkg/beam/io/textio
+ url: https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam/io/textio
+ - transform: TFRecordIO
+ description: PTransforms for reading and writing [TensorFlow TFRecord](https://www.tensorflow.org/tutorials/load_data/tfrecord) files.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.TFRecordIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/TFRecordIO.html
+ - language: py
+ name: apache_beam.io.tfrecordio
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.tfrecordio.html
+ - transform: XmlIO
+ description: Transforms for reading and writing XML files using [JAXB](https://www.oracle.com/technical-resources/articles/javase/jaxb.html) mappers.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.xml.XmlIO
+ url: https://beam.apache.org/releases/javadoc/2.3.0/org/apache/beam/sdk/io/xml/XmlIO.html
+ - transform: TikaIO
+ description: Transforms for parsing arbitrary files using [Apache Tika](https://tika.apache.org/).
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.tika.TikaIO
+ url: https://beam.apache.org/releases/javadoc/2.3.0/org/apache/beam/sdk/io/tika/TikaIO.html
+ - transform: ParquetIO
+ description: IO for reading from and writing to [Parquet](https://parquet.apache.org/) files.
+ docs: /documentation/io/built-in/parquet/
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.parquet.ParquetIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/parquet/ParquetIO.html
+ - language: py
+ name: apache_beam.io.parquetio
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.parquetio.html
+ - transform: ThriftIO
+ description: PTransforms for reading and writing files containing [Thrift](https://thrift.apache.org/)-encoded data.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.thrift.ThriftIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/thrift/ThriftIO.html
+ - transform: VcfIO
+ description: A source for reading from [VCF files](https://samtools.github.io/hts-specs/VCFv4.2.pdf) (version 4.x).
+ implementations:
+ - language: py
+ name: apache_beam.io.vcfio
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.vcfio.html
+ - transform: S3IO
+ description: A source for reading from and writing to [Amazon S3](https://aws.amazon.com/s3/).
+ implementations:
+ - language: py
+ name: apache_beam.io.aws.s3io
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.aws.s3io.html
+ - transform: GcsIO
+ description: A source for reading from and writing to [Google Cloud Storage](https://cloud.google.com/storage).
+ implementations:
+ - language: py
+ name: apache_beam.io.gcp.gcsio
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.gcp.gcsio.html
+ - name: FileSystem
+ description: Beam provides a File system interface that defines APIs for writing file systems agnostic code. Several I/O connectors are implemented as a FileSystem implementation.
+ rows:
+ - transform: HadoopFileSystem
+ description: "`FileSystem` implementation for accessing [Hadoop](https://hadoop.apache.org/) Distributed File System files."
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.hdfs.HadoopFileSystemRegistrar
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/hdfs/HadoopFileSystemRegistrar.html
+ - language: py
+ name: apache_beam.io.hadoopfilesystem
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.hadoopfilesystem.html
+ - transform: GcsFileSystem
+ description: "`FileSystem` implementation for [Google Cloud Storage](https://cloud.google.com/storage)."
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.extensions.gcp.storage.GcsFileSystemRegistrar
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/extensions/gcp/storage/GcsFileSystemRegistrar.html
+ - language: py
+ name: apache_beam.io.gcp.gcsfilesystem
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.gcp.gcsfilesystem.html
+ - language: go
+ name: github.com/apache/beam/sdks/go/pkg/beam/io/filesystem/gcs
+ url: https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam/io/filesystem/gcs
+ - transform: LocalFileSystem
+ description: "`FileSystem` implementation for accessing files on disk."
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.LocalFileSystemRegistrar
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/LocalFileSystemRegistrar.html
+ - language: py
+ name: apache_beam.io.localfilesystem
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.localfilesystem.html
+ - language: go
+ name: github.com/apache/beam/sdks/go/pkg/beam/io/filesystem/local
+ url: https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam/io/filesystem/local
+ - transform: S3FileSystem
+ description: "`FileSystem` implementation for [Amazon S3](https://aws.amazon.com/s3/)."
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.aws.s3.S3FileSystemRegistrar
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/hdfs/package-summary.html
+ - transform: In-memory
+ description: "`FileSystem` implementation in memory; useful for testing."
+ implementations:
+ - language: go
+ name: github.com/apache/beam/sdks/go/pkg/beam/io/filesystem/memfs
+ url: https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam/io/filesystem/memfs
+ - name: Messaging
+ description: These I/O connectors typically involve working with unbounded sources that come from messaging sources.
+ rows:
+ - transform: KinesisIO
+ description: PTransforms for reading from and writing to [Kinesis](https://aws.amazon.com/kinesis/) streams.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.kinesis.KinesisIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/kinesis/KinesisIO.html
+ - transform: AmqpIO
+ description: AMQP 1.0 protocol using the Apache QPid Proton-J library
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.amqp.AmqpIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/amqp/AmqpIO.html
+ - transform: KafkaIO
+ description: Read and Write PTransforms for [Apache Kafka](https://kafka.apache.org/).
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.kafka.KafkaIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/kafka/KafkaIO.html
+ - language: py
+ name: apache_beam.io.external.kafka
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.external.kafka.html
+ - transform: PubSubIO
+ description: Read and Write PTransforms for [Google Cloud Pub/Sub](https://cloud.google.com/pubsub) streams.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.gcp.pubsub.PubsubIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.html
+ - language: py
+ name: apache_beam.io.gcp.pubsub
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.gcp.pubsub.html
+ - language: py
+ name: apache_beam.io.external.gcp.pubsub
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.external.gcp.pubsub.html
+ - language: go
+ name: github.com/apache/beam/sdks/go/pkg/beam/io/pubsubio
+ url: https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam/io/pubsubio
+ - transform: JmsIO
+ description: An unbounded source for [JMS](https://www.oracle.com/java/technologies/java-message-service.html) destinations (queues or topics).
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.jms.JmsIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/jms/JmsIO.html
+ - transform: MqttIO
+ description: An unbounded source for [MQTT](https://mqtt.org/) broker.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.mqtt.MqttIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/mqtt/MqttIO.html
+ - transform: RabbitMqIO
+ description: A IO to publish or consume messages with a RabbitMQ broker.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.rabbitmq.RabbitMqIO
+ url: https://github.com/apache/beam/blob/master/sdks/java/io/rabbitmq/src/main/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIO.java
Review comment:
I wasn't sure why javadoc for RabbitMqIO and KuduIO wasn't available, so I just linked to the GitHub files instead. Any idea @pabloem ?
##########
File path: website/www/site/data/io_matrix.yaml
##########
@@ -0,0 +1,377 @@
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+categories:
+ - name: File-based
+ description: These I/O connectors involve working with files.
+ rows:
+ - transform: FileIO
+ description: "General-purpose transforms for working with files: listing files (matching), reading and writing."
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.FileIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/FileIO.html
+ - language: py
+ name: apache_beam.io.FileIO
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.fileio.html
+ - transform: AvroIO
+ description: PTransforms for reading from and writing to [Avro](https://avro.apache.org/) files.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.AvroIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/AvroIO.html
+ - language: py
+ name: apache_beam.io.avroio
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.avroio.html
+ - language: go
+ name: github.com/apache/beam/sdks/go/pkg/beam/io/avroio
+ url: https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam/io/avroio
+ - transform: TextIO
+ description: PTransforms for reading and writing text files.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.TextIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/TextIO.html
+ - language: py
+ name: apache_beam.io.textio
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.textio.html
+ - language: go
+ name: github.com/apache/beam/sdks/go/pkg/beam/io/textio
+ url: https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam/io/textio
+ - transform: TFRecordIO
+ description: PTransforms for reading and writing [TensorFlow TFRecord](https://www.tensorflow.org/tutorials/load_data/tfrecord) files.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.TFRecordIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/TFRecordIO.html
+ - language: py
+ name: apache_beam.io.tfrecordio
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.tfrecordio.html
+ - transform: XmlIO
+ description: Transforms for reading and writing XML files using [JAXB](https://www.oracle.com/technical-resources/articles/javase/jaxb.html) mappers.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.xml.XmlIO
+ url: https://beam.apache.org/releases/javadoc/2.3.0/org/apache/beam/sdk/io/xml/XmlIO.html
+ - transform: TikaIO
+ description: Transforms for parsing arbitrary files using [Apache Tika](https://tika.apache.org/).
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.tika.TikaIO
+ url: https://beam.apache.org/releases/javadoc/2.3.0/org/apache/beam/sdk/io/tika/TikaIO.html
+ - transform: ParquetIO
+ description: IO for reading from and writing to [Parquet](https://parquet.apache.org/) files.
+ docs: /documentation/io/built-in/parquet/
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.parquet.ParquetIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/parquet/ParquetIO.html
+ - language: py
+ name: apache_beam.io.parquetio
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.parquetio.html
+ - transform: ThriftIO
+ description: PTransforms for reading and writing files containing [Thrift](https://thrift.apache.org/)-encoded data.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.thrift.ThriftIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/thrift/ThriftIO.html
+ - transform: VcfIO
+ description: A source for reading from [VCF files](https://samtools.github.io/hts-specs/VCFv4.2.pdf) (version 4.x).
+ implementations:
+ - language: py
+ name: apache_beam.io.vcfio
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.vcfio.html
+ - transform: S3IO
+ description: A source for reading from and writing to [Amazon S3](https://aws.amazon.com/s3/).
+ implementations:
+ - language: py
+ name: apache_beam.io.aws.s3io
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.aws.s3io.html
+ - transform: GcsIO
+ description: A source for reading from and writing to [Google Cloud Storage](https://cloud.google.com/storage).
+ implementations:
+ - language: py
+ name: apache_beam.io.gcp.gcsio
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.gcp.gcsio.html
+ - name: FileSystem
+ description: Beam provides a File system interface that defines APIs for writing file systems agnostic code. Several I/O connectors are implemented as a FileSystem implementation.
+ rows:
+ - transform: HadoopFileSystem
+ description: "`FileSystem` implementation for accessing [Hadoop](https://hadoop.apache.org/) Distributed File System files."
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.hdfs.HadoopFileSystemRegistrar
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/hdfs/HadoopFileSystemRegistrar.html
+ - language: py
+ name: apache_beam.io.hadoopfilesystem
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.hadoopfilesystem.html
+ - transform: GcsFileSystem
+ description: "`FileSystem` implementation for [Google Cloud Storage](https://cloud.google.com/storage)."
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.extensions.gcp.storage.GcsFileSystemRegistrar
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/extensions/gcp/storage/GcsFileSystemRegistrar.html
+ - language: py
+ name: apache_beam.io.gcp.gcsfilesystem
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.gcp.gcsfilesystem.html
+ - language: go
+ name: github.com/apache/beam/sdks/go/pkg/beam/io/filesystem/gcs
+ url: https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam/io/filesystem/gcs
+ - transform: LocalFileSystem
+ description: "`FileSystem` implementation for accessing files on disk."
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.LocalFileSystemRegistrar
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/LocalFileSystemRegistrar.html
+ - language: py
+ name: apache_beam.io.localfilesystem
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.localfilesystem.html
+ - language: go
+ name: github.com/apache/beam/sdks/go/pkg/beam/io/filesystem/local
+ url: https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam/io/filesystem/local
+ - transform: S3FileSystem
+ description: "`FileSystem` implementation for [Amazon S3](https://aws.amazon.com/s3/)."
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.aws.s3.S3FileSystemRegistrar
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/hdfs/package-summary.html
+ - transform: In-memory
+ description: "`FileSystem` implementation in memory; useful for testing."
+ implementations:
+ - language: go
+ name: github.com/apache/beam/sdks/go/pkg/beam/io/filesystem/memfs
+ url: https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam/io/filesystem/memfs
+ - name: Messaging
+ description: These I/O connectors typically involve working with unbounded sources that come from messaging sources.
+ rows:
+ - transform: KinesisIO
+ description: PTransforms for reading from and writing to [Kinesis](https://aws.amazon.com/kinesis/) streams.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.kinesis.KinesisIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/kinesis/KinesisIO.html
+ - transform: AmqpIO
+ description: AMQP 1.0 protocol using the Apache QPid Proton-J library
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.amqp.AmqpIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/amqp/AmqpIO.html
+ - transform: KafkaIO
+ description: Read and Write PTransforms for [Apache Kafka](https://kafka.apache.org/).
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.kafka.KafkaIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/kafka/KafkaIO.html
+ - language: py
+ name: apache_beam.io.external.kafka
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.external.kafka.html
+ - transform: PubSubIO
+ description: Read and Write PTransforms for [Google Cloud Pub/Sub](https://cloud.google.com/pubsub) streams.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.gcp.pubsub.PubsubIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.html
+ - language: py
+ name: apache_beam.io.gcp.pubsub
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.gcp.pubsub.html
+ - language: py
+ name: apache_beam.io.external.gcp.pubsub
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.external.gcp.pubsub.html
+ - language: go
+ name: github.com/apache/beam/sdks/go/pkg/beam/io/pubsubio
+ url: https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam/io/pubsubio
+ - transform: JmsIO
+ description: An unbounded source for [JMS](https://www.oracle.com/java/technologies/java-message-service.html) destinations (queues or topics).
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.jms.JmsIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/jms/JmsIO.html
+ - transform: MqttIO
+ description: An unbounded source for [MQTT](https://mqtt.org/) broker.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.mqtt.MqttIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/mqtt/MqttIO.html
+ - transform: RabbitMqIO
+ description: A IO to publish or consume messages with a RabbitMQ broker.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.rabbitmq.RabbitMqIO
+ url: https://github.com/apache/beam/blob/master/sdks/java/io/rabbitmq/src/main/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIO.java
Review comment:
https://issues.apache.org/jira/browse/BEAM-10098
##########
File path: website/www/site/data/io_matrix.yaml
##########
@@ -0,0 +1,377 @@
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+categories:
+ - name: File-based
+ description: These I/O connectors involve working with files.
+ rows:
+ - transform: FileIO
+ description: "General-purpose transforms for working with files: listing files (matching), reading and writing."
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.FileIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/FileIO.html
+ - language: py
+ name: apache_beam.io.FileIO
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.fileio.html
+ - transform: AvroIO
+ description: PTransforms for reading from and writing to [Avro](https://avro.apache.org/) files.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.AvroIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/AvroIO.html
+ - language: py
+ name: apache_beam.io.avroio
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.avroio.html
+ - language: go
+ name: github.com/apache/beam/sdks/go/pkg/beam/io/avroio
+ url: https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam/io/avroio
+ - transform: TextIO
+ description: PTransforms for reading and writing text files.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.TextIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/TextIO.html
+ - language: py
+ name: apache_beam.io.textio
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.textio.html
+ - language: go
+ name: github.com/apache/beam/sdks/go/pkg/beam/io/textio
+ url: https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam/io/textio
+ - transform: TFRecordIO
+ description: PTransforms for reading and writing [TensorFlow TFRecord](https://www.tensorflow.org/tutorials/load_data/tfrecord) files.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.TFRecordIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/TFRecordIO.html
+ - language: py
+ name: apache_beam.io.tfrecordio
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.tfrecordio.html
+ - transform: XmlIO
+ description: Transforms for reading and writing XML files using [JAXB](https://www.oracle.com/technical-resources/articles/javase/jaxb.html) mappers.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.xml.XmlIO
+ url: https://beam.apache.org/releases/javadoc/2.3.0/org/apache/beam/sdk/io/xml/XmlIO.html
+ - transform: TikaIO
+ description: Transforms for parsing arbitrary files using [Apache Tika](https://tika.apache.org/).
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.tika.TikaIO
+ url: https://beam.apache.org/releases/javadoc/2.3.0/org/apache/beam/sdk/io/tika/TikaIO.html
+ - transform: ParquetIO
+ description: IO for reading from and writing to [Parquet](https://parquet.apache.org/) files.
+ docs: /documentation/io/built-in/parquet/
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.parquet.ParquetIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/parquet/ParquetIO.html
+ - language: py
+ name: apache_beam.io.parquetio
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.parquetio.html
+ - transform: ThriftIO
+ description: PTransforms for reading and writing files containing [Thrift](https://thrift.apache.org/)-encoded data.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.thrift.ThriftIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/thrift/ThriftIO.html
+ - transform: VcfIO
+ description: A source for reading from [VCF files](https://samtools.github.io/hts-specs/VCFv4.2.pdf) (version 4.x).
+ implementations:
+ - language: py
+ name: apache_beam.io.vcfio
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.vcfio.html
+ - transform: S3IO
+ description: A source for reading from and writing to [Amazon S3](https://aws.amazon.com/s3/).
+ implementations:
+ - language: py
+ name: apache_beam.io.aws.s3io
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.aws.s3io.html
+ - transform: GcsIO
+ description: A source for reading from and writing to [Google Cloud Storage](https://cloud.google.com/storage).
+ implementations:
+ - language: py
+ name: apache_beam.io.gcp.gcsio
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.gcp.gcsio.html
+ - name: FileSystem
+ description: Beam provides a File system interface that defines APIs for writing file systems agnostic code. Several I/O connectors are implemented as a FileSystem implementation.
+ rows:
+ - transform: HadoopFileSystem
+ description: "`FileSystem` implementation for accessing [Hadoop](https://hadoop.apache.org/) Distributed File System files."
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.hdfs.HadoopFileSystemRegistrar
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/hdfs/HadoopFileSystemRegistrar.html
+ - language: py
+ name: apache_beam.io.hadoopfilesystem
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.hadoopfilesystem.html
+ - transform: GcsFileSystem
+ description: "`FileSystem` implementation for [Google Cloud Storage](https://cloud.google.com/storage)."
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.extensions.gcp.storage.GcsFileSystemRegistrar
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/extensions/gcp/storage/GcsFileSystemRegistrar.html
+ - language: py
+ name: apache_beam.io.gcp.gcsfilesystem
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.gcp.gcsfilesystem.html
+ - language: go
+ name: github.com/apache/beam/sdks/go/pkg/beam/io/filesystem/gcs
+ url: https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam/io/filesystem/gcs
+ - transform: LocalFileSystem
+ description: "`FileSystem` implementation for accessing files on disk."
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.LocalFileSystemRegistrar
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/LocalFileSystemRegistrar.html
+ - language: py
+ name: apache_beam.io.localfilesystem
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.localfilesystem.html
+ - language: go
+ name: github.com/apache/beam/sdks/go/pkg/beam/io/filesystem/local
+ url: https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam/io/filesystem/local
+ - transform: S3FileSystem
+ description: "`FileSystem` implementation for [Amazon S3](https://aws.amazon.com/s3/)."
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.aws.s3.S3FileSystemRegistrar
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/hdfs/package-summary.html
+ - transform: In-memory
+ description: "`FileSystem` implementation in memory; useful for testing."
+ implementations:
+ - language: go
+ name: github.com/apache/beam/sdks/go/pkg/beam/io/filesystem/memfs
+ url: https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam/io/filesystem/memfs
+ - name: Messaging
+ description: These I/O connectors typically involve working with unbounded sources that come from messaging sources.
+ rows:
+ - transform: KinesisIO
+ description: PTransforms for reading from and writing to [Kinesis](https://aws.amazon.com/kinesis/) streams.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.kinesis.KinesisIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/kinesis/KinesisIO.html
+ - transform: AmqpIO
+ description: AMQP 1.0 protocol using the Apache QPid Proton-J library
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.amqp.AmqpIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/amqp/AmqpIO.html
+ - transform: KafkaIO
+ description: Read and Write PTransforms for [Apache Kafka](https://kafka.apache.org/).
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.kafka.KafkaIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/kafka/KafkaIO.html
+ - language: py
+ name: apache_beam.io.external.kafka
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.external.kafka.html
+ - transform: PubSubIO
+ description: Read and Write PTransforms for [Google Cloud Pub/Sub](https://cloud.google.com/pubsub) streams.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.gcp.pubsub.PubsubIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.html
+ - language: py
+ name: apache_beam.io.gcp.pubsub
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.gcp.pubsub.html
+ - language: py
+ name: apache_beam.io.external.gcp.pubsub
+ url: https://beam.apache.org/releases/pydoc/current/apache_beam.io.external.gcp.pubsub.html
+ - language: go
+ name: github.com/apache/beam/sdks/go/pkg/beam/io/pubsubio
+ url: https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam/io/pubsubio
+ - transform: JmsIO
+ description: An unbounded source for [JMS](https://www.oracle.com/java/technologies/java-message-service.html) destinations (queues or topics).
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.jms.JmsIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/jms/JmsIO.html
+ - transform: MqttIO
+ description: An unbounded source for [MQTT](https://mqtt.org/) broker.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.mqtt.MqttIO
+ url: https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/mqtt/MqttIO.html
+ - transform: RabbitMqIO
+ description: A IO to publish or consume messages with a RabbitMQ broker.
+ implementations:
+ - language: java
+ name: org.apache.beam.sdk.io.rabbitmq.RabbitMqIO
+ url: https://github.com/apache/beam/blob/master/sdks/java/io/rabbitmq/src/main/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIO.java
Review comment:
I've also made https://issues.apache.org/jira/browse/BEAM-10099
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on pull request #11802: [BEAM-9916] Update I/O documentation links and create more complete I/O matrix
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on pull request #11802:
URL: https://github.com/apache/beam/pull/11802#issuecomment-634093933
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #11802: [BEAM-9916] Update I/O documentation links and create more complete I/O matrix
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #11802:
URL: https://github.com/apache/beam/pull/11802#issuecomment-634186277
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org