You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ka...@apache.org on 2022/08/25 05:43:54 UTC
[spark] branch master updated (c0785238047 -> 7e4064c02de)
This is an automated email from the ASF dual-hosted git repository.
kabhwan pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
from c0785238047 [SPARK-40213][SQL] Support ASCII value conversion for Latin-1 characters
add 7e4064c02de [SPARK-40039][SS] Introducing a streaming checkpoint file manager based on Hadoop's Abortable interface
No new revisions were added by this update.
Summary of changes:
docs/cloud-integration.md | 12 ++-
hadoop-cloud/README.md | 20 ++++
hadoop-cloud/pom.xml | 47 +++++++++
...AbortableStreamBasedCheckpointFileManager.scala | 95 +++++++++++++++++
...ableStreamBasedCheckpointFileManagerSuite.scala | 83 +++++++++++++++
.../internal/io/cloud/IntegrationTestSuite.java | 5 +-
.../io/cloud/abortable/AbortableFileSystem.java | 114 +++++++++++++++++++++
.../abortable/AbstractAbortableFileSystem.java | 45 +++-----
project/SparkBuild.scala | 3 +-
.../streaming/CheckpointFileManager.scala | 57 ++++++-----
.../streaming/CheckpointFileManagerSuite.scala | 94 ++++++++++++-----
11 files changed, 485 insertions(+), 90 deletions(-)
create mode 100644 hadoop-cloud/README.md
create mode 100644 hadoop-cloud/src/hadoop-3/main/scala/org/apache/spark/internal/io/cloud/AbortableStreamBasedCheckpointFileManager.scala
create mode 100644 hadoop-cloud/src/hadoop-3/test/scala/org/apache/spark/internal/io/cloud/AbortableStreamBasedCheckpointFileManagerSuite.scala
copy common/tags/src/test/java/org/apache/spark/tags/ExtendedLevelDBTest.java => hadoop-cloud/src/test/java/org/apache/spark/internal/io/cloud/IntegrationTestSuite.java (92%)
create mode 100644 hadoop-cloud/src/test/java/org/apache/spark/internal/io/cloud/abortable/AbortableFileSystem.java
copy common/network-yarn/src/main/java/org/apache/spark/network/yarn/util/HadoopConfigProvider.java => hadoop-cloud/src/test/java/org/apache/spark/internal/io/cloud/abortable/AbstractAbortableFileSystem.java (52%)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org