You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ge...@apache.org on 2024/04/26 17:23:37 UTC
(spark) branch master updated: [SPARK-47963][CORE] Make the external Spark ecosystem can use structured logging mechanisms
This is an automated email from the ASF dual-hosted git repository.
gengliang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 9cf6dc873ff3 [SPARK-47963][CORE] Make the external Spark ecosystem can use structured logging mechanisms
9cf6dc873ff3 is described below
commit 9cf6dc873ff34412df6256cdc7613eed40716570
Author: panbingkun <pa...@baidu.com>
AuthorDate: Fri Apr 26 10:23:25 2024 -0700
[SPARK-47963][CORE] Make the external Spark ecosystem can use structured logging mechanisms
### What changes were proposed in this pull request?
The pr aims to make `the external Spark ecosystem` can use `structured logging mechanisms`.
### Why are the changes needed?
Currently, the class of MDC's key is `LogKey`, while LogKey is an `enumeration`. If we want to use structured logging mechanism in a `third-party Spark ecosystem`, we cannot currently use it (we cannot expand and `add a new LogKey enumeration value`)
So I propose changing the `LogKey` from `enumeration` to the following,
<img width="550" alt="image" src="https://github.com/apache/spark/assets/15246973/ffb0f41e-4f2c-4537-b43f-d55353fe03f7">
as we already have a similar way of writing it in the Spark code, eg:
https://github.com/apache/spark/blob/6f01982094f6cb6e55a9edfbe06f9bfa9e3bce01/core/src/main/scala/org/apache/spark/TaskEndReason.scala#L39-L46
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
- Add new UT (Give an example to developers who use the LogKey mechanism in external systems related to Spark.)
- Manually test.
- Pass GA.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #46193 from panbingkun/SPARK-47963.
Lead-authored-by: panbingkun <pa...@baidu.com>
Co-authored-by: panbingkun <pb...@gmail.com>
Signed-off-by: Gengliang Wang <ge...@apache.org>
---
.../scala/org/apache/spark/internal/LogKey.scala | 875 +++++++++++----------
.../scala/org/apache/spark/internal/Logging.scala | 1 -
.../main/scala/org/apache/spark/internal/README.md | 2 +-
.../scala/org/apache/spark/util/LogKeySuite.scala | 33 +-
.../scala/org/apache/spark/util/MDCSuite.scala | 2 +-
.../apache/spark/util/PatternLoggingSuite.scala | 4 +
.../apache/spark/util/StructuredLoggingSuite.scala | 49 +-
.../org/apache/spark/sql/avro/AvroUtils.scala | 2 +-
.../sql/connect/client/GrpcRetryHandler.scala | 2 +-
.../execution/ExecuteGrpcResponseSender.scala | 2 +-
.../execution/ExecuteResponseObserver.scala | 16 +-
.../sql/connect/planner/SparkConnectPlanner.scala | 2 +-
.../planner/StreamingForeachBatchHelper.scala | 2 +-
.../planner/StreamingQueryListenerHelper.scala | 2 +-
.../sql/connect/service/LoggingInterceptor.scala | 2 +-
.../spark/sql/connect/service/SessionHolder.scala | 2 +-
.../service/SparkConnectExecutionManager.scala | 12 +-
.../sql/connect/service/SparkConnectServer.scala | 2 +-
.../sql/connect/service/SparkConnectService.scala | 2 +-
.../service/SparkConnectSessionManager.scala | 2 +-
.../service/SparkConnectStreamingQueryCache.scala | 2 +-
.../connect/ui/SparkConnectServerListener.scala | 2 +-
.../spark/sql/connect/utils/ErrorUtils.scala | 2 +-
.../sql/connect/ProtoToParsedPlanTestSuite.scala | 2 +-
.../sql/jdbc/DockerJDBCIntegrationSuite.scala | 2 +-
.../sql/kafka010/KafkaBatchPartitionReader.scala | 2 +-
.../spark/sql/kafka010/KafkaContinuousStream.scala | 2 +-
.../spark/sql/kafka010/KafkaMicroBatchStream.scala | 2 +-
.../sql/kafka010/KafkaOffsetReaderAdmin.scala | 2 +-
.../sql/kafka010/KafkaOffsetReaderConsumer.scala | 2 +-
.../apache/spark/sql/kafka010/KafkaRelation.scala | 2 +-
.../org/apache/spark/sql/kafka010/KafkaSink.scala | 2 +-
.../apache/spark/sql/kafka010/KafkaSource.scala | 2 +-
.../apache/spark/sql/kafka010/KafkaSourceRDD.scala | 2 +-
.../sql/kafka010/consumer/FetchedDataPool.scala | 2 +-
.../sql/kafka010/consumer/KafkaDataConsumer.scala | 2 +-
.../kafka010/producer/CachedKafkaProducer.scala | 2 +-
.../producer/InternalKafkaProducerPool.scala | 2 +-
.../apache/spark/sql/kafka010/KafkaTestUtils.scala | 10 +-
.../kafka010/KafkaDelegationTokenProvider.scala | 2 +-
.../streaming/kafka010/ConsumerStrategy.scala | 2 +-
.../kafka010/DirectKafkaInputDStream.scala | 2 +-
.../streaming/kafka010/KafkaDataConsumer.scala | 2 +-
.../apache/spark/streaming/kafka010/KafkaRDD.scala | 2 +-
.../spark/streaming/kafka010/KafkaUtils.scala | 2 +-
.../streaming/kinesis/KinesisBackedBlockRDD.scala | 2 +-
.../streaming/kinesis/KinesisCheckpointer.scala | 2 +-
.../spark/streaming/kinesis/KinesisReceiver.scala | 2 +-
.../streaming/kinesis/KinesisRecordProcessor.scala | 2 +-
.../spark/streaming/kinesis/KinesisTestUtils.scala | 2 +-
.../executor/profiler/ExecutorJVMProfiler.scala | 2 +-
.../executor/profiler/ExecutorProfilerPlugin.scala | 2 +-
.../scala/org/apache/spark/ContextCleaner.scala | 2 +-
.../scala/org/apache/spark/MapOutputTracker.scala | 2 +-
.../main/scala/org/apache/spark/SparkContext.scala | 10 +-
.../scala/org/apache/spark/TaskContextImpl.scala | 2 +-
.../spark/api/python/PythonGatewayServer.scala | 2 +-
.../apache/spark/api/python/PythonHadoopUtil.scala | 2 +-
.../org/apache/spark/api/r/RBackendHandler.scala | 2 +-
.../apache/spark/broadcast/TorrentBroadcast.scala | 2 +-
.../scala/org/apache/spark/deploy/Client.scala | 2 +-
.../org/apache/spark/deploy/SparkSubmit.scala | 2 +-
.../spark/deploy/StandaloneResourceUtils.scala | 2 +-
.../main/scala/org/apache/spark/deploy/Utils.scala | 2 +-
.../apache/spark/deploy/history/EventFilter.scala | 2 +-
.../spark/deploy/history/FsHistoryProvider.scala | 2 +-
.../org/apache/spark/deploy/master/Master.scala | 2 +-
.../spark/deploy/rest/RestSubmissionClient.scala | 2 +-
.../security/HBaseDelegationTokenProvider.scala | 2 +-
.../org/apache/spark/deploy/worker/Worker.scala | 2 +-
.../apache/spark/deploy/worker/WorkerWatcher.scala | 2 +-
.../apache/spark/deploy/worker/ui/LogPage.scala | 2 +-
.../executor/CoarseGrainedExecutorBackend.scala | 2 +-
.../scala/org/apache/spark/executor/Executor.scala | 2 +-
.../spark/executor/ExecutorClassLoader.scala | 6 +-
.../spark/internal/io/SparkHadoopWriter.scala | 2 +-
.../spark/mapred/SparkHadoopMapRedUtil.scala | 2 +-
.../org/apache/spark/metrics/MetricsConfig.scala | 2 +-
.../org/apache/spark/metrics/MetricsSystem.scala | 2 +-
.../main/scala/org/apache/spark/rdd/PipedRDD.scala | 2 +-
.../scala/org/apache/spark/rpc/netty/Inbox.scala | 2 +-
.../org/apache/spark/rpc/netty/MessageLoop.scala | 2 +-
.../apache/spark/scheduler/AsyncEventQueue.scala | 2 +-
.../org/apache/spark/scheduler/DAGScheduler.scala | 2 +-
.../org/apache/spark/scheduler/HealthTracker.scala | 2 +-
.../apache/spark/scheduler/LiveListenerBus.scala | 2 +-
.../apache/spark/scheduler/ReplayListenerBus.scala | 2 +-
.../spark/scheduler/SchedulableBuilder.scala | 2 +-
.../apache/spark/scheduler/TaskResultGetter.scala | 2 +-
.../apache/spark/scheduler/TaskSchedulerImpl.scala | 19 +-
.../apache/spark/scheduler/TaskSetManager.scala | 2 +-
.../cluster/CoarseGrainedSchedulerBackend.scala | 2 +-
.../cluster/StandaloneSchedulerBackend.scala | 2 +-
.../apache/spark/serializer/KryoSerializer.scala | 2 +-
.../spark/shuffle/IndexShuffleBlockResolver.scala | 2 +-
.../org/apache/spark/status/AppStatusStore.scala | 2 +-
.../org/apache/spark/storage/BlockManager.scala | 2 +-
.../spark/storage/BlockManagerDecommissioner.scala | 2 +-
.../spark/storage/BlockManagerMasterEndpoint.scala | 2 +-
.../storage/BlockManagerStorageEndpoint.scala | 2 +-
.../apache/spark/storage/DiskBlockManager.scala | 2 +-
.../spark/storage/DiskBlockObjectWriter.scala | 2 +-
.../spark/storage/PushBasedFetchHelper.scala | 2 +-
.../storage/ShuffleBlockFetcherIterator.scala | 2 +-
.../scala/org/apache/spark/ui/DriverLogPage.scala | 2 +-
.../main/scala/org/apache/spark/ui/SparkUI.scala | 2 +-
.../src/main/scala/org/apache/spark/ui/WebUI.scala | 2 +-
.../scala/org/apache/spark/util/EventLoop.scala | 2 +-
.../scala/org/apache/spark/util/ListenerBus.scala | 2 +-
.../apache/spark/util/ShutdownHookManager.scala | 2 +-
.../spark/util/SparkUncaughtExceptionHandler.scala | 2 +-
.../main/scala/org/apache/spark/util/Utils.scala | 2 +-
.../apache/spark/util/logging/FileAppender.scala | 2 +-
.../spark/util/logging/RollingFileAppender.scala | 2 +-
.../org/apache/spark/graphx/GraphLoader.scala | 2 +-
.../scala/org/apache/spark/graphx/Pregel.scala | 2 +-
.../org/apache/spark/graphx/lib/PageRank.scala | 2 +-
.../apache/spark/ml/classification/LinearSVC.scala | 2 +-
.../ml/classification/LogisticRegression.scala | 2 +-
.../org/apache/spark/ml/clustering/KMeans.scala | 2 +-
.../optim/IterativelyReweightedLeastSquares.scala | 2 +-
.../spark/ml/optim/WeightedLeastSquares.scala | 2 +-
.../org/apache/spark/ml/r/RWrapperUtils.scala | 2 +-
.../spark/ml/tree/impl/GradientBoostedTrees.scala | 2 +-
.../apache/spark/ml/tree/impl/RandomForest.scala | 2 +-
.../apache/spark/ml/tuning/CrossValidator.scala | 2 +-
.../spark/ml/tuning/TrainValidationSplit.scala | 2 +-
.../org/apache/spark/ml/util/DatasetUtils.scala | 2 +-
.../org/apache/spark/ml/util/Instrumentation.scala | 2 +-
.../scala/org/apache/spark/ml/util/ReadWrite.scala | 2 +-
.../spark/mllib/clustering/BisectingKMeans.scala | 2 +-
.../org/apache/spark/mllib/clustering/KMeans.scala | 2 +-
.../spark/mllib/clustering/LocalKMeans.scala | 2 +-
.../clustering/PowerIterationClustering.scala | 2 +-
.../spark/mllib/clustering/StreamingKMeans.scala | 2 +-
.../evaluation/BinaryClassificationMetrics.scala | 2 +-
.../org/apache/spark/mllib/feature/Word2Vec.scala | 2 +-
.../org/apache/spark/mllib/fpm/PrefixSpan.scala | 2 +-
.../regression/StreamingLinearAlgorithm.scala | 2 +-
.../apache/spark/mllib/util/DataValidators.scala | 2 +-
.../org/apache/spark/mllib/util/MLUtils.scala | 2 +-
.../spark/ml/feature/VectorIndexerSuite.scala | 2 +-
.../apache/spark/ml/recommendation/ALSSuite.scala | 2 +-
.../apache/spark/mllib/linalg/VectorsSuite.scala | 2 +-
.../mllib/tree/GradientBoostedTreesSuite.scala | 2 +-
.../apache/spark/deploy/k8s/KubernetesConf.scala | 2 +-
.../apache/spark/deploy/k8s/KubernetesUtils.scala | 2 +-
.../deploy/k8s/SparkKubernetesClientFactory.scala | 2 +-
.../k8s/features/DriverCommandFeatureStep.scala | 2 +-
.../k8s/submit/KubernetesClientApplication.scala | 2 +-
.../deploy/k8s/submit/KubernetesClientUtils.scala | 2 +-
.../k8s/submit/LoggingPodStatusWatcher.scala | 2 +-
.../cluster/k8s/ExecutorPodsAllocator.scala | 44 +-
.../cluster/k8s/ExecutorPodsLifecycleManager.scala | 2 +-
.../cluster/k8s/ExecutorPodsSnapshot.scala | 2 +-
.../scheduler/cluster/k8s/ExecutorRollPlugin.scala | 2 +-
.../cluster/k8s/KubernetesClusterManager.scala | 2 +-
.../k8s/KubernetesClusterSchedulerBackend.scala | 2 +-
...ernetesLocalDiskShuffleExecutorComponents.scala | 22 +-
.../k8s/integrationtest/DepsTestsSuite.scala | 4 +-
.../spark/deploy/yarn/ApplicationMaster.scala | 36 +-
.../org/apache/spark/deploy/yarn/Client.scala | 60 +-
.../apache/spark/deploy/yarn/ClientArguments.scala | 2 +-
.../spark/deploy/yarn/ExecutorRunnable.scala | 2 +-
.../spark/deploy/yarn/ResourceRequestHelper.scala | 2 +-
.../spark/deploy/yarn/SparkRackResolver.scala | 2 +-
.../apache/spark/deploy/yarn/YarnAllocator.scala | 106 +--
.../yarn/YarnAllocatorNodeHealthTracker.scala | 2 +-
.../cluster/YarnClientSchedulerBackend.scala | 2 +-
.../scheduler/cluster/YarnSchedulerBackend.scala | 18 +-
.../sql/catalyst/analysis/FunctionRegistry.scala | 2 +-
.../sql/catalyst/analysis/HintErrorLogger.scala | 2 +-
.../catalyst/analysis/StreamingJoinHelper.scala | 2 +-
.../analysis/UnsupportedOperationChecker.scala | 2 +-
.../spark/sql/catalyst/catalog/interface.scala | 2 +-
.../spark/sql/catalyst/csv/CSVHeaderChecker.scala | 2 +-
.../catalyst/expressions/V2ExpressionUtils.scala | 2 +-
.../expressions/codegen/CodeGenerator.scala | 2 +-
.../spark/sql/catalyst/optimizer/Optimizer.scala | 4 +-
.../ReplaceNullWithFalseInPredicate.scala | 2 +-
.../spark/sql/catalyst/optimizer/joins.scala | 2 +-
.../spark/sql/catalyst/parser/AstBuilder.scala | 2 +-
.../spark/sql/catalyst/rules/RuleExecutor.scala | 2 +-
.../spark/sql/catalyst/util/CharVarcharUtils.scala | 2 +-
.../apache/spark/sql/catalyst/util/ParseMode.scala | 2 +-
.../catalyst/util/ResolveDefaultColumnsUtil.scala | 2 +-
.../spark/sql/catalyst/util/StringUtils.scala | 2 +-
.../spark/sql/catalyst/xml/ValidatorUtil.scala | 2 +-
.../main/scala/org/apache/spark/sql/Column.scala | 2 +-
.../scala/org/apache/spark/sql/SparkSession.scala | 2 +-
.../spark/sql/api/python/PythonSQLUtils.scala | 2 +-
.../org/apache/spark/sql/api/r/SQLUtils.scala | 2 +-
.../catalyst/analysis/ResolveSessionCatalog.scala | 2 +-
.../execution/BaseScriptTransformationExec.scala | 2 +-
.../apache/spark/sql/execution/CacheManager.scala | 2 +-
.../apache/spark/sql/execution/ExistingRDD.scala | 2 +-
.../spark/sql/execution/QueryExecution.scala | 2 +-
.../spark/sql/execution/SparkStrategies.scala | 2 +-
.../sql/execution/WholeStageCodegenExec.scala | 2 +-
.../execution/adaptive/AdaptiveSparkPlanExec.scala | 2 +-
.../adaptive/InsertAdaptiveSparkPlan.scala | 2 +-
.../execution/command/AnalyzeTablesCommand.scala | 2 +-
.../spark/sql/execution/command/CommandUtils.scala | 2 +-
.../command/InsertIntoDataSourceDirCommand.scala | 2 +-
.../spark/sql/execution/command/SetCommand.scala | 2 +-
.../execution/command/createDataSourceTables.scala | 2 +-
.../apache/spark/sql/execution/command/ddl.scala | 2 +-
.../datasources/BasicWriteStatsTracker.scala | 2 +-
.../sql/execution/datasources/DataSource.scala | 2 +-
.../execution/datasources/DataSourceManager.scala | 2 +-
.../execution/datasources/FileFormatWriter.scala | 2 +-
.../sql/execution/datasources/FilePartition.scala | 2 +-
.../sql/execution/datasources/FileScanRDD.scala | 2 +-
.../execution/datasources/FileStatusCache.scala | 2 +-
.../execution/datasources/csv/CSVDataSource.scala | 2 +-
.../execution/datasources/jdbc/JDBCRelation.scala | 2 +-
.../sql/execution/datasources/jdbc/JdbcUtils.scala | 2 +-
.../datasources/json/JsonOutputWriter.scala | 2 +-
.../sql/execution/datasources/orc/OrcUtils.scala | 2 +-
.../datasources/parquet/ParquetFileFormat.scala | 2 +-
.../datasources/parquet/ParquetUtils.scala | 2 +-
.../execution/datasources/v2/CacheTableExec.scala | 2 +-
.../execution/datasources/v2/CreateIndexExec.scala | 2 +-
.../datasources/v2/CreateNamespaceExec.scala | 2 +-
.../execution/datasources/v2/CreateTableExec.scala | 2 +-
.../datasources/v2/DataSourceV2Strategy.scala | 2 +-
.../execution/datasources/v2/DropIndexExec.scala | 2 +-
.../datasources/v2/FilePartitionReader.scala | 2 +-
.../sql/execution/datasources/v2/FileScan.scala | 2 +-
.../v2/V2ScanPartitioningAndOrdering.scala | 2 +-
.../datasources/v2/WriteToDataSourceV2Exec.scala | 2 +-
.../execution/exchange/BroadcastExchangeExec.scala | 2 +-
.../ApplyInPandasWithStatePythonRunner.scala | 2 +-
.../python/AttachDistributedSequenceExec.scala | 2 +-
.../AsyncProgressTrackingMicroBatchExecution.scala | 2 +-
.../streaming/AvailableNowDataStreamWrapper.scala | 2 +-
.../streaming/CheckpointFileManager.scala | 2 +-
.../streaming/CompactibleFileStreamLog.scala | 16 +-
.../sql/execution/streaming/FileStreamSink.scala | 2 +-
.../execution/streaming/FileStreamSinkLog.scala | 2 +-
.../sql/execution/streaming/FileStreamSource.scala | 36 +-
.../sql/execution/streaming/HDFSMetadataLog.scala | 2 +-
.../execution/streaming/IncrementalExecution.scala | 2 +-
.../streaming/ManifestFileCommitProtocol.scala | 2 +-
.../execution/streaming/MetadataLogFileIndex.scala | 2 +-
.../execution/streaming/MicroBatchExecution.scala | 53 +-
.../spark/sql/execution/streaming/OffsetSeq.scala | 4 +-
.../sql/execution/streaming/ProgressReporter.scala | 18 +-
.../execution/streaming/ResolveWriteToStream.scala | 2 +-
.../sql/execution/streaming/StreamExecution.scala | 2 +-
.../sql/execution/streaming/TimerStateImpl.scala | 2 +-
.../sql/execution/streaming/TriggerExecutor.scala | 2 +-
.../sql/execution/streaming/WatermarkTracker.scala | 2 +-
.../streaming/continuous/ContinuousExecution.scala | 2 +-
.../continuous/ContinuousQueuedDataReader.scala | 2 +-
.../continuous/ContinuousTextSocketSource.scala | 2 +-
.../streaming/continuous/ContinuousWriteRDD.scala | 2 +-
.../WriteToContinuousDataSourceExec.scala | 2 +-
.../sources/RateStreamMicroBatchStream.scala | 2 +-
.../sources/TextSocketMicroBatchStream.scala | 2 +-
.../state/HDFSBackedStateStoreProvider.scala | 55 +-
.../sql/execution/streaming/state/RocksDB.scala | 62 +-
.../streaming/state/RocksDBFileManager.scala | 97 +--
.../streaming/state/RocksDBMemoryManager.scala | 2 +-
.../state/RocksDBStateStoreProvider.scala | 2 +-
.../sql/execution/streaming/state/StateStore.scala | 26 +-
.../streaming/state/StateStoreChangelog.scala | 2 +-
.../state/StreamingSessionWindowStateManager.scala | 2 +-
.../state/SymmetricHashJoinStateManager.scala | 2 +-
.../sql/execution/ui/SQLAppStatusListener.scala | 2 +-
.../apache/spark/sql/internal/SharedState.scala | 2 +-
.../apache/spark/sql/jdbc/PostgresDialect.scala | 2 +-
.../sql/streaming/StreamingQueryManager.scala | 2 +-
.../SparkExecuteStatementOperation.scala | 2 +-
.../thriftserver/SparkGetCatalogsOperation.scala | 2 +-
.../thriftserver/SparkGetColumnsOperation.scala | 2 +-
.../thriftserver/SparkGetSchemasOperation.scala | 2 +-
.../thriftserver/SparkGetTableTypesOperation.scala | 2 +-
.../thriftserver/SparkGetTablesOperation.scala | 2 +-
.../thriftserver/SparkGetTypeInfoOperation.scala | 2 +-
.../sql/hive/thriftserver/SparkOperation.scala | 2 +-
.../sql/hive/thriftserver/SparkSQLCLIDriver.scala | 2 +-
.../sql/hive/thriftserver/SparkSQLDriver.scala | 2 +-
.../ui/HiveThriftServer2Listener.scala | 2 +-
.../spark/sql/hive/HiveExternalCatalog.scala | 2 +-
.../spark/sql/hive/HiveMetastoreCatalog.scala | 2 +-
.../org/apache/spark/sql/hive/HiveUtils.scala | 14 +-
.../org/apache/spark/sql/hive/TableReader.scala | 2 +-
.../spark/sql/hive/client/HiveClientImpl.scala | 2 +-
.../apache/spark/sql/hive/client/HiveShim.scala | 2 +-
.../sql/hive/client/IsolatedClientLoader.scala | 2 +-
.../spark/sql/hive/execution/HiveFileFormat.scala | 2 +-
.../spark/sql/hive/execution/HiveTempPath.scala | 2 +-
.../spark/sql/hive/orc/OrcFileOperator.scala | 2 +-
.../security/HiveDelegationTokenProvider.scala | 2 +-
.../org/apache/spark/streaming/Checkpoint.scala | 2 +-
.../apache/spark/streaming/dstream/DStream.scala | 2 +-
.../streaming/dstream/DStreamCheckpointData.scala | 2 +-
.../spark/streaming/dstream/FileInputDStream.scala | 2 +-
.../spark/streaming/dstream/InputDStream.scala | 2 +-
.../spark/streaming/receiver/BlockGenerator.scala | 2 +-
.../streaming/receiver/ReceivedBlockHandler.scala | 2 +-
.../streaming/receiver/ReceiverSupervisor.scala | 2 +-
.../receiver/ReceiverSupervisorImpl.scala | 2 +-
.../spark/streaming/scheduler/JobGenerator.scala | 2 +-
.../streaming/scheduler/ReceivedBlockTracker.scala | 2 +-
.../streaming/scheduler/ReceiverTracker.scala | 2 +-
.../streaming/util/BatchedWriteAheadLog.scala | 2 +-
.../streaming/util/FileBasedWriteAheadLog.scala | 2 +-
.../apache/spark/streaming/MasterFailureTest.scala | 2 +-
310 files changed, 1163 insertions(+), 1109 deletions(-)
diff --git a/common/utils/src/main/scala/org/apache/spark/internal/LogKey.scala b/common/utils/src/main/scala/org/apache/spark/internal/LogKey.scala
index 6df7cb5a5867..04990ddc4c9d 100644
--- a/common/utils/src/main/scala/org/apache/spark/internal/LogKey.scala
+++ b/common/utils/src/main/scala/org/apache/spark/internal/LogKey.scala
@@ -16,444 +16,447 @@
*/
package org.apache.spark.internal
+/**
+ * All structured logging `keys` used in `MDC` must be extends `LogKey`
+ */
+trait LogKey
+
/**
* Various keys used for mapped diagnostic contexts(MDC) in logging.
* All structured logging keys should be defined here for standardization.
*/
-object LogKey extends Enumeration {
- val ACCUMULATOR_ID = Value
- val ACTUAL_NUM_FILES = Value
- val ACTUAL_PARTITION_COLUMN = Value
- val ALPHA = Value
- val ANALYSIS_ERROR = Value
- val APP_ATTEMPT_ID = Value
- val APP_DESC = Value
- val APP_ID = Value
- val APP_NAME = Value
- val APP_STATE = Value
- val ARGS = Value
- val BACKUP_FILE = Value
- val BATCH_ID = Value
- val BATCH_TIMESTAMP = Value
- val BATCH_WRITE = Value
- val BLOCK_ID = Value
- val BLOCK_MANAGER_ID = Value
- val BROADCAST_ID = Value
- val BUCKET = Value
- val BYTECODE_SIZE = Value
- val CACHED_TABLE_PARTITION_METADATA_SIZE = Value
- val CACHE_AUTO_REMOVED_SIZE = Value
- val CACHE_UNTIL_HIGHEST_CONSUMED_SIZE = Value
- val CACHE_UNTIL_LAST_PRODUCED_SIZE = Value
- val CALL_SITE_LONG_FORM = Value
- val CATALOG_NAME = Value
- val CATEGORICAL_FEATURES = Value
- val CHECKPOINT_FILE = Value
- val CHECKPOINT_LOCATION = Value
- val CHECKPOINT_PATH = Value
- val CHECKPOINT_ROOT = Value
- val CHECKPOINT_TIME = Value
- val CHECKSUM_FILE_NUM = Value
- val CHOSEN_WATERMARK = Value
- val CLASS_LOADER = Value
- val CLASS_NAME = Value
- val CLUSTER_CENTROIDS = Value
- val CLUSTER_ID = Value
- val CLUSTER_LABEL = Value
- val CLUSTER_LEVEL = Value
- val CLUSTER_WEIGHT = Value
- val CODEC_LEVEL = Value
- val CODEC_NAME = Value
- val CODEGEN_STAGE_ID = Value
- val COLUMN_DATA_TYPE_SOURCE = Value
- val COLUMN_DATA_TYPE_TARGET = Value
- val COLUMN_DEFAULT_VALUE = Value
- val COLUMN_NAME = Value
- val COMMAND = Value
- val COMMAND_OUTPUT = Value
- val COMMITTED_VERSION = Value
- val COMPACT_INTERVAL = Value
- val COMPONENT = Value
- val CONFIG = Value
- val CONFIG2 = Value
- val CONFIG3 = Value
- val CONFIG4 = Value
- val CONFIG5 = Value
- val CONSUMER = Value
- val CONTAINER = Value
- val CONTAINER_ID = Value
- val CONTAINER_STATE = Value
- val COST = Value
- val COUNT = Value
- val CROSS_VALIDATION_METRIC = Value
- val CROSS_VALIDATION_METRICS = Value
- val CSV_HEADER_COLUMN_NAME = Value
- val CSV_HEADER_COLUMN_NAMES = Value
- val CSV_HEADER_LENGTH = Value
- val CSV_SCHEMA_FIELD_NAME = Value
- val CSV_SCHEMA_FIELD_NAMES = Value
- val CSV_SOURCE = Value
- val CURRENT_BATCH_ID = Value
- val CURRENT_PATH = Value
- val DATA = Value
- val DATABASE_NAME = Value
- val DATAFRAME_CACHE_ENTRY = Value
- val DATAFRAME_ID = Value
- val DATA_FILE_NUM = Value
- val DATA_SOURCE = Value
- val DATA_SOURCES = Value
- val DATA_SOURCE_PROVIDER = Value
- val DEFAULT_COMPACT_INTERVAL = Value
- val DEFAULT_ISOLATION_LEVEL = Value
- val DEFAULT_VALUE = Value
- val DELAY = Value
- val DELEGATE = Value
- val DELTA = Value
- val DESCRIPTION = Value
- val DESIRED_PARTITIONS_SIZE = Value
- val DFS_FILE = Value
- val DIFF_DELTA = Value
- val DIVISIBLE_CLUSTER_INDICES_SIZE = Value
- val DRIVER_ID = Value
- val DROPPED_PARTITIONS = Value
- val DURATION = Value
- val EFFECTIVE_STORAGE_LEVEL = Value
- val ELAPSED_TIME = Value
- val ENCODING = Value
- val END_INDEX = Value
- val END_POINT = Value
- val END_VERSION = Value
- val ENGINE = Value
- val EPOCH = Value
- val ERROR = Value
- val ESTIMATOR_PARAMETER_MAP = Value
- val EVENT_LOOP = Value
- val EVENT_QUEUE = Value
- val EXECUTE_INFO = Value
- val EXECUTE_KEY = Value
- val EXECUTION_PLAN_LEAVES = Value
- val EXECUTOR_DESIRED_COUNT = Value
- val EXECUTOR_ENVS = Value
- val EXECUTOR_ENV_REGEX = Value
- val EXECUTOR_ID = Value
- val EXECUTOR_IDS = Value
- val EXECUTOR_LAUNCH_COMMANDS = Value
- val EXECUTOR_LAUNCH_COUNT = Value
- val EXECUTOR_RESOURCES = Value
- val EXECUTOR_STATE = Value
- val EXECUTOR_TARGET_COUNT = Value
- val EXISTING_FILE = Value
- val EXIT_CODE = Value
- val EXPECTED_NUM_FILES = Value
- val EXPECTED_PARTITION_COLUMN = Value
- val EXPIRY_TIMESTAMP = Value
- val EXPR = Value
- val EXPR_TERMS = Value
- val EXTENDED_EXPLAIN_GENERATOR = Value
- val FAILURES = Value
- val FALLBACK_VERSION = Value
- val FEATURE_COLUMN = Value
- val FEATURE_DIMENSION = Value
- val FIELD_NAME = Value
- val FILE_ABSOLUTE_PATH = Value
- val FILE_END_OFFSET = Value
- val FILE_FORMAT = Value
- val FILE_FORMAT2 = Value
- val FILE_MODIFICATION_TIME = Value
- val FILE_NAME = Value
- val FILE_START_OFFSET = Value
- val FILE_VERSION = Value
- val FINAL_PATH = Value
- val FINISH_TRIGGER_DURATION = Value
- val FROM_OFFSET = Value
- val FROM_TIME = Value
- val FUNCTION_NAME = Value
- val FUNCTION_PARAMETER = Value
- val GLOBAL_WATERMARK = Value
- val GROUP_ID = Value
- val HADOOP_VERSION = Value
- val HASH_JOIN_KEYS = Value
- val HEARTBEAT_INTERVAL = Value
- val HISTORY_DIR = Value
- val HIVE_CLIENT_VERSION = Value
- val HIVE_METASTORE_VERSION = Value
- val HIVE_OPERATION_STATE = Value
- val HIVE_OPERATION_TYPE = Value
- val HOST = Value
- val HOST_PORT = Value
- val IDENTIFIER = Value
- val INCOMPATIBLE_TYPES = Value
- val INDEX = Value
- val INDEX_FILE_NUM = Value
- val INDEX_NAME = Value
- val INFERENCE_MODE = Value
- val INITIAL_CAPACITY = Value
- val INITIAL_HEARTBEAT_INTERVAL = Value
- val INIT_MODE = Value
- val INTERVAL = Value
- val ISOLATION_LEVEL = Value
- val JOB_ID = Value
- val JOIN_CONDITION = Value
- val JOIN_CONDITION_SUB_EXPR = Value
- val K8S_CONTEXT = Value
- val KAFKA_PULLS_COUNT = Value
- val KAFKA_RECORDS_PULLED_COUNT = Value
- val KEY = Value
- val KEYTAB = Value
- val LABEL_COLUMN = Value
- val LARGEST_CLUSTER_INDEX = Value
- val LAST_ACCESS_TIME = Value
- val LAST_VALID_TIME = Value
- val LATEST_BATCH_ID = Value
- val LATEST_COMMITTED_BATCH_ID = Value
- val LEARNING_RATE = Value
- val LEFT_EXPR = Value
- val LINE = Value
- val LINE_NUM = Value
- val LISTENER = Value
- val LOADED_VERSION = Value
- val LOAD_FACTOR = Value
- val LOAD_TIME = Value
- val LOGICAL_PLAN_COLUMNS = Value
- val LOGICAL_PLAN_LEAVES = Value
- val LOG_ID = Value
- val LOG_OFFSET = Value
- val LOG_TYPE = Value
- val LOWER_BOUND = Value
- val MALFORMATTED_STIRNG = Value
- val MASTER_URL = Value
- val MAX_ATTEMPTS = Value
- val MAX_CACHE_UNTIL_HIGHEST_CONSUMED_SIZE = Value
- val MAX_CACHE_UNTIL_LAST_PRODUCED_SIZE = Value
- val MAX_CAPACITY = Value
- val MAX_CATEGORIES = Value
- val MAX_EXECUTOR_FAILURES = Value
- val MAX_FILE_VERSION = Value
- val MAX_MEMORY_SIZE = Value
- val MAX_PARTITIONS_SIZE = Value
- val MAX_SIZE = Value
- val MAX_TABLE_PARTITION_METADATA_SIZE = Value
- val MEMORY_SIZE = Value
- val MERGE_DIR_NAME = Value
- val MESSAGE = Value
- val METADATA_DIRECTORY = Value
- val METADATA_JSON = Value
- val METHOD_NAME = Value
- val METRICS_JSON = Value
- val MIN_COMPACTION_BATCH_ID = Value
- val MIN_FREQUENT_PATTERN_COUNT = Value
- val MIN_POINT_PER_CLUSTER = Value
- val MIN_SIZE = Value
- val MIN_VERSION_NUMBER = Value
- val MODEL_WEIGHTS = Value
- val NAMESPACE = Value
- val NEW_FEATURE_COLUMN_NAME = Value
- val NEW_LABEL_COLUMN_NAME = Value
- val NEW_PATH = Value
- val NEW_VALUE = Value
- val NODES = Value
- val NODE_LOCATION = Value
- val NORM = Value
- val NUM_BIN = Value
- val NUM_BYTES = Value
- val NUM_CLASSES = Value
- val NUM_COLUMNS = Value
- val NUM_EXAMPLES = Value
- val NUM_FEATURES = Value
- val NUM_FILES = Value
- val NUM_FILES_COPIED = Value
- val NUM_FILES_FAILED_TO_DELETE = Value
- val NUM_FILES_REUSED = Value
- val NUM_FREQUENT_ITEMS = Value
- val NUM_ITERATIONS = Value
- val NUM_LOCAL_FREQUENT_PATTERN = Value
- val NUM_PARTITION = Value
- val NUM_POINT = Value
- val NUM_PREFIXES = Value
- val NUM_SEQUENCES = Value
- val OBJECT_ID = Value
- val OFFSET = Value
- val OFFSETS = Value
- val OFFSET_SEQUENCE_METADATA = Value
- val OLD_BLOCK_MANAGER_ID = Value
- val OLD_VALUE = Value
- val OPTIMIZED_PLAN_COLUMNS = Value
- val OPTIMIZER_CLASS_NAME = Value
- val OPTIONS = Value
- val OP_ID = Value
- val OP_TYPE = Value
- val OVERHEAD_MEMORY_SIZE = Value
- val PARSE_MODE = Value
- val PARTITIONED_FILE_READER = Value
- val PARTITIONS_SIZE = Value
- val PARTITION_ID = Value
- val PARTITION_SPECIFICATION = Value
- val PARTITION_SPECS = Value
- val PATH = Value
- val PATHS = Value
- val PIPELINE_STAGE_UID = Value
- val POD_COUNT = Value
- val POD_ID = Value
- val POD_NAME = Value
- val POD_NAMESPACE = Value
- val POD_PHASE = Value
- val POD_SHARED_SLOT_COUNT = Value
- val POD_STATE = Value
- val POD_TARGET_COUNT = Value
- val POLICY = Value
- val PORT = Value
- val PRETTY_ID_STRING = Value
- val PRINCIPAL = Value
- val PROCESSING_TIME = Value
- val PRODUCER_ID = Value
- val PROVIDER = Value
- val PVC_METADATA_NAME = Value
- val QUERY_CACHE_VALUE = Value
- val QUERY_HINT = Value
- val QUERY_ID = Value
- val QUERY_PLAN = Value
- val QUERY_PLAN_COMPARISON = Value
- val QUERY_PLAN_LENGTH_ACTUAL = Value
- val QUERY_PLAN_LENGTH_MAX = Value
- val QUERY_RUN_ID = Value
- val RANGE = Value
- val RDD_ID = Value
- val READ_LIMIT = Value
- val REASON = Value
- val REATTACHABLE = Value
- val RECEIVED_BLOCK_INFO = Value
- val RECEIVED_BLOCK_TRACKER_LOG_EVENT = Value
- val RECEIVER_ID = Value
- val RECEIVER_IDS = Value
- val RECORDS = Value
- val REDACTED_STATEMENT = Value
- val REDUCE_ID = Value
- val RELATION_NAME = Value
- val RELATIVE_TOLERANCE = Value
- val REMAINING_PARTITIONS = Value
- val REPORT_DETAILS = Value
- val RESOURCE = Value
- val RESOURCE_NAME = Value
- val RESOURCE_PROFILE_ID = Value
- val RESOURCE_PROFILE_IDS = Value
- val RETRY_COUNT = Value
- val RETRY_INTERVAL = Value
- val RIGHT_EXPR = Value
- val RMSE = Value
- val ROCKS_DB_LOG_LEVEL = Value
- val ROCKS_DB_LOG_MESSAGE = Value
- val RPC_ENDPOINT_REF = Value
- val RULE_BATCH_NAME = Value
- val RULE_NAME = Value
- val RULE_NUMBER_OF_RUNS = Value
- val RUN_ID = Value
- val SCHEMA = Value
- val SCHEMA2 = Value
- val SERVICE_NAME = Value
- val SESSION_HOLD_INFO = Value
- val SESSION_ID = Value
- val SESSION_KEY = Value
- val SHARD_ID = Value
- val SHUFFLE_BLOCK_INFO = Value
- val SHUFFLE_ID = Value
- val SHUFFLE_MERGE_ID = Value
- val SHUFFLE_SERVICE_NAME = Value
- val SIZE = Value
- val SLEEP_TIME = Value
- val SLIDE_DURATION = Value
- val SMALLEST_CLUSTER_INDEX = Value
- val SNAPSHOT_VERSION = Value
- val SPARK_DATA_STREAM = Value
- val SPARK_PLAN_ID = Value
- val SQL_TEXT = Value
- val SRC_PATH = Value
- val STAGE_ATTEMPT = Value
- val STAGE_ID = Value
- val START_INDEX = Value
- val STATEMENT_ID = Value
- val STATE_STORE_ID = Value
- val STATE_STORE_PROVIDER = Value
- val STATE_STORE_VERSION = Value
- val STATUS = Value
- val STDERR = Value
- val STORAGE_LEVEL = Value
- val STORAGE_LEVEL_DESERIALIZED = Value
- val STORAGE_LEVEL_REPLICATION = Value
- val STORE_ID = Value
- val STREAMING_DATA_SOURCE_DESCRIPTION = Value
- val STREAMING_DATA_SOURCE_NAME = Value
- val STREAMING_OFFSETS_END = Value
- val STREAMING_OFFSETS_START = Value
- val STREAMING_QUERY_PROGRESS = Value
- val STREAMING_SOURCE = Value
- val STREAMING_TABLE = Value
- val STREAMING_WRITE = Value
- val STREAM_ID = Value
- val STREAM_NAME = Value
- val SUBMISSION_ID = Value
- val SUBSAMPLING_RATE = Value
- val SUB_QUERY = Value
- val TABLE_NAME = Value
- val TABLE_TYPES = Value
- val TARGET_PATH = Value
- val TASK_ATTEMPT_ID = Value
- val TASK_ID = Value
- val TASK_NAME = Value
- val TASK_SET_NAME = Value
- val TASK_STATE = Value
- val TEMP_FILE = Value
- val TEMP_PATH = Value
- val TEST_SIZE = Value
- val THREAD = Value
- val THREAD_NAME = Value
- val TID = Value
- val TIME = Value
- val TIMEOUT = Value
- val TIMER = Value
- val TIMER_LABEL = Value
- val TIME_UNITS = Value
- val TIP = Value
- val TOKEN_REGEX = Value
- val TOPIC = Value
- val TOPIC_PARTITION = Value
- val TOPIC_PARTITIONS = Value
- val TOPIC_PARTITION_OFFSET = Value
- val TOPIC_PARTITION_OFFSET_RANGE = Value
- val TOTAL = Value
- val TOTAL_EFFECTIVE_TIME = Value
- val TOTAL_RECORDS_READ = Value
- val TOTAL_TIME = Value
- val TOTAL_TIME_READ = Value
- val TO_TIME = Value
- val TRAINING_SIZE = Value
- val TRAIN_VALIDATION_SPLIT_METRIC = Value
- val TRAIN_VALIDATION_SPLIT_METRICS = Value
- val TRAIN_WORD_COUNT = Value
- val TREE_NODE = Value
- val TRIGGER_INTERVAL = Value
- val UI_FILTER = Value
- val UI_FILTER_PARAMS = Value
- val UI_PROXY_BASE = Value
- val UNSUPPORTED_EXPR = Value
- val UNSUPPORTED_HINT_REASON = Value
- val UNTIL_OFFSET = Value
- val UPPER_BOUND = Value
- val URI = Value
- val USER_ID = Value
- val USER_NAME = Value
- val VALUE = Value
- val VERSION_NUMBER = Value
- val VIRTUAL_CORES = Value
- val VOCAB_SIZE = Value
- val WAIT_RESULT_TIME = Value
- val WAIT_SEND_TIME = Value
- val WAIT_TIME = Value
- val WATERMARK_CONSTRAINT = Value
- val WEIGHTED_NUM = Value
- val WORKER_URL = Value
- val WRITE_AHEAD_LOG_INFO = Value
- val WRITE_JOB_UUID = Value
- val XSD_PATH = Value
-
- type LogKey = Value
+object LogKeys {
+ case object ACCUMULATOR_ID extends LogKey
+ case object ACTUAL_NUM_FILES extends LogKey
+ case object ACTUAL_PARTITION_COLUMN extends LogKey
+ case object ALPHA extends LogKey
+ case object ANALYSIS_ERROR extends LogKey
+ case object APP_ATTEMPT_ID extends LogKey
+ case object APP_DESC extends LogKey
+ case object APP_ID extends LogKey
+ case object APP_NAME extends LogKey
+ case object APP_STATE extends LogKey
+ case object ARGS extends LogKey
+ case object BACKUP_FILE extends LogKey
+ case object BATCH_ID extends LogKey
+ case object BATCH_TIMESTAMP extends LogKey
+ case object BATCH_WRITE extends LogKey
+ case object BLOCK_ID extends LogKey
+ case object BLOCK_MANAGER_ID extends LogKey
+ case object BROADCAST_ID extends LogKey
+ case object BUCKET extends LogKey
+ case object BYTECODE_SIZE extends LogKey
+ case object CACHED_TABLE_PARTITION_METADATA_SIZE extends LogKey
+ case object CACHE_AUTO_REMOVED_SIZE extends LogKey
+ case object CACHE_UNTIL_HIGHEST_CONSUMED_SIZE extends LogKey
+ case object CACHE_UNTIL_LAST_PRODUCED_SIZE extends LogKey
+ case object CALL_SITE_LONG_FORM extends LogKey
+ case object CATALOG_NAME extends LogKey
+ case object CATEGORICAL_FEATURES extends LogKey
+ case object CHECKPOINT_FILE extends LogKey
+ case object CHECKPOINT_LOCATION extends LogKey
+ case object CHECKPOINT_PATH extends LogKey
+ case object CHECKPOINT_ROOT extends LogKey
+ case object CHECKPOINT_TIME extends LogKey
+ case object CHECKSUM_FILE_NUM extends LogKey
+ case object CHOSEN_WATERMARK extends LogKey
+ case object CLASS_LOADER extends LogKey
+ case object CLASS_NAME extends LogKey
+ case object CLUSTER_CENTROIDS extends LogKey
+ case object CLUSTER_ID extends LogKey
+ case object CLUSTER_LABEL extends LogKey
+ case object CLUSTER_LEVEL extends LogKey
+ case object CLUSTER_WEIGHT extends LogKey
+ case object CODEC_LEVEL extends LogKey
+ case object CODEC_NAME extends LogKey
+ case object CODEGEN_STAGE_ID extends LogKey
+ case object COLUMN_DATA_TYPE_SOURCE extends LogKey
+ case object COLUMN_DATA_TYPE_TARGET extends LogKey
+ case object COLUMN_DEFAULT_VALUE extends LogKey
+ case object COLUMN_NAME extends LogKey
+ case object COMMAND extends LogKey
+ case object COMMAND_OUTPUT extends LogKey
+ case object COMMITTED_VERSION extends LogKey
+ case object COMPACT_INTERVAL extends LogKey
+ case object COMPONENT extends LogKey
+ case object CONFIG extends LogKey
+ case object CONFIG2 extends LogKey
+ case object CONFIG3 extends LogKey
+ case object CONFIG4 extends LogKey
+ case object CONFIG5 extends LogKey
+ case object CONSUMER extends LogKey
+ case object CONTAINER extends LogKey
+ case object CONTAINER_ID extends LogKey
+ case object CONTAINER_STATE extends LogKey
+ case object COST extends LogKey
+ case object COUNT extends LogKey
+ case object CROSS_VALIDATION_METRIC extends LogKey
+ case object CROSS_VALIDATION_METRICS extends LogKey
+ case object CSV_HEADER_COLUMN_NAME extends LogKey
+ case object CSV_HEADER_COLUMN_NAMES extends LogKey
+ case object CSV_HEADER_LENGTH extends LogKey
+ case object CSV_SCHEMA_FIELD_NAME extends LogKey
+ case object CSV_SCHEMA_FIELD_NAMES extends LogKey
+ case object CSV_SOURCE extends LogKey
+ case object CURRENT_BATCH_ID extends LogKey
+ case object CURRENT_PATH extends LogKey
+ case object DATA extends LogKey
+ case object DATABASE_NAME extends LogKey
+ case object DATAFRAME_CACHE_ENTRY extends LogKey
+ case object DATAFRAME_ID extends LogKey
+ case object DATA_FILE_NUM extends LogKey
+ case object DATA_SOURCE extends LogKey
+ case object DATA_SOURCES extends LogKey
+ case object DATA_SOURCE_PROVIDER extends LogKey
+ case object DEFAULT_COMPACT_INTERVAL extends LogKey
+ case object DEFAULT_ISOLATION_LEVEL extends LogKey
+ case object DEFAULT_VALUE extends LogKey
+ case object DELAY extends LogKey
+ case object DELEGATE extends LogKey
+ case object DELTA extends LogKey
+ case object DESCRIPTION extends LogKey
+ case object DESIRED_PARTITIONS_SIZE extends LogKey
+ case object DFS_FILE extends LogKey
+ case object DIFF_DELTA extends LogKey
+ case object DIVISIBLE_CLUSTER_INDICES_SIZE extends LogKey
+ case object DRIVER_ID extends LogKey
+ case object DROPPED_PARTITIONS extends LogKey
+ case object DURATION extends LogKey
+ case object EFFECTIVE_STORAGE_LEVEL extends LogKey
+ case object ELAPSED_TIME extends LogKey
+ case object ENCODING extends LogKey
+ case object END_INDEX extends LogKey
+ case object END_POINT extends LogKey
+ case object END_VERSION extends LogKey
+ case object ENGINE extends LogKey
+ case object EPOCH extends LogKey
+ case object ERROR extends LogKey
+ case object ESTIMATOR_PARAMETER_MAP extends LogKey
+ case object EVENT_LOOP extends LogKey
+ case object EVENT_QUEUE extends LogKey
+ case object EXCEPTION extends LogKey
+ case object EXECUTE_INFO extends LogKey
+ case object EXECUTE_KEY extends LogKey
+ case object EXECUTION_PLAN_LEAVES extends LogKey
+ case object EXECUTOR_DESIRED_COUNT extends LogKey
+ case object EXECUTOR_ENVS extends LogKey
+ case object EXECUTOR_ENV_REGEX extends LogKey
+ case object EXECUTOR_ID extends LogKey
+ case object EXECUTOR_IDS extends LogKey
+ case object EXECUTOR_LAUNCH_COMMANDS extends LogKey
+ case object EXECUTOR_LAUNCH_COUNT extends LogKey
+ case object EXECUTOR_RESOURCES extends LogKey
+ case object EXECUTOR_STATE extends LogKey
+ case object EXECUTOR_TARGET_COUNT extends LogKey
+ case object EXISTING_FILE extends LogKey
+ case object EXIT_CODE extends LogKey
+ case object EXPECTED_NUM_FILES extends LogKey
+ case object EXPECTED_PARTITION_COLUMN extends LogKey
+ case object EXPIRY_TIMESTAMP extends LogKey
+ case object EXPR extends LogKey
+ case object EXPR_TERMS extends LogKey
+ case object EXTENDED_EXPLAIN_GENERATOR extends LogKey
+ case object FAILURES extends LogKey
+ case object FALLBACK_VERSION extends LogKey
+ case object FEATURE_COLUMN extends LogKey
+ case object FEATURE_DIMENSION extends LogKey
+ case object FIELD_NAME extends LogKey
+ case object FILE_ABSOLUTE_PATH extends LogKey
+ case object FILE_END_OFFSET extends LogKey
+ case object FILE_FORMAT extends LogKey
+ case object FILE_FORMAT2 extends LogKey
+ case object FILE_MODIFICATION_TIME extends LogKey
+ case object FILE_NAME extends LogKey
+ case object FILE_START_OFFSET extends LogKey
+ case object FILE_VERSION extends LogKey
+ case object FINAL_PATH extends LogKey
+ case object FINISH_TRIGGER_DURATION extends LogKey
+ case object FROM_OFFSET extends LogKey
+ case object FROM_TIME extends LogKey
+ case object FUNCTION_NAME extends LogKey
+ case object FUNCTION_PARAMETER extends LogKey
+ case object GLOBAL_WATERMARK extends LogKey
+ case object GROUP_ID extends LogKey
+ case object HADOOP_VERSION extends LogKey
+ case object HASH_JOIN_KEYS extends LogKey
+ case object HEARTBEAT_INTERVAL extends LogKey
+ case object HISTORY_DIR extends LogKey
+ case object HIVE_CLIENT_VERSION extends LogKey
+ case object HIVE_METASTORE_VERSION extends LogKey
+ case object HIVE_OPERATION_STATE extends LogKey
+ case object HIVE_OPERATION_TYPE extends LogKey
+ case object HOST extends LogKey
+ case object HOST_PORT extends LogKey
+ case object IDENTIFIER extends LogKey
+ case object INCOMPATIBLE_TYPES extends LogKey
+ case object INDEX extends LogKey
+ case object INDEX_FILE_NUM extends LogKey
+ case object INDEX_NAME extends LogKey
+ case object INFERENCE_MODE extends LogKey
+ case object INITIAL_CAPACITY extends LogKey
+ case object INITIAL_HEARTBEAT_INTERVAL extends LogKey
+ case object INIT_MODE extends LogKey
+ case object INTERVAL extends LogKey
+ case object ISOLATION_LEVEL extends LogKey
+ case object JOB_ID extends LogKey
+ case object JOIN_CONDITION extends LogKey
+ case object JOIN_CONDITION_SUB_EXPR extends LogKey
+ case object K8S_CONTEXT extends LogKey
+ case object KAFKA_PULLS_COUNT extends LogKey
+ case object KAFKA_RECORDS_PULLED_COUNT extends LogKey
+ case object KEY extends LogKey
+ case object KEYTAB extends LogKey
+ case object LABEL_COLUMN extends LogKey
+ case object LARGEST_CLUSTER_INDEX extends LogKey
+ case object LAST_ACCESS_TIME extends LogKey
+ case object LAST_VALID_TIME extends LogKey
+ case object LATEST_BATCH_ID extends LogKey
+ case object LATEST_COMMITTED_BATCH_ID extends LogKey
+ case object LEARNING_RATE extends LogKey
+ case object LEFT_EXPR extends LogKey
+ case object LINE extends LogKey
+ case object LINE_NUM extends LogKey
+ case object LISTENER extends LogKey
+ case object LOADED_VERSION extends LogKey
+ case object LOAD_FACTOR extends LogKey
+ case object LOAD_TIME extends LogKey
+ case object LOGICAL_PLAN_COLUMNS extends LogKey
+ case object LOGICAL_PLAN_LEAVES extends LogKey
+ case object LOG_ID extends LogKey
+ case object LOG_OFFSET extends LogKey
+ case object LOG_TYPE extends LogKey
+ case object LOWER_BOUND extends LogKey
+ case object MALFORMATTED_STIRNG extends LogKey
+ case object MASTER_URL extends LogKey
+ case object MAX_ATTEMPTS extends LogKey
+ case object MAX_CACHE_UNTIL_HIGHEST_CONSUMED_SIZE extends LogKey
+ case object MAX_CACHE_UNTIL_LAST_PRODUCED_SIZE extends LogKey
+ case object MAX_CAPACITY extends LogKey
+ case object MAX_CATEGORIES extends LogKey
+ case object MAX_EXECUTOR_FAILURES extends LogKey
+ case object MAX_FILE_VERSION extends LogKey
+ case object MAX_MEMORY_SIZE extends LogKey
+ case object MAX_PARTITIONS_SIZE extends LogKey
+ case object MAX_SIZE extends LogKey
+ case object MAX_TABLE_PARTITION_METADATA_SIZE extends LogKey
+ case object MEMORY_SIZE extends LogKey
+ case object MERGE_DIR_NAME extends LogKey
+ case object MESSAGE extends LogKey
+ case object METADATA_DIRECTORY extends LogKey
+ case object METADATA_JSON extends LogKey
+ case object METHOD_NAME extends LogKey
+ case object METRICS_JSON extends LogKey
+ case object MIN_COMPACTION_BATCH_ID extends LogKey
+ case object MIN_FREQUENT_PATTERN_COUNT extends LogKey
+ case object MIN_POINT_PER_CLUSTER extends LogKey
+ case object MIN_SIZE extends LogKey
+ case object MIN_VERSION_NUMBER extends LogKey
+ case object MODEL_WEIGHTS extends LogKey
+ case object NAMESPACE extends LogKey
+ case object NEW_FEATURE_COLUMN_NAME extends LogKey
+ case object NEW_LABEL_COLUMN_NAME extends LogKey
+ case object NEW_PATH extends LogKey
+ case object NEW_VALUE extends LogKey
+ case object NODES extends LogKey
+ case object NODE_LOCATION extends LogKey
+ case object NORM extends LogKey
+ case object NUM_BIN extends LogKey
+ case object NUM_BYTES extends LogKey
+ case object NUM_CLASSES extends LogKey
+ case object NUM_COLUMNS extends LogKey
+ case object NUM_EXAMPLES extends LogKey
+ case object NUM_FEATURES extends LogKey
+ case object NUM_FILES extends LogKey
+ case object NUM_FILES_COPIED extends LogKey
+ case object NUM_FILES_FAILED_TO_DELETE extends LogKey
+ case object NUM_FILES_REUSED extends LogKey
+ case object NUM_FREQUENT_ITEMS extends LogKey
+ case object NUM_ITERATIONS extends LogKey
+ case object NUM_LOCAL_FREQUENT_PATTERN extends LogKey
+ case object NUM_PARTITION extends LogKey
+ case object NUM_POINT extends LogKey
+ case object NUM_PREFIXES extends LogKey
+ case object NUM_SEQUENCES extends LogKey
+ case object OBJECT_ID extends LogKey
+ case object OFFSET extends LogKey
+ case object OFFSETS extends LogKey
+ case object OFFSET_SEQUENCE_METADATA extends LogKey
+ case object OLD_BLOCK_MANAGER_ID extends LogKey
+ case object OLD_VALUE extends LogKey
+ case object OPTIMIZED_PLAN_COLUMNS extends LogKey
+ case object OPTIMIZER_CLASS_NAME extends LogKey
+ case object OPTIONS extends LogKey
+ case object OP_ID extends LogKey
+ case object OP_TYPE extends LogKey
+ case object OVERHEAD_MEMORY_SIZE extends LogKey
+ case object PARSE_MODE extends LogKey
+ case object PARTITIONED_FILE_READER extends LogKey
+ case object PARTITIONS_SIZE extends LogKey
+ case object PARTITION_ID extends LogKey
+ case object PARTITION_SPECIFICATION extends LogKey
+ case object PARTITION_SPECS extends LogKey
+ case object PATH extends LogKey
+ case object PATHS extends LogKey
+ case object PIPELINE_STAGE_UID extends LogKey
+ case object POD_COUNT extends LogKey
+ case object POD_ID extends LogKey
+ case object POD_NAME extends LogKey
+ case object POD_NAMESPACE extends LogKey
+ case object POD_PHASE extends LogKey
+ case object POD_SHARED_SLOT_COUNT extends LogKey
+ case object POD_STATE extends LogKey
+ case object POD_TARGET_COUNT extends LogKey
+ case object POLICY extends LogKey
+ case object PORT extends LogKey
+ case object PRETTY_ID_STRING extends LogKey
+ case object PRINCIPAL extends LogKey
+ case object PROCESSING_TIME extends LogKey
+ case object PRODUCER_ID extends LogKey
+ case object PROVIDER extends LogKey
+ case object PVC_METADATA_NAME extends LogKey
+ case object QUERY_CACHE_VALUE extends LogKey
+ case object QUERY_HINT extends LogKey
+ case object QUERY_ID extends LogKey
+ case object QUERY_PLAN extends LogKey
+ case object QUERY_PLAN_COMPARISON extends LogKey
+ case object QUERY_PLAN_LENGTH_ACTUAL extends LogKey
+ case object QUERY_PLAN_LENGTH_MAX extends LogKey
+ case object QUERY_RUN_ID extends LogKey
+ case object RANGE extends LogKey
+ case object RDD_ID extends LogKey
+ case object READ_LIMIT extends LogKey
+ case object REASON extends LogKey
+ case object REATTACHABLE extends LogKey
+ case object RECEIVED_BLOCK_INFO extends LogKey
+ case object RECEIVED_BLOCK_TRACKER_LOG_EVENT extends LogKey
+ case object RECEIVER_ID extends LogKey
+ case object RECEIVER_IDS extends LogKey
+ case object RECORDS extends LogKey
+ case object REDACTED_STATEMENT extends LogKey
+ case object REDUCE_ID extends LogKey
+ case object RELATION_NAME extends LogKey
+ case object RELATIVE_TOLERANCE extends LogKey
+ case object REMAINING_PARTITIONS extends LogKey
+ case object REPORT_DETAILS extends LogKey
+ case object RESOURCE extends LogKey
+ case object RESOURCE_NAME extends LogKey
+ case object RESOURCE_PROFILE_ID extends LogKey
+ case object RESOURCE_PROFILE_IDS extends LogKey
+ case object RETRY_COUNT extends LogKey
+ case object RETRY_INTERVAL extends LogKey
+ case object RIGHT_EXPR extends LogKey
+ case object RMSE extends LogKey
+ case object ROCKS_DB_LOG_LEVEL extends LogKey
+ case object ROCKS_DB_LOG_MESSAGE extends LogKey
+ case object RPC_ENDPOINT_REF extends LogKey
+ case object RULE_BATCH_NAME extends LogKey
+ case object RULE_NAME extends LogKey
+ case object RULE_NUMBER_OF_RUNS extends LogKey
+ case object RUN_ID extends LogKey
+ case object SCHEMA extends LogKey
+ case object SCHEMA2 extends LogKey
+ case object SERVICE_NAME extends LogKey
+ case object SESSION_HOLD_INFO extends LogKey
+ case object SESSION_ID extends LogKey
+ case object SESSION_KEY extends LogKey
+ case object SHARD_ID extends LogKey
+ case object SHUFFLE_BLOCK_INFO extends LogKey
+ case object SHUFFLE_ID extends LogKey
+ case object SHUFFLE_MERGE_ID extends LogKey
+ case object SHUFFLE_SERVICE_NAME extends LogKey
+ case object SIZE extends LogKey
+ case object SLEEP_TIME extends LogKey
+ case object SLIDE_DURATION extends LogKey
+ case object SMALLEST_CLUSTER_INDEX extends LogKey
+ case object SNAPSHOT_VERSION extends LogKey
+ case object SPARK_DATA_STREAM extends LogKey
+ case object SPARK_PLAN_ID extends LogKey
+ case object SQL_TEXT extends LogKey
+ case object SRC_PATH extends LogKey
+ case object STAGE_ATTEMPT extends LogKey
+ case object STAGE_ID extends LogKey
+ case object START_INDEX extends LogKey
+ case object STATEMENT_ID extends LogKey
+ case object STATE_STORE_ID extends LogKey
+ case object STATE_STORE_PROVIDER extends LogKey
+ case object STATE_STORE_VERSION extends LogKey
+ case object STATUS extends LogKey
+ case object STDERR extends LogKey
+ case object STORAGE_LEVEL extends LogKey
+ case object STORAGE_LEVEL_DESERIALIZED extends LogKey
+ case object STORAGE_LEVEL_REPLICATION extends LogKey
+ case object STORE_ID extends LogKey
+ case object STREAMING_DATA_SOURCE_DESCRIPTION extends LogKey
+ case object STREAMING_DATA_SOURCE_NAME extends LogKey
+ case object STREAMING_OFFSETS_END extends LogKey
+ case object STREAMING_OFFSETS_START extends LogKey
+ case object STREAMING_QUERY_PROGRESS extends LogKey
+ case object STREAMING_SOURCE extends LogKey
+ case object STREAMING_TABLE extends LogKey
+ case object STREAMING_WRITE extends LogKey
+ case object STREAM_ID extends LogKey
+ case object STREAM_NAME extends LogKey
+ case object SUBMISSION_ID extends LogKey
+ case object SUBSAMPLING_RATE extends LogKey
+ case object SUB_QUERY extends LogKey
+ case object TABLE_NAME extends LogKey
+ case object TABLE_TYPES extends LogKey
+ case object TARGET_PATH extends LogKey
+ case object TASK_ATTEMPT_ID extends LogKey
+ case object TASK_ID extends LogKey
+ case object TASK_NAME extends LogKey
+ case object TASK_SET_NAME extends LogKey
+ case object TASK_STATE extends LogKey
+ case object TEMP_FILE extends LogKey
+ case object TEMP_PATH extends LogKey
+ case object TEST_SIZE extends LogKey
+ case object THREAD extends LogKey
+ case object THREAD_NAME extends LogKey
+ case object TID extends LogKey
+ case object TIME extends LogKey
+ case object TIMEOUT extends LogKey
+ case object TIMER extends LogKey
+ case object TIME_UNITS extends LogKey
+ case object TIP extends LogKey
+ case object TOKEN_REGEX extends LogKey
+ case object TOPIC extends LogKey
+ case object TOPIC_PARTITION extends LogKey
+ case object TOPIC_PARTITIONS extends LogKey
+ case object TOPIC_PARTITION_OFFSET extends LogKey
+ case object TOPIC_PARTITION_OFFSET_RANGE extends LogKey
+ case object TOTAL extends LogKey
+ case object TOTAL_EFFECTIVE_TIME extends LogKey
+ case object TOTAL_RECORDS_READ extends LogKey
+ case object TOTAL_TIME extends LogKey
+ case object TOTAL_TIME_READ extends LogKey
+ case object TO_TIME extends LogKey
+ case object TRAINING_SIZE extends LogKey
+ case object TRAIN_VALIDATION_SPLIT_METRIC extends LogKey
+ case object TRAIN_VALIDATION_SPLIT_METRICS extends LogKey
+ case object TRAIN_WORD_COUNT extends LogKey
+ case object TREE_NODE extends LogKey
+ case object TRIGGER_INTERVAL extends LogKey
+ case object UI_FILTER extends LogKey
+ case object UI_FILTER_PARAMS extends LogKey
+ case object UI_PROXY_BASE extends LogKey
+ case object UNSUPPORTED_EXPR extends LogKey
+ case object UNSUPPORTED_HINT_REASON extends LogKey
+ case object UNTIL_OFFSET extends LogKey
+ case object UPPER_BOUND extends LogKey
+ case object URI extends LogKey
+ case object USER_ID extends LogKey
+ case object USER_NAME extends LogKey
+ case object VALUE extends LogKey
+ case object VERSION_NUMBER extends LogKey
+ case object VIRTUAL_CORES extends LogKey
+ case object VOCAB_SIZE extends LogKey
+ case object WAIT_RESULT_TIME extends LogKey
+ case object WAIT_SEND_TIME extends LogKey
+ case object WAIT_TIME extends LogKey
+ case object WATERMARK_CONSTRAINT extends LogKey
+ case object WEIGHTED_NUM extends LogKey
+ case object WORKER_URL extends LogKey
+ case object WRITE_AHEAD_LOG_INFO extends LogKey
+ case object WRITE_JOB_UUID extends LogKey
+ case object XSD_PATH extends LogKey
}
diff --git a/common/utils/src/main/scala/org/apache/spark/internal/Logging.scala b/common/utils/src/main/scala/org/apache/spark/internal/Logging.scala
index 607f3637e641..085b22bee5f3 100644
--- a/common/utils/src/main/scala/org/apache/spark/internal/Logging.scala
+++ b/common/utils/src/main/scala/org/apache/spark/internal/Logging.scala
@@ -29,7 +29,6 @@ import org.apache.logging.log4j.core.filter.AbstractFilter
import org.slf4j.{Logger, LoggerFactory}
import org.apache.spark.internal.Logging.SparkShellLoggingFilter
-import org.apache.spark.internal.LogKey.LogKey
import org.apache.spark.util.SparkClassUtils
/**
diff --git a/common/utils/src/main/scala/org/apache/spark/internal/README.md b/common/utils/src/main/scala/org/apache/spark/internal/README.md
index 81c542fd3d9c..c0190b965834 100644
--- a/common/utils/src/main/scala/org/apache/spark/internal/README.md
+++ b/common/utils/src/main/scala/org/apache/spark/internal/README.md
@@ -2,7 +2,7 @@
## LogKey
-LogKeys serve as identifiers for mapped diagnostic contexts (MDC) within logs. Follow these guidelines when adding new LogKeys:
+`LogKey`s serve as identifiers for mapped diagnostic contexts (MDC) within logs. Follow these guidelines when adding a new LogKey:
* Define all structured logging keys in `LogKey.scala`, and sort them alphabetically for ease of search.
* Use `UPPER_SNAKE_CASE` for key names.
* Key names should be both simple and broad, yet include specific identifiers like `STAGE_ID`, `TASK_ID`, and `JOB_ID` when needed for clarity. For instance, use `MAX_ATTEMPTS` as a general key instead of creating separate keys for each scenario such as `EXECUTOR_STATE_SYNC_MAX_ATTEMPTS` and `MAX_TASK_FAILURES`. This balances simplicity with the detail needed for effective logging.
diff --git a/common/utils/src/test/scala/org/apache/spark/util/LogKeySuite.scala b/common/utils/src/test/scala/org/apache/spark/util/LogKeySuite.scala
index 24a24538ad72..17e360f510a2 100644
--- a/common/utils/src/test/scala/org/apache/spark/util/LogKeySuite.scala
+++ b/common/utils/src/test/scala/org/apache/spark/util/LogKeySuite.scala
@@ -22,16 +22,16 @@ import java.nio.file.{Files, Path}
import java.util.{ArrayList => JList}
import scala.jdk.CollectionConverters._
+import scala.reflect.runtime.universe._
import org.apache.commons.io.FileUtils
import org.scalatest.funsuite.AnyFunSuite // scalastyle:ignore funsuite
-import org.apache.spark.internal.{Logging, LogKey}
-import org.apache.spark.internal.LogKey.LogKey
+import org.apache.spark.internal.{Logging, LogKeys}
// scalastyle:off line.size.limit
/**
- * To re-generate the LogKey class file, run:
+ * To re-generate the file `LogKey.scala`, run:
* {{{
* SPARK_GENERATE_GOLDEN_FILES=1 build/sbt "common-utils/testOnly org.apache.spark.util.LogKeySuite"
* }}}
@@ -57,20 +57,20 @@ class LogKeySuite
private val logKeyFilePath = getWorkspaceFilePath("common", "utils", "src", "main", "scala",
"org", "apache", "spark", "internal", "LogKey.scala")
- // regenerate the file `LogKey.scala` with its enumeration fields sorted alphabetically
+ // regenerate the file `LogKey.scala` with its members sorted alphabetically
private def regenerateLogKeyFile(
- originalKeys: Seq[LogKey], sortedKeys: Seq[LogKey]): Unit = {
+ originalKeys: Seq[String], sortedKeys: Seq[String]): Unit = {
if (originalKeys != sortedKeys) {
val logKeyFile = logKeyFilePath.toFile
- logInfo(s"Regenerating LogKey file $logKeyFile")
+ logInfo(s"Regenerating the file $logKeyFile")
val originalContents = FileUtils.readLines(logKeyFile, StandardCharsets.UTF_8)
val sortedContents = new JList[String]()
var firstMatch = false
originalContents.asScala.foreach { line =>
- if (line.trim.startsWith("val ") && line.trim.endsWith(" = Value")) {
+ if (line.trim.startsWith("case object ") && line.trim.endsWith(" extends LogKey")) {
if (!firstMatch) {
- sortedKeys.foreach { logKey =>
- sortedContents.add(s" val ${logKey.toString} = Value")
+ sortedKeys.foreach { key =>
+ sortedContents.add(s" case object $key extends LogKey")
}
firstMatch = true
}
@@ -83,14 +83,21 @@ class LogKeySuite
}
}
- test("LogKey enumeration fields are correctly sorted") {
- val originalKeys = LogKey.values.toSeq
- val sortedKeys = originalKeys.sortBy(_.toString)
+ test("The members of LogKeys are correctly sorted") {
+ val originalKeys = getAllLogKeys.reverse
+ val sortedKeys = originalKeys.sorted
if (regenerateGoldenFiles) {
regenerateLogKeyFile(originalKeys, sortedKeys)
} else {
assert(originalKeys === sortedKeys,
- "LogKey enumeration fields must be sorted alphabetically")
+ "The members of LogKeys must be sorted alphabetically")
}
}
+
+ private def getAllLogKeys: Seq[String] = {
+ val logKeysType = typeOf[LogKeys.type]
+ val classSymbol = logKeysType.typeSymbol.asClass
+ val members = classSymbol.typeSignature.members
+ members.filter(m => m.isTerm && !m.isMethod).map(_.name.toString).toSeq
+ }
}
diff --git a/common/utils/src/test/scala/org/apache/spark/util/MDCSuite.scala b/common/utils/src/test/scala/org/apache/spark/util/MDCSuite.scala
index 1983f185e8c8..7631c2566221 100644
--- a/common/utils/src/test/scala/org/apache/spark/util/MDCSuite.scala
+++ b/common/utils/src/test/scala/org/apache/spark/util/MDCSuite.scala
@@ -22,7 +22,7 @@ import scala.jdk.CollectionConverters._
import org.scalatest.funsuite.AnyFunSuite // scalastyle:ignore funsuite
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{EXIT_CODE, OFFSET, RANGE}
+import org.apache.spark.internal.LogKeys.{EXIT_CODE, OFFSET, RANGE}
class MDCSuite
extends AnyFunSuite // scalastyle:ignore funsuite
diff --git a/common/utils/src/test/scala/org/apache/spark/util/PatternLoggingSuite.scala b/common/utils/src/test/scala/org/apache/spark/util/PatternLoggingSuite.scala
index d06405ab6d99..3baa720f38a9 100644
--- a/common/utils/src/test/scala/org/apache/spark/util/PatternLoggingSuite.scala
+++ b/common/utils/src/test/scala/org/apache/spark/util/PatternLoggingSuite.scala
@@ -47,6 +47,10 @@ class PatternLoggingSuite extends LoggingSuiteBase with BeforeAndAfterAll {
override def expectedPatternForMsgWithMDCAndException(level: Level): String =
s""".*$level $className: Error in executor 1.\njava.lang.RuntimeException: OOM\n[\\s\\S]*"""
+ override def expectedPatternForExternalSystemCustomLogKey(level: Level): String = {
+ s""".*$level $className: External system custom log message.\n"""
+ }
+
override def verifyMsgWithConcat(level: Level, logOutput: String): Unit = {
val pattern =
s""".*$level $className: Min Size: 2, Max Size: 4. Please double check.\n"""
diff --git a/common/utils/src/test/scala/org/apache/spark/util/StructuredLoggingSuite.scala b/common/utils/src/test/scala/org/apache/spark/util/StructuredLoggingSuite.scala
index fe03c190fa85..f9966a985baa 100644
--- a/common/utils/src/test/scala/org/apache/spark/util/StructuredLoggingSuite.scala
+++ b/common/utils/src/test/scala/org/apache/spark/util/StructuredLoggingSuite.scala
@@ -25,8 +25,7 @@ import com.fasterxml.jackson.module.scala.DefaultScalaModule
import org.apache.logging.log4j.Level
import org.scalatest.funsuite.AnyFunSuite // scalastyle:ignore funsuite
-import org.apache.spark.internal.{LogEntry, Logging, MDC}
-import org.apache.spark.internal.LogKey.{EXECUTOR_ID, MAX_SIZE, MIN_SIZE}
+import org.apache.spark.internal.{LogEntry, Logging, LogKey, LogKeys, MDC}
trait LoggingSuiteBase
extends AnyFunSuite // scalastyle:ignore funsuite
@@ -54,14 +53,14 @@ trait LoggingSuiteBase
def basicMsg: String = "This is a log message"
- def msgWithMDC: LogEntry = log"Lost executor ${MDC(EXECUTOR_ID, "1")}."
+ def msgWithMDC: LogEntry = log"Lost executor ${MDC(LogKeys.EXECUTOR_ID, "1")}."
- def msgWithMDCValueIsNull: LogEntry = log"Lost executor ${MDC(EXECUTOR_ID, null)}."
+ def msgWithMDCValueIsNull: LogEntry = log"Lost executor ${MDC(LogKeys.EXECUTOR_ID, null)}."
- def msgWithMDCAndException: LogEntry = log"Error in executor ${MDC(EXECUTOR_ID, "1")}."
+ def msgWithMDCAndException: LogEntry = log"Error in executor ${MDC(LogKeys.EXECUTOR_ID, "1")}."
- def msgWithConcat: LogEntry = log"Min Size: ${MDC(MIN_SIZE, "2")}, " +
- log"Max Size: ${MDC(MAX_SIZE, "4")}. " +
+ def msgWithConcat: LogEntry = log"Min Size: ${MDC(LogKeys.MIN_SIZE, "2")}, " +
+ log"Max Size: ${MDC(LogKeys.MAX_SIZE, "4")}. " +
log"Please double check."
// test for basic message (without any mdc)
@@ -79,6 +78,9 @@ trait LoggingSuiteBase
// test for message and exception
def expectedPatternForMsgWithMDCAndException(level: Level): String
+ // test for external system custom LogKey
+ def expectedPatternForExternalSystemCustomLogKey(level: Level): String
+
def verifyMsgWithConcat(level: Level, logOutput: String): Unit
test("Basic logging") {
@@ -144,6 +146,21 @@ trait LoggingSuiteBase
}
}
+ private val externalSystemCustomLog =
+ log"${MDC(CUSTOM_LOG_KEY, "External system custom log message.")}"
+ test("Logging with external system custom LogKey") {
+ Seq(
+ (Level.ERROR, () => logError(externalSystemCustomLog)),
+ (Level.WARN, () => logWarning(externalSystemCustomLog)),
+ (Level.INFO, () => logInfo(externalSystemCustomLog)),
+ (Level.DEBUG, () => logDebug(externalSystemCustomLog)),
+ (Level.TRACE, () => logTrace(externalSystemCustomLog))).foreach {
+ case (level, logFunc) =>
+ val logOutput = captureLogOutput(logFunc)
+ assert(expectedPatternForExternalSystemCustomLogKey(level).r.matches(logOutput))
+ }
+ }
+
test("Logging with concat") {
Seq(
(Level.ERROR, () => logError(msgWithConcat)),
@@ -244,6 +261,21 @@ class StructuredLoggingSuite extends LoggingSuiteBase {
}""")
}
+ override def expectedPatternForExternalSystemCustomLogKey(level: Level): String = {
+ compactAndToRegexPattern(
+ s"""
+ {
+ "ts": "<timestamp>",
+ "level": "$level",
+ "msg": "External system custom log message.",
+ "context": {
+ "custom_log_key": "External system custom log message."
+ },
+ "logger": "$className"
+ }"""
+ )
+ }
+
override def verifyMsgWithConcat(level: Level, logOutput: String): Unit = {
val pattern1 = compactAndToRegexPattern(
s"""
@@ -273,3 +305,6 @@ class StructuredLoggingSuite extends LoggingSuiteBase {
assert(pattern1.r.matches(logOutput) || pattern2.r.matches(logOutput))
}
}
+
+// External system custom LogKey must be `extends LogKey`
+case object CUSTOM_LOG_KEY extends LogKey
diff --git a/connector/avro/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala b/connector/avro/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala
index 4bedd625e609..7cbc30f1fb3d 100644
--- a/connector/avro/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala
+++ b/connector/avro/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala
@@ -32,7 +32,7 @@ import org.apache.hadoop.mapreduce.Job
import org.apache.spark.{SparkException, SparkIllegalArgumentException}
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{CODEC_LEVEL, CODEC_NAME, CONFIG, PATH}
+import org.apache.spark.internal.LogKeys.{CODEC_LEVEL, CODEC_NAME, CONFIG, PATH}
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.avro.AvroCompressionCodec._
import org.apache.spark.sql.avro.AvroOptions.IGNORE_EXTENSION
diff --git a/connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/GrpcRetryHandler.scala b/connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/GrpcRetryHandler.scala
index 508dad3d748d..afdc75dc8f70 100644
--- a/connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/GrpcRetryHandler.scala
+++ b/connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/GrpcRetryHandler.scala
@@ -22,7 +22,7 @@ import scala.util.control.NonFatal
import io.grpc.stub.StreamObserver
import org.apache.spark.internal.Logging
-import org.apache.spark.internal.LogKey.{ERROR, POLICY, RETRY_COUNT, WAIT_TIME}
+import org.apache.spark.internal.LogKeys.{ERROR, POLICY, RETRY_COUNT, WAIT_TIME}
import org.apache.spark.internal.MDC
private[sql] class GrpcRetryHandler(
diff --git a/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/execution/ExecuteGrpcResponseSender.scala b/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/execution/ExecuteGrpcResponseSender.scala
index d4709db081fc..3e360372d560 100644
--- a/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/execution/ExecuteGrpcResponseSender.scala
+++ b/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/execution/ExecuteGrpcResponseSender.scala
@@ -25,7 +25,7 @@ import io.grpc.stub.{ServerCallStreamObserver, StreamObserver}
import org.apache.spark.{SparkEnv, SparkSQLException}
import org.apache.spark.connect.proto.ExecutePlanResponse
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey._
+import org.apache.spark.internal.LogKeys._
import org.apache.spark.sql.catalyst.util.DateTimeConstants.NANOS_PER_MILLIS
import org.apache.spark.sql.connect.common.ProtoUtils
import org.apache.spark.sql.connect.config.Connect.{CONNECT_EXECUTE_REATTACHABLE_SENDER_MAX_STREAM_DURATION, CONNECT_EXECUTE_REATTACHABLE_SENDER_MAX_STREAM_SIZE, CONNECT_PROGRESS_REPORT_INTERVAL}
diff --git a/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/execution/ExecuteResponseObserver.scala b/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/execution/ExecuteResponseObserver.scala
index 1a6ceffd41c3..9d83d93083dc 100644
--- a/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/execution/ExecuteResponseObserver.scala
+++ b/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/execution/ExecuteResponseObserver.scala
@@ -27,7 +27,7 @@ import io.grpc.stub.StreamObserver
import org.apache.spark.{SparkEnv, SparkSQLException}
import org.apache.spark.connect.proto
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey
+import org.apache.spark.internal.LogKeys
import org.apache.spark.sql.connect.config.Connect.CONNECT_EXECUTE_REATTACHABLE_OBSERVER_RETRY_BUFFER_SIZE
import org.apache.spark.sql.connect.service.ExecuteHolder
@@ -245,13 +245,13 @@ private[connect] class ExecuteResponseObserver[T <: Message](val executeHolder:
removeResponsesUntilIndex(lastProducedIndex)
// scalastyle:off line.size.limit
logInfo(
- log"Release all for opId=${MDC(LogKey.OP_ID, executeHolder.operationId)}. Execution stats: " +
- log"total=${MDC(LogKey.TOTAL, totalSize)} " +
- log"autoRemoved=${MDC(LogKey.CACHE_AUTO_REMOVED_SIZE, autoRemovedSize)} " +
- log"cachedUntilConsumed=${MDC(LogKey.CACHE_UNTIL_HIGHEST_CONSUMED_SIZE, cachedSizeUntilHighestConsumed)} " +
- log"cachedUntilProduced=${MDC(LogKey.CACHE_UNTIL_LAST_PRODUCED_SIZE, cachedSizeUntilLastProduced)} " +
- log"maxCachedUntilConsumed=${MDC(LogKey.MAX_CACHE_UNTIL_HIGHEST_CONSUMED_SIZE, cachedSizeUntilHighestConsumed.max)} " +
- log"maxCachedUntilProduced=${MDC(LogKey.MAX_CACHE_UNTIL_LAST_PRODUCED_SIZE, cachedSizeUntilLastProduced.max)}")
+ log"Release all for opId=${MDC(LogKeys.OP_ID, executeHolder.operationId)}. Execution stats: " +
+ log"total=${MDC(LogKeys.TOTAL, totalSize)} " +
+ log"autoRemoved=${MDC(LogKeys.CACHE_AUTO_REMOVED_SIZE, autoRemovedSize)} " +
+ log"cachedUntilConsumed=${MDC(LogKeys.CACHE_UNTIL_HIGHEST_CONSUMED_SIZE, cachedSizeUntilHighestConsumed)} " +
+ log"cachedUntilProduced=${MDC(LogKeys.CACHE_UNTIL_LAST_PRODUCED_SIZE, cachedSizeUntilLastProduced)} " +
+ log"maxCachedUntilConsumed=${MDC(LogKeys.MAX_CACHE_UNTIL_HIGHEST_CONSUMED_SIZE, cachedSizeUntilHighestConsumed.max)} " +
+ log"maxCachedUntilProduced=${MDC(LogKeys.MAX_CACHE_UNTIL_LAST_PRODUCED_SIZE, cachedSizeUntilLastProduced.max)}")
// scalastyle:on line.size.limit
}
diff --git a/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala b/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala
index fab222146d9f..c0ae0d2c6a13 100644
--- a/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala
+++ b/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala
@@ -39,7 +39,7 @@ import org.apache.spark.connect.proto.Parse.ParseFormat
import org.apache.spark.connect.proto.StreamingQueryManagerCommandResult.StreamingQueryInstance
import org.apache.spark.connect.proto.WriteStreamOperationStart.TriggerCase
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.SESSION_ID
+import org.apache.spark.internal.LogKeys.SESSION_ID
import org.apache.spark.ml.{functions => MLFunctions}
import org.apache.spark.resource.{ExecutorResourceRequest, ResourceProfile, TaskResourceProfile, TaskResourceRequest}
import org.apache.spark.sql.{Column, Dataset, Encoders, ForeachWriter, Observation, RelationalGroupedDataset, SparkSession}
diff --git a/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/StreamingForeachBatchHelper.scala b/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/StreamingForeachBatchHelper.scala
index ef5faac77e3e..df883a5c8681 100644
--- a/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/StreamingForeachBatchHelper.scala
+++ b/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/StreamingForeachBatchHelper.scala
@@ -27,7 +27,7 @@ import scala.util.control.NonFatal
import org.apache.spark.SparkException
import org.apache.spark.api.python.{PythonException, PythonWorkerUtils, SimplePythonFunction, SpecialLengths, StreamingPythonRunner}
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{DATAFRAME_ID, QUERY_ID, RUN_ID, SESSION_ID}
+import org.apache.spark.internal.LogKeys.{DATAFRAME_ID, QUERY_ID, RUN_ID, SESSION_ID}
import org.apache.spark.sql.DataFrame
import org.apache.spark.sql.connect.service.SessionHolder
import org.apache.spark.sql.connect.service.SparkConnectService
diff --git a/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/StreamingQueryListenerHelper.scala b/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/StreamingQueryListenerHelper.scala
index 74e9e32f208d..c342050a212e 100644
--- a/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/StreamingQueryListenerHelper.scala
+++ b/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/StreamingQueryListenerHelper.scala
@@ -22,7 +22,7 @@ import java.io.EOFException
import org.apache.spark.SparkException
import org.apache.spark.api.python.{PythonException, PythonWorkerUtils, SimplePythonFunction, SpecialLengths, StreamingPythonRunner}
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.FUNCTION_NAME
+import org.apache.spark.internal.LogKeys.FUNCTION_NAME
import org.apache.spark.sql.connect.service.{SessionHolder, SparkConnectService}
import org.apache.spark.sql.streaming.StreamingQueryListener
diff --git a/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/LoggingInterceptor.scala b/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/LoggingInterceptor.scala
index c82cadbd5f7a..a071579692fb 100644
--- a/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/LoggingInterceptor.scala
+++ b/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/LoggingInterceptor.scala
@@ -29,7 +29,7 @@ import io.grpc.ServerCallHandler
import io.grpc.ServerInterceptor
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{DESCRIPTION, MESSAGE}
+import org.apache.spark.internal.LogKeys.{DESCRIPTION, MESSAGE}
/**
* A gRPC interceptor to log RPC requests and responses. It logs the protobufs as JSON. Useful for
diff --git a/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SessionHolder.scala b/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SessionHolder.scala
index bb32ac1275fb..2f35662ea59a 100644
--- a/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SessionHolder.scala
+++ b/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SessionHolder.scala
@@ -33,7 +33,7 @@ import org.apache.spark.{SparkEnv, SparkException, SparkSQLException}
import org.apache.spark.api.python.PythonFunction.PythonAccumulator
import org.apache.spark.connect.proto
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey._
+import org.apache.spark.internal.LogKeys._
import org.apache.spark.sql.DataFrame
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
diff --git a/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectExecutionManager.scala b/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectExecutionManager.scala
index 4fe7f3eceb81..6681a5f509c6 100644
--- a/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectExecutionManager.scala
+++ b/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectExecutionManager.scala
@@ -29,7 +29,7 @@ import com.google.common.cache.CacheBuilder
import org.apache.spark.{SparkEnv, SparkSQLException}
import org.apache.spark.connect.proto
-import org.apache.spark.internal.{Logging, LogKey, MDC}
+import org.apache.spark.internal.{Logging, LogKeys, MDC}
import org.apache.spark.sql.connect.config.Connect.{CONNECT_EXECUTE_MANAGER_ABANDONED_TOMBSTONES_SIZE, CONNECT_EXECUTE_MANAGER_DETACHED_TIMEOUT, CONNECT_EXECUTE_MANAGER_MAINTENANCE_INTERVAL}
import org.apache.spark.util.ThreadUtils
@@ -95,7 +95,7 @@ private[connect] class SparkConnectExecutionManager() extends Logging {
sessionHolder.addExecuteHolder(executeHolder)
executions.put(executeHolder.key, executeHolder)
lastExecutionTimeMs = None
- logInfo(log"ExecuteHolder ${MDC(LogKey.EXECUTE_KEY, executeHolder.key)} is created.")
+ logInfo(log"ExecuteHolder ${MDC(LogKeys.EXECUTE_KEY, executeHolder.key)} is created.")
}
schedulePeriodicChecks() // Starts the maintenance thread if it hasn't started.
@@ -122,7 +122,7 @@ private[connect] class SparkConnectExecutionManager() extends Logging {
if (executions.isEmpty) {
lastExecutionTimeMs = Some(System.currentTimeMillis())
}
- logInfo(log"ExecuteHolder ${MDC(LogKey.EXECUTE_KEY, key)} is removed.")
+ logInfo(log"ExecuteHolder ${MDC(LogKeys.EXECUTE_KEY, key)} is removed.")
}
// close the execution outside the lock
executeHolder.foreach { e =>
@@ -147,7 +147,7 @@ private[connect] class SparkConnectExecutionManager() extends Logging {
sessionExecutionHolders.foreach { case (_, executeHolder) =>
val info = executeHolder.getExecuteInfo
logInfo(
- log"Execution ${MDC(LogKey.EXECUTE_INFO, info)} removed in removeSessionExecutions.")
+ log"Execution ${MDC(LogKeys.EXECUTE_INFO, info)} removed in removeSessionExecutions.")
removeExecuteHolder(executeHolder.key, abandoned = true)
}
}
@@ -202,7 +202,7 @@ private[connect] class SparkConnectExecutionManager() extends Logging {
val interval = SparkEnv.get.conf.get(CONNECT_EXECUTE_MANAGER_MAINTENANCE_INTERVAL)
logInfo(
log"Starting thread for cleanup of abandoned executions every " +
- log"${MDC(LogKey.INTERVAL, interval)} ms")
+ log"${MDC(LogKeys.INTERVAL, interval)} ms")
scheduledExecutor = Some(Executors.newSingleThreadScheduledExecutor())
scheduledExecutor.get.scheduleAtFixedRate(
() => {
@@ -242,7 +242,7 @@ private[connect] class SparkConnectExecutionManager() extends Logging {
toRemove.foreach { executeHolder =>
val info = executeHolder.getExecuteInfo
logInfo(
- log"Found execution ${MDC(LogKey.EXECUTE_INFO, info)} that was abandoned " +
+ log"Found execution ${MDC(LogKeys.EXECUTE_INFO, info)} that was abandoned " +
log"and expired and will be removed.")
removeExecuteHolder(executeHolder.key, abandoned = true)
}
diff --git a/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectServer.scala b/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectServer.scala
index c55600886a39..1938ec00f48d 100644
--- a/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectServer.scala
+++ b/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectServer.scala
@@ -22,7 +22,7 @@ import java.net.InetSocketAddress
import scala.jdk.CollectionConverters._
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{HOST, PORT}
+import org.apache.spark.internal.LogKeys.{HOST, PORT}
import org.apache.spark.sql.SparkSession
/**
diff --git a/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectService.scala b/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectService.scala
index 4b35971286dd..7b2d6f0bd5fe 100644
--- a/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectService.scala
+++ b/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectService.scala
@@ -36,7 +36,7 @@ import org.apache.spark.connect.proto
import org.apache.spark.connect.proto.{AddArtifactsRequest, AddArtifactsResponse, SparkConnectServiceGrpc}
import org.apache.spark.connect.proto.SparkConnectServiceGrpc.AsyncService
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.HOST
+import org.apache.spark.internal.LogKeys.HOST
import org.apache.spark.internal.config.UI.UI_ENABLED
import org.apache.spark.sql.connect.config.Connect.{CONNECT_GRPC_BINDING_ADDRESS, CONNECT_GRPC_BINDING_PORT, CONNECT_GRPC_MARSHALLER_RECURSION_LIMIT, CONNECT_GRPC_MAX_INBOUND_MESSAGE_SIZE}
import org.apache.spark.sql.connect.execution.ConnectProgressExecutionListener
diff --git a/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectSessionManager.scala b/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectSessionManager.scala
index 1a34964932ef..edaaa640bf12 100644
--- a/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectSessionManager.scala
+++ b/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectSessionManager.scala
@@ -30,7 +30,7 @@ import com.google.common.cache.CacheBuilder
import org.apache.spark.{SparkEnv, SparkSQLException}
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{INTERVAL, SESSION_HOLD_INFO}
+import org.apache.spark.internal.LogKeys.{INTERVAL, SESSION_HOLD_INFO}
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.connect.config.Connect.{CONNECT_SESSION_MANAGER_CLOSED_SESSIONS_TOMBSTONES_SIZE, CONNECT_SESSION_MANAGER_DEFAULT_SESSION_TIMEOUT, CONNECT_SESSION_MANAGER_MAINTENANCE_INTERVAL}
import org.apache.spark.util.ThreadUtils
diff --git a/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectStreamingQueryCache.scala b/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectStreamingQueryCache.scala
index 4c9b3baa689b..91eae18f2d5d 100644
--- a/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectStreamingQueryCache.scala
+++ b/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectStreamingQueryCache.scala
@@ -27,7 +27,7 @@ import scala.concurrent.duration.{Duration, DurationInt, FiniteDuration}
import scala.util.control.NonFatal
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{DURATION, NEW_VALUE, OLD_VALUE, QUERY_CACHE_VALUE, QUERY_ID, SESSION_ID}
+import org.apache.spark.internal.LogKeys.{DURATION, NEW_VALUE, OLD_VALUE, QUERY_CACHE_VALUE, QUERY_ID, SESSION_ID}
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.streaming.StreamingQuery
import org.apache.spark.util.{Clock, SystemClock, ThreadUtils}
diff --git a/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerListener.scala b/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerListener.scala
index a1bbab7dbdbc..65db08be7f90 100644
--- a/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerListener.scala
+++ b/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerListener.scala
@@ -22,7 +22,7 @@ import scala.collection.mutable.ArrayBuffer
import org.apache.spark.{SparkConf, SparkContext, SparkEnv}
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{OP_ID, SESSION_ID}
+import org.apache.spark.internal.LogKeys.{OP_ID, SESSION_ID}
import org.apache.spark.internal.config.Status.LIVE_ENTITY_UPDATE_PERIOD
import org.apache.spark.scheduler._
import org.apache.spark.sql.connect.config.Connect.{CONNECT_UI_SESSION_LIMIT, CONNECT_UI_STATEMENT_LIMIT}
diff --git a/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/utils/ErrorUtils.scala b/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/utils/ErrorUtils.scala
index b1bfe71930fb..773a97e92973 100644
--- a/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/utils/ErrorUtils.scala
+++ b/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/utils/ErrorUtils.scala
@@ -39,7 +39,7 @@ import org.apache.spark.{SparkEnv, SparkException, SparkThrowable}
import org.apache.spark.api.python.PythonException
import org.apache.spark.connect.proto.FetchErrorDetailsResponse
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{OP_TYPE, SESSION_ID, USER_ID}
+import org.apache.spark.internal.LogKeys.{OP_TYPE, SESSION_ID, USER_ID}
import org.apache.spark.sql.connect.config.Connect
import org.apache.spark.sql.connect.service.{ExecuteEventsManager, SessionHolder, SessionKey, SparkConnectService}
import org.apache.spark.sql.internal.SQLConf
diff --git a/connector/connect/server/src/test/scala/org/apache/spark/sql/connect/ProtoToParsedPlanTestSuite.scala b/connector/connect/server/src/test/scala/org/apache/spark/sql/connect/ProtoToParsedPlanTestSuite.scala
index d404779d7a92..264c6aa70ae2 100644
--- a/connector/connect/server/src/test/scala/org/apache/spark/sql/connect/ProtoToParsedPlanTestSuite.scala
+++ b/connector/connect/server/src/test/scala/org/apache/spark/sql/connect/ProtoToParsedPlanTestSuite.scala
@@ -26,7 +26,7 @@ import scala.util.{Failure, Success, Try}
import org.apache.spark.{SparkConf, SparkFunSuite}
import org.apache.spark.connect.proto
-import org.apache.spark.internal.LogKey.PATH
+import org.apache.spark.internal.LogKeys.PATH
import org.apache.spark.internal.MDC
import org.apache.spark.sql.catalyst.{catalog, QueryPlanningTracker}
import org.apache.spark.sql.catalyst.analysis.{caseSensitiveResolution, Analyzer, FunctionRegistry, Resolver, TableFunctionRegistry}
diff --git a/connector/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/DockerJDBCIntegrationSuite.scala b/connector/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/DockerJDBCIntegrationSuite.scala
index fc095c5f5b31..ded7bb3a6bf6 100644
--- a/connector/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/DockerJDBCIntegrationSuite.scala
+++ b/connector/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/DockerJDBCIntegrationSuite.scala
@@ -36,7 +36,7 @@ import com.github.dockerjava.zerodep.ZerodepDockerHttpClient
import org.scalatest.concurrent.{Eventually, PatienceConfiguration}
import org.scalatest.time.SpanSugar._
-import org.apache.spark.internal.LogKey.{CLASS_NAME, CONTAINER, STATUS}
+import org.apache.spark.internal.LogKeys.{CLASS_NAME, CONTAINER, STATUS}
import org.apache.spark.internal.MDC
import org.apache.spark.sql.QueryTest
import org.apache.spark.sql.test.SharedSparkSession
diff --git a/connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaBatchPartitionReader.scala b/connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaBatchPartitionReader.scala
index d3fe3264afe1..cb1c7055483b 100644
--- a/connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaBatchPartitionReader.scala
+++ b/connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaBatchPartitionReader.scala
@@ -21,7 +21,7 @@ import java.{util => ju}
import org.apache.spark.TaskContext
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey._
+import org.apache.spark.internal.LogKeys._
import org.apache.spark.sql.catalyst.InternalRow
import org.apache.spark.sql.catalyst.expressions.UnsafeRow
import org.apache.spark.sql.connector.metric.CustomTaskMetric
diff --git a/connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaContinuousStream.scala b/connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaContinuousStream.scala
index e5e22243a582..10bdbb1d9d44 100644
--- a/connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaContinuousStream.scala
+++ b/connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaContinuousStream.scala
@@ -25,7 +25,7 @@ import org.apache.kafka.common.TopicPartition
import org.apache.spark.TaskContext
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{ERROR, OFFSETS, TIP}
+import org.apache.spark.internal.LogKeys.{ERROR, OFFSETS, TIP}
import org.apache.spark.sql.catalyst.InternalRow
import org.apache.spark.sql.catalyst.expressions.UnsafeRow
import org.apache.spark.sql.connector.read.InputPartition
diff --git a/connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchStream.scala b/connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchStream.scala
index 3313d42d1a30..c79da13017b9 100644
--- a/connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchStream.scala
+++ b/connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchStream.scala
@@ -26,7 +26,7 @@ import org.apache.kafka.common.TopicPartition
import org.apache.spark.SparkEnv
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{ERROR, OFFSETS, TIP}
+import org.apache.spark.internal.LogKeys.{ERROR, OFFSETS, TIP}
import org.apache.spark.internal.config.Network.NETWORK_TIMEOUT
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.connector.read.{InputPartition, PartitionReaderFactory}
diff --git a/connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReaderAdmin.scala b/connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReaderAdmin.scala
index 5ed8576e8888..39c842216cf0 100644
--- a/connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReaderAdmin.scala
+++ b/connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReaderAdmin.scala
@@ -31,7 +31,7 @@ import org.apache.kafka.common.requests.OffsetFetchResponse
import org.apache.spark.SparkEnv
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{OFFSETS, RETRY_COUNT, TOPIC_PARTITION_OFFSET}
+import org.apache.spark.internal.LogKeys.{OFFSETS, RETRY_COUNT, TOPIC_PARTITION_OFFSET}
import org.apache.spark.scheduler.ExecutorCacheTaskLocation
import org.apache.spark.sql.catalyst.util.CaseInsensitiveMap
import org.apache.spark.sql.kafka010.KafkaSourceProvider.StrategyOnNoMatchStartingOffset
diff --git a/connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReaderConsumer.scala b/connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReaderConsumer.scala
index 34d44fdf1059..17db894f504b 100644
--- a/connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReaderConsumer.scala
+++ b/connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReaderConsumer.scala
@@ -28,7 +28,7 @@ import org.apache.kafka.common.TopicPartition
import org.apache.spark.SparkEnv
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{OFFSETS, RETRY_COUNT, TOPIC_PARTITION_OFFSET}
+import org.apache.spark.internal.LogKeys.{OFFSETS, RETRY_COUNT, TOPIC_PARTITION_OFFSET}
import org.apache.spark.scheduler.ExecutorCacheTaskLocation
import org.apache.spark.sql.catalyst.util.CaseInsensitiveMap
import org.apache.spark.sql.kafka010.KafkaSourceProvider.StrategyOnNoMatchStartingOffset
diff --git a/connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaRelation.scala b/connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaRelation.scala
index 97b866067ea8..b77bb94aaf46 100644
--- a/connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaRelation.scala
+++ b/connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaRelation.scala
@@ -18,7 +18,7 @@
package org.apache.spark.sql.kafka010
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.TOPIC_PARTITIONS
+import org.apache.spark.internal.LogKeys.TOPIC_PARTITIONS
import org.apache.spark.internal.config.Network.NETWORK_TIMEOUT
import org.apache.spark.rdd.RDD
import org.apache.spark.sql.{Row, SQLContext}
diff --git a/connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSink.scala b/connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSink.scala
index 5a75682f54f9..fb473e71d5a7 100644
--- a/connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSink.scala
+++ b/connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSink.scala
@@ -20,7 +20,7 @@ package org.apache.spark.sql.kafka010
import java.{util => ju}
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.BATCH_ID
+import org.apache.spark.internal.LogKeys.BATCH_ID
import org.apache.spark.sql.DataFrame
import org.apache.spark.sql.execution.streaming.Sink
diff --git a/connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSource.scala b/connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSource.scala
index b0ab469690e2..d43b22d9de92 100644
--- a/connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSource.scala
+++ b/connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSource.scala
@@ -23,7 +23,7 @@ import org.apache.kafka.common.TopicPartition
import org.apache.spark.SparkContext
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{ERROR, FROM_OFFSET, OFFSETS, TIP, TOPIC_PARTITIONS, UNTIL_OFFSET}
+import org.apache.spark.internal.LogKeys.{ERROR, FROM_OFFSET, OFFSETS, TIP, TOPIC_PARTITIONS, UNTIL_OFFSET}
import org.apache.spark.internal.config.Network.NETWORK_TIMEOUT
import org.apache.spark.scheduler.ExecutorCacheTaskLocation
import org.apache.spark.sql._
diff --git a/connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSourceRDD.scala b/connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSourceRDD.scala
index 4eb73e6d39f0..bc7f8b6b44f9 100644
--- a/connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSourceRDD.scala
+++ b/connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSourceRDD.scala
@@ -22,7 +22,7 @@ import java.{util => ju}
import org.apache.kafka.clients.consumer.ConsumerRecord
import org.apache.spark.{Partition, SparkContext, TaskContext}
-import org.apache.spark.internal.LogKey.{FROM_OFFSET, PARTITION_ID, TOPIC}
+import org.apache.spark.internal.LogKeys.{FROM_OFFSET, PARTITION_ID, TOPIC}
import org.apache.spark.internal.MDC
import org.apache.spark.rdd.RDD
import org.apache.spark.sql.kafka010.consumer.KafkaDataConsumer
diff --git a/connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/consumer/FetchedDataPool.scala b/connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/consumer/FetchedDataPool.scala
index 981aa71bf947..9f68cb6fd088 100644
--- a/connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/consumer/FetchedDataPool.scala
+++ b/connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/consumer/FetchedDataPool.scala
@@ -27,7 +27,7 @@ import org.apache.kafka.clients.consumer.ConsumerRecord
import org.apache.spark.SparkConf
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{DATA, KEY}
+import org.apache.spark.internal.LogKeys.{DATA, KEY}
import org.apache.spark.sql.kafka010.{FETCHED_DATA_CACHE_EVICTOR_THREAD_RUN_INTERVAL, FETCHED_DATA_CACHE_TIMEOUT}
import org.apache.spark.sql.kafka010.consumer.KafkaDataConsumer.{AvailableOffsetRange, CacheKey, UNKNOWN_OFFSET}
import org.apache.spark.util.{Clock, SystemClock, ThreadUtils, Utils}
diff --git a/connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/consumer/KafkaDataConsumer.scala b/connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/consumer/KafkaDataConsumer.scala
index 72ceebb700d6..836c0fb13d4e 100644
--- a/connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/consumer/KafkaDataConsumer.scala
+++ b/connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/consumer/KafkaDataConsumer.scala
@@ -31,7 +31,7 @@ import org.apache.kafka.common.TopicPartition
import org.apache.spark.{SparkEnv, TaskContext}
import org.apache.spark.deploy.security.HadoopDelegationTokenManager
import org.apache.spark.internal.{Logging, MDC, MessageWithContext}
-import org.apache.spark.internal.LogKey._
+import org.apache.spark.internal.LogKeys._
import org.apache.spark.kafka010.{KafkaConfigUpdater, KafkaTokenUtil}
import org.apache.spark.sql.catalyst.util.DateTimeConstants.NANOS_PER_MILLIS
import org.apache.spark.sql.kafka010.KafkaExceptions
diff --git a/connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/producer/CachedKafkaProducer.scala b/connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/producer/CachedKafkaProducer.scala
index afd426694d7b..c3457cf8982d 100644
--- a/connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/producer/CachedKafkaProducer.scala
+++ b/connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/producer/CachedKafkaProducer.scala
@@ -24,7 +24,7 @@ import scala.util.control.NonFatal
import org.apache.kafka.clients.producer.KafkaProducer
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.PRODUCER_ID
+import org.apache.spark.internal.LogKeys.PRODUCER_ID
private[kafka010] class CachedKafkaProducer(
val cacheKey: Seq[(String, Object)],
diff --git a/connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/producer/InternalKafkaProducerPool.scala b/connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/producer/InternalKafkaProducerPool.scala
index f35023d744b6..79e0a91dd896 100644
--- a/connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/producer/InternalKafkaProducerPool.scala
+++ b/connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/producer/InternalKafkaProducerPool.scala
@@ -28,7 +28,7 @@ import org.apache.kafka.clients.producer.KafkaProducer
import org.apache.spark.{SparkConf, SparkEnv}
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.PRODUCER_ID
+import org.apache.spark.internal.LogKeys.PRODUCER_ID
import org.apache.spark.kafka010.{KafkaConfigUpdater, KafkaRedactionUtil}
import org.apache.spark.sql.kafka010.{PRODUCER_CACHE_EVICTOR_THREAD_RUN_INTERVAL, PRODUCER_CACHE_TIMEOUT}
import org.apache.spark.util.{Clock, ShutdownHookManager, SystemClock, ThreadUtils, Utils}
diff --git a/connector/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaTestUtils.scala b/connector/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaTestUtils.scala
index 068e3423cd26..0737658e6525 100644
--- a/connector/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaTestUtils.scala
+++ b/connector/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaTestUtils.scala
@@ -52,7 +52,7 @@ import org.scalatest.time.SpanSugar._
import org.apache.spark.{SparkConf, SparkException}
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey
+import org.apache.spark.internal.LogKeys
import org.apache.spark.kafka010.KafkaTokenUtil
import org.apache.spark.util.{SecurityUtils, ShutdownHookManager, Utils}
import org.apache.spark.util.ArrayImplicits._
@@ -70,7 +70,7 @@ class KafkaTestUtils(
private val JAVA_AUTH_CONFIG = "java.security.auth.login.config"
private val localHostNameForURI = Utils.localHostNameForURI()
- logInfo(log"Local host name is ${MDC(LogKey.URI, localHostNameForURI)}")
+ logInfo(log"Local host name is ${MDC(LogKeys.URI, localHostNameForURI)}")
// MiniKDC uses canonical host name on host part, hence we need to provide canonical host name
// on the 'host' part of the principal.
@@ -333,7 +333,7 @@ class KafkaTestUtils(
Utils.deleteRecursively(new File(f))
} catch {
case e: IOException if Utils.isWindows =>
- logWarning(log"${MDC(LogKey.ERROR, e.getMessage)}")
+ logWarning(log"${MDC(LogKeys.ERROR, e.getMessage)}")
}
}
@@ -654,13 +654,13 @@ class KafkaTestUtils(
Utils.deleteRecursively(snapshotDir)
} catch {
case e: IOException if Utils.isWindows =>
- logWarning(log"${MDC(LogKey.ERROR, e.getMessage)}")
+ logWarning(log"${MDC(LogKeys.ERROR, e.getMessage)}")
}
try {
Utils.deleteRecursively(logDir)
} catch {
case e: IOException if Utils.isWindows =>
- logWarning(log"${MDC(LogKey.ERROR, e.getMessage)}")
+ logWarning(log"${MDC(LogKeys.ERROR, e.getMessage)}")
}
System.clearProperty(ZOOKEEPER_AUTH_PROVIDER)
}
diff --git a/connector/kafka-0-10-token-provider/src/main/scala/org/apache/spark/kafka010/KafkaDelegationTokenProvider.scala b/connector/kafka-0-10-token-provider/src/main/scala/org/apache/spark/kafka010/KafkaDelegationTokenProvider.scala
index d0bcf90babc1..3616f93659fb 100644
--- a/connector/kafka-0-10-token-provider/src/main/scala/org/apache/spark/kafka010/KafkaDelegationTokenProvider.scala
+++ b/connector/kafka-0-10-token-provider/src/main/scala/org/apache/spark/kafka010/KafkaDelegationTokenProvider.scala
@@ -25,7 +25,7 @@ import org.apache.kafka.common.security.auth.SecurityProtocol.{SASL_PLAINTEXT, S
import org.apache.spark.SparkConf
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{CLUSTER_ID, SERVICE_NAME}
+import org.apache.spark.internal.LogKeys.{CLUSTER_ID, SERVICE_NAME}
import org.apache.spark.security.HadoopDelegationTokenProvider
private[spark] class KafkaDelegationTokenProvider
diff --git a/connector/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/ConsumerStrategy.scala b/connector/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/ConsumerStrategy.scala
index 2bc2acf9aaf9..2320f1908da5 100644
--- a/connector/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/ConsumerStrategy.scala
+++ b/connector/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/ConsumerStrategy.scala
@@ -26,7 +26,7 @@ import org.apache.kafka.clients.consumer._
import org.apache.kafka.common.TopicPartition
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.CONFIG
+import org.apache.spark.internal.LogKeys.CONFIG
import org.apache.spark.kafka010.KafkaConfigUpdater
/**
diff --git a/connector/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/DirectKafkaInputDStream.scala b/connector/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/DirectKafkaInputDStream.scala
index 86ee20849626..cefaa3de182a 100644
--- a/connector/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/DirectKafkaInputDStream.scala
+++ b/connector/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/DirectKafkaInputDStream.scala
@@ -28,7 +28,7 @@ import org.apache.kafka.clients.consumer._
import org.apache.kafka.common.TopicPartition
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{OFFSET, TIME, TOPIC_PARTITION, TOPIC_PARTITION_OFFSET_RANGE}
+import org.apache.spark.internal.LogKeys.{OFFSET, TIME, TOPIC_PARTITION, TOPIC_PARTITION_OFFSET_RANGE}
import org.apache.spark.storage.StorageLevel
import org.apache.spark.streaming.{StreamingContext, Time}
import org.apache.spark.streaming.dstream._
diff --git a/connector/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/KafkaDataConsumer.scala b/connector/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/KafkaDataConsumer.scala
index 91df53c9e06b..75b046430ef5 100644
--- a/connector/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/KafkaDataConsumer.scala
+++ b/connector/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/KafkaDataConsumer.scala
@@ -27,7 +27,7 @@ import org.apache.kafka.common.{KafkaException, TopicPartition}
import org.apache.spark.TaskContext
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey._
+import org.apache.spark.internal.LogKeys._
import org.apache.spark.kafka010.KafkaConfigUpdater
private[kafka010] sealed trait KafkaDataConsumer[K, V] {
diff --git a/connector/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/KafkaRDD.scala b/connector/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/KafkaRDD.scala
index 5bc89864cf0a..263703476657 100644
--- a/connector/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/KafkaRDD.scala
+++ b/connector/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/KafkaRDD.scala
@@ -24,7 +24,7 @@ import org.apache.kafka.common.TopicPartition
import org.apache.spark.{Partition, SparkContext, TaskContext}
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{FROM_OFFSET, PARTITION_ID, TOPIC, UNTIL_OFFSET}
+import org.apache.spark.internal.LogKeys.{FROM_OFFSET, PARTITION_ID, TOPIC, UNTIL_OFFSET}
import org.apache.spark.internal.config.Network._
import org.apache.spark.partial.{BoundedDouble, PartialResult}
import org.apache.spark.rdd.RDD
diff --git a/connector/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/KafkaUtils.scala b/connector/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/KafkaUtils.scala
index f3e4c45b3aa9..d15e5e25f561 100644
--- a/connector/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/KafkaUtils.scala
+++ b/connector/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/KafkaUtils.scala
@@ -25,7 +25,7 @@ import org.apache.kafka.common.TopicPartition
import org.apache.spark.SparkContext
import org.apache.spark.api.java.{ JavaRDD, JavaSparkContext }
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{CONFIG, GROUP_ID}
+import org.apache.spark.internal.LogKeys.{CONFIG, GROUP_ID}
import org.apache.spark.rdd.RDD
import org.apache.spark.streaming.StreamingContext
import org.apache.spark.streaming.api.java.{ JavaInputDStream, JavaStreamingContext }
diff --git a/connector/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisBackedBlockRDD.scala b/connector/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisBackedBlockRDD.scala
index 6b0c091534b7..5a2f904f2c65 100644
--- a/connector/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisBackedBlockRDD.scala
+++ b/connector/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisBackedBlockRDD.scala
@@ -30,7 +30,7 @@ import com.amazonaws.services.kinesis.model._
import org.apache.spark._
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{ERROR, RETRY_COUNT}
+import org.apache.spark.internal.LogKeys.{ERROR, RETRY_COUNT}
import org.apache.spark.rdd.{BlockRDD, BlockRDDPartition}
import org.apache.spark.storage.BlockId
import org.apache.spark.util.NextIterator
diff --git a/connector/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisCheckpointer.scala b/connector/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisCheckpointer.scala
index f6740999ce1c..7e5a170a870c 100644
--- a/connector/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisCheckpointer.scala
+++ b/connector/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisCheckpointer.scala
@@ -23,7 +23,7 @@ import scala.util.control.NonFatal
import com.amazonaws.services.kinesis.clientlibrary.interfaces.IRecordProcessorCheckpointer
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{SHARD_ID, WORKER_URL}
+import org.apache.spark.internal.LogKeys.{SHARD_ID, WORKER_URL}
import org.apache.spark.streaming.Duration
import org.apache.spark.streaming.util.RecurringTimer
import org.apache.spark.util.{Clock, SystemClock}
diff --git a/connector/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisReceiver.scala b/connector/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisReceiver.scala
index 47b03c2b7537..953817e625e4 100644
--- a/connector/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisReceiver.scala
+++ b/connector/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisReceiver.scala
@@ -29,7 +29,7 @@ import com.amazonaws.services.kinesis.metrics.interfaces.MetricsLevel
import com.amazonaws.services.kinesis.model.Record
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.WORKER_URL
+import org.apache.spark.internal.LogKeys.WORKER_URL
import org.apache.spark.storage.{StorageLevel, StreamBlockId}
import org.apache.spark.streaming.Duration
import org.apache.spark.streaming.kinesis.KinesisInitialPositions.AtTimestamp
diff --git a/connector/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisRecordProcessor.scala b/connector/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisRecordProcessor.scala
index 8424dde7d9c4..aaafb3215d03 100644
--- a/connector/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisRecordProcessor.scala
+++ b/connector/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisRecordProcessor.scala
@@ -27,7 +27,7 @@ import com.amazonaws.services.kinesis.clientlibrary.lib.worker.ShutdownReason
import com.amazonaws.services.kinesis.model.Record
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{REASON, RETRY_INTERVAL, SHARD_ID, WORKER_URL}
+import org.apache.spark.internal.LogKeys.{REASON, RETRY_INTERVAL, SHARD_ID, WORKER_URL}
/**
* Kinesis-specific implementation of the Kinesis Client Library (KCL) IRecordProcessor.
diff --git a/connector/kinesis-asl/src/test/scala/org/apache/spark/streaming/kinesis/KinesisTestUtils.scala b/connector/kinesis-asl/src/test/scala/org/apache/spark/streaming/kinesis/KinesisTestUtils.scala
index cd4c61396a12..652822c5fdc9 100644
--- a/connector/kinesis-asl/src/test/scala/org/apache/spark/streaming/kinesis/KinesisTestUtils.scala
+++ b/connector/kinesis-asl/src/test/scala/org/apache/spark/streaming/kinesis/KinesisTestUtils.scala
@@ -34,7 +34,7 @@ import com.amazonaws.services.kinesis.{AmazonKinesis, AmazonKinesisClient}
import com.amazonaws.services.kinesis.model._
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{STREAM_NAME, TABLE_NAME}
+import org.apache.spark.internal.LogKeys.{STREAM_NAME, TABLE_NAME}
/**
* Shared utility methods for performing Kinesis tests that actually transfer data.
diff --git a/connector/profiler/src/main/scala/org/apache/spark/executor/profiler/ExecutorJVMProfiler.scala b/connector/profiler/src/main/scala/org/apache/spark/executor/profiler/ExecutorJVMProfiler.scala
index 15ffbbd9d730..4c3f467fa15b 100644
--- a/connector/profiler/src/main/scala/org/apache/spark/executor/profiler/ExecutorJVMProfiler.scala
+++ b/connector/profiler/src/main/scala/org/apache/spark/executor/profiler/ExecutorJVMProfiler.scala
@@ -25,7 +25,7 @@ import org.apache.hadoop.fs.{FileSystem, FSDataOutputStream, Path}
import org.apache.spark.SparkConf
import org.apache.spark.deploy.SparkHadoopUtil
-import org.apache.spark.internal.LogKey.PATH
+import org.apache.spark.internal.LogKeys.PATH
import org.apache.spark.internal.{Logging, MDC}
import org.apache.spark.util.ThreadUtils
diff --git a/connector/profiler/src/main/scala/org/apache/spark/executor/profiler/ExecutorProfilerPlugin.scala b/connector/profiler/src/main/scala/org/apache/spark/executor/profiler/ExecutorProfilerPlugin.scala
index fb9abfe59aa7..ff753f04868f 100644
--- a/connector/profiler/src/main/scala/org/apache/spark/executor/profiler/ExecutorProfilerPlugin.scala
+++ b/connector/profiler/src/main/scala/org/apache/spark/executor/profiler/ExecutorProfilerPlugin.scala
@@ -23,7 +23,7 @@ import scala.util.Random
import org.apache.spark.SparkConf
import org.apache.spark.api.plugin.{DriverPlugin, ExecutorPlugin, PluginContext, SparkPlugin}
-import org.apache.spark.internal.LogKey.EXECUTOR_ID
+import org.apache.spark.internal.LogKeys.EXECUTOR_ID
import org.apache.spark.internal.{Logging, MDC}
diff --git a/core/src/main/scala/org/apache/spark/ContextCleaner.scala b/core/src/main/scala/org/apache/spark/ContextCleaner.scala
index c16a84c13187..fb56389cde77 100644
--- a/core/src/main/scala/org/apache/spark/ContextCleaner.scala
+++ b/core/src/main/scala/org/apache/spark/ContextCleaner.scala
@@ -25,7 +25,7 @@ import scala.jdk.CollectionConverters._
import org.apache.spark.broadcast.Broadcast
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{ACCUMULATOR_ID, BROADCAST_ID, LISTENER, RDD_ID, SHUFFLE_ID}
+import org.apache.spark.internal.LogKeys.{ACCUMULATOR_ID, BROADCAST_ID, LISTENER, RDD_ID, SHUFFLE_ID}
import org.apache.spark.internal.config._
import org.apache.spark.rdd.{RDD, ReliableRDDCheckpointData}
import org.apache.spark.scheduler.SparkListener
diff --git a/core/src/main/scala/org/apache/spark/MapOutputTracker.scala b/core/src/main/scala/org/apache/spark/MapOutputTracker.scala
index 48569eb71379..a323dc9fcb47 100644
--- a/core/src/main/scala/org/apache/spark/MapOutputTracker.scala
+++ b/core/src/main/scala/org/apache/spark/MapOutputTracker.scala
@@ -35,7 +35,7 @@ import org.roaringbitmap.RoaringBitmap
import org.apache.spark.broadcast.{Broadcast, BroadcastManager}
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey._
+import org.apache.spark.internal.LogKeys._
import org.apache.spark.internal.config._
import org.apache.spark.io.CompressionCodec
import org.apache.spark.rpc.{RpcCallContext, RpcEndpoint, RpcEndpointRef, RpcEnv}
diff --git a/core/src/main/scala/org/apache/spark/SparkContext.scala b/core/src/main/scala/org/apache/spark/SparkContext.scala
index 5e231544e249..b1c640dc935a 100644
--- a/core/src/main/scala/org/apache/spark/SparkContext.scala
+++ b/core/src/main/scala/org/apache/spark/SparkContext.scala
@@ -46,7 +46,7 @@ import org.apache.spark.errors.SparkCoreErrors
import org.apache.spark.executor.{Executor, ExecutorMetrics, ExecutorMetricsSource}
import org.apache.spark.input.{FixedLengthBinaryInputFormat, PortableDataStream, StreamInputFormat, WholeTextFileInputFormat}
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey
+import org.apache.spark.internal.LogKeys
import org.apache.spark.internal.config._
import org.apache.spark.internal.config.Tests._
import org.apache.spark.internal.config.UI._
@@ -755,7 +755,7 @@ class SparkContext(config: SparkConf) extends Logging {
} catch {
case e: Exception =>
logError(
- log"Exception getting thread dump from executor ${MDC(LogKey.EXECUTOR_ID, executorId)}",
+ log"Exception getting thread dump from executor ${MDC(LogKeys.EXECUTOR_ID, executorId)}",
e)
None
}
@@ -788,7 +788,7 @@ class SparkContext(config: SparkConf) extends Logging {
case e: Exception =>
logError(
log"Exception getting heap histogram from " +
- log"executor ${MDC(LogKey.EXECUTOR_ID, executorId)}", e)
+ log"executor ${MDC(LogKeys.EXECUTOR_ID, executorId)}", e)
None
}
}
@@ -2150,7 +2150,7 @@ class SparkContext(config: SparkConf) extends Logging {
Seq(env.rpcEnv.fileServer.addJar(file))
} catch {
case NonFatal(e) =>
- logError(log"Failed to add ${MDC(LogKey.PATH, path)} to Spark environment", e)
+ logError(log"Failed to add ${MDC(LogKeys.PATH, path)} to Spark environment", e)
Nil
}
}
@@ -2171,7 +2171,7 @@ class SparkContext(config: SparkConf) extends Logging {
Seq(path)
} catch {
case NonFatal(e) =>
- logError(log"Failed to add ${MDC(LogKey.PATH, path)} to Spark environment", e)
+ logError(log"Failed to add ${MDC(LogKeys.PATH, path)} to Spark environment", e)
Nil
}
} else {
diff --git a/core/src/main/scala/org/apache/spark/TaskContextImpl.scala b/core/src/main/scala/org/apache/spark/TaskContextImpl.scala
index e433cc10ae73..8167952d6b87 100644
--- a/core/src/main/scala/org/apache/spark/TaskContextImpl.scala
+++ b/core/src/main/scala/org/apache/spark/TaskContextImpl.scala
@@ -25,7 +25,7 @@ import scala.jdk.CollectionConverters._
import org.apache.spark.executor.TaskMetrics
import org.apache.spark.internal.{config, Logging, MDC}
-import org.apache.spark.internal.LogKey.LISTENER
+import org.apache.spark.internal.LogKeys.LISTENER
import org.apache.spark.memory.TaskMemoryManager
import org.apache.spark.metrics.MetricsSystem
import org.apache.spark.metrics.source.Source
diff --git a/core/src/main/scala/org/apache/spark/api/python/PythonGatewayServer.scala b/core/src/main/scala/org/apache/spark/api/python/PythonGatewayServer.scala
index fb0584b45846..7737822f2af2 100644
--- a/core/src/main/scala/org/apache/spark/api/python/PythonGatewayServer.scala
+++ b/core/src/main/scala/org/apache/spark/api/python/PythonGatewayServer.scala
@@ -24,7 +24,7 @@ import java.nio.file.Files
import org.apache.spark.SparkConf
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{CLASS_NAME, PATH}
+import org.apache.spark.internal.LogKeys.{CLASS_NAME, PATH}
/**
* Process that starts a Py4J server on an ephemeral port.
diff --git a/core/src/main/scala/org/apache/spark/api/python/PythonHadoopUtil.scala b/core/src/main/scala/org/apache/spark/api/python/PythonHadoopUtil.scala
index 4f7c5bc0b0c0..46dab16b8276 100644
--- a/core/src/main/scala/org/apache/spark/api/python/PythonHadoopUtil.scala
+++ b/core/src/main/scala/org/apache/spark/api/python/PythonHadoopUtil.scala
@@ -26,7 +26,7 @@ import org.apache.hadoop.io._
import org.apache.spark.SparkException
import org.apache.spark.broadcast.Broadcast
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.CLASS_NAME
+import org.apache.spark.internal.LogKeys.CLASS_NAME
import org.apache.spark.rdd.RDD
import org.apache.spark.util.{SerializableConfiguration, Utils}
diff --git a/core/src/main/scala/org/apache/spark/api/r/RBackendHandler.scala b/core/src/main/scala/org/apache/spark/api/r/RBackendHandler.scala
index 1a05c8f35b7f..1922c22a7133 100644
--- a/core/src/main/scala/org/apache/spark/api/r/RBackendHandler.scala
+++ b/core/src/main/scala/org/apache/spark/api/r/RBackendHandler.scala
@@ -27,7 +27,7 @@ import io.netty.handler.timeout.ReadTimeoutException
import org.apache.spark.{SparkConf, SparkEnv}
import org.apache.spark.api.r.SerDe._
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{METHOD_NAME, OBJECT_ID}
+import org.apache.spark.internal.LogKeys.{METHOD_NAME, OBJECT_ID}
import org.apache.spark.internal.config.R._
import org.apache.spark.util.{ThreadUtils, Utils}
import org.apache.spark.util.ArrayImplicits._
diff --git a/core/src/main/scala/org/apache/spark/broadcast/TorrentBroadcast.scala b/core/src/main/scala/org/apache/spark/broadcast/TorrentBroadcast.scala
index b6ba9bbf29f3..a8b514a3a537 100644
--- a/core/src/main/scala/org/apache/spark/broadcast/TorrentBroadcast.scala
+++ b/core/src/main/scala/org/apache/spark/broadcast/TorrentBroadcast.scala
@@ -28,7 +28,7 @@ import scala.util.Random
import org.apache.spark._
import org.apache.spark.internal.{config, Logging, MDC}
-import org.apache.spark.internal.LogKey.BROADCAST_ID
+import org.apache.spark.internal.LogKeys.BROADCAST_ID
import org.apache.spark.io.CompressionCodec
import org.apache.spark.serializer.Serializer
import org.apache.spark.storage._
diff --git a/core/src/main/scala/org/apache/spark/deploy/Client.scala b/core/src/main/scala/org/apache/spark/deploy/Client.scala
index d38f94fd1ac2..4b0e12861f05 100644
--- a/core/src/main/scala/org/apache/spark/deploy/Client.scala
+++ b/core/src/main/scala/org/apache/spark/deploy/Client.scala
@@ -32,7 +32,7 @@ import org.apache.spark.deploy.DeployMessages._
import org.apache.spark.deploy.master.{DriverState, Master}
import org.apache.spark.deploy.master.DriverState.DriverState
import org.apache.spark.internal.{config, Logging, MDC}
-import org.apache.spark.internal.LogKey.{DRIVER_ID, ERROR, HOST_PORT}
+import org.apache.spark.internal.LogKeys.{DRIVER_ID, ERROR, HOST_PORT}
import org.apache.spark.internal.config.Network.RPC_ASK_TIMEOUT
import org.apache.spark.resource.ResourceUtils
import org.apache.spark.rpc.{RpcAddress, RpcEndpointRef, RpcEnv, ThreadSafeRpcEndpoint}
diff --git a/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala b/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
index c0df74f8d0cc..77b73fcfac1d 100644
--- a/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
+++ b/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
@@ -40,7 +40,7 @@ import org.apache.spark._
import org.apache.spark.api.r.RUtils
import org.apache.spark.deploy.rest._
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.CLASS_NAME
+import org.apache.spark.internal.LogKeys.CLASS_NAME
import org.apache.spark.internal.config._
import org.apache.spark.internal.config.UI._
import org.apache.spark.launcher.SparkLauncher
diff --git a/core/src/main/scala/org/apache/spark/deploy/StandaloneResourceUtils.scala b/core/src/main/scala/org/apache/spark/deploy/StandaloneResourceUtils.scala
index d317d6449f29..7b98461b01ac 100644
--- a/core/src/main/scala/org/apache/spark/deploy/StandaloneResourceUtils.scala
+++ b/core/src/main/scala/org/apache/spark/deploy/StandaloneResourceUtils.scala
@@ -28,7 +28,7 @@ import org.json4s.jackson.JsonMethods.{compact, render}
import org.apache.spark.SparkException
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.COMPONENT
+import org.apache.spark.internal.LogKeys.COMPONENT
import org.apache.spark.resource.{ResourceAllocation, ResourceID, ResourceInformation, ResourceRequirement}
import org.apache.spark.util.ArrayImplicits._
import org.apache.spark.util.Utils
diff --git a/core/src/main/scala/org/apache/spark/deploy/Utils.scala b/core/src/main/scala/org/apache/spark/deploy/Utils.scala
index 4d2546cb808c..b3d871d75e6c 100644
--- a/core/src/main/scala/org/apache/spark/deploy/Utils.scala
+++ b/core/src/main/scala/org/apache/spark/deploy/Utils.scala
@@ -23,7 +23,7 @@ import jakarta.servlet.http.HttpServletRequest
import org.apache.spark.SparkConf
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{LOG_TYPE, PATH}
+import org.apache.spark.internal.LogKeys.{LOG_TYPE, PATH}
import org.apache.spark.ui.JettyUtils.createServletHandler
import org.apache.spark.ui.WebUI
import org.apache.spark.util.Utils.{getFileLength, offsetBytes}
diff --git a/core/src/main/scala/org/apache/spark/deploy/history/EventFilter.scala b/core/src/main/scala/org/apache/spark/deploy/history/EventFilter.scala
index 59e52e649498..05f109831499 100644
--- a/core/src/main/scala/org/apache/spark/deploy/history/EventFilter.scala
+++ b/core/src/main/scala/org/apache/spark/deploy/history/EventFilter.scala
@@ -24,7 +24,7 @@ import org.apache.hadoop.fs.{FileSystem, Path}
import org.apache.spark.deploy.history.EventFilter.FilterStatistics
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{LINE, LINE_NUM, PATH}
+import org.apache.spark.internal.LogKeys.{LINE, LINE_NUM, PATH}
import org.apache.spark.scheduler._
import org.apache.spark.util.{JsonProtocol, Utils}
diff --git a/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala b/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
index 98cbd7b3eba8..c1f5a279f2fb 100644
--- a/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
+++ b/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
@@ -38,7 +38,7 @@ import org.apache.hadoop.security.AccessControlException
import org.apache.spark.{SecurityManager, SparkConf, SparkException}
import org.apache.spark.deploy.SparkHadoopUtil
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.PATH
+import org.apache.spark.internal.LogKeys.PATH
import org.apache.spark.internal.config._
import org.apache.spark.internal.config.History._
import org.apache.spark.internal.config.Status._
diff --git a/core/src/main/scala/org/apache/spark/deploy/master/Master.scala b/core/src/main/scala/org/apache/spark/deploy/master/Master.scala
index e02d45105727..46be03fc82a3 100644
--- a/core/src/main/scala/org/apache/spark/deploy/master/Master.scala
+++ b/core/src/main/scala/org/apache/spark/deploy/master/Master.scala
@@ -33,7 +33,7 @@ import org.apache.spark.deploy.master.MasterMessages._
import org.apache.spark.deploy.master.ui.MasterWebUI
import org.apache.spark.deploy.rest.StandaloneRestServer
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{APP_DESC, APP_ID, EXECUTOR_ID, RETRY_COUNT}
+import org.apache.spark.internal.LogKeys.{APP_DESC, APP_ID, EXECUTOR_ID, RETRY_COUNT}
import org.apache.spark.internal.config._
import org.apache.spark.internal.config.Deploy._
import org.apache.spark.internal.config.Deploy.WorkerSelectionPolicy._
diff --git a/core/src/main/scala/org/apache/spark/deploy/rest/RestSubmissionClient.scala b/core/src/main/scala/org/apache/spark/deploy/rest/RestSubmissionClient.scala
index 9107b2f5528c..a6d128de164f 100644
--- a/core/src/main/scala/org/apache/spark/deploy/rest/RestSubmissionClient.scala
+++ b/core/src/main/scala/org/apache/spark/deploy/rest/RestSubmissionClient.scala
@@ -34,7 +34,7 @@ import jakarta.servlet.http.HttpServletResponse
import org.apache.spark.{SPARK_VERSION => sparkVersion, SparkConf, SparkException}
import org.apache.spark.deploy.SparkApplication
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{CLASS_NAME, ERROR, SUBMISSION_ID}
+import org.apache.spark.internal.LogKeys.{CLASS_NAME, ERROR, SUBMISSION_ID}
import org.apache.spark.util.Utils
/**
diff --git a/core/src/main/scala/org/apache/spark/deploy/security/HBaseDelegationTokenProvider.scala b/core/src/main/scala/org/apache/spark/deploy/security/HBaseDelegationTokenProvider.scala
index 1b2e41bc0a2e..1d13deed2af5 100644
--- a/core/src/main/scala/org/apache/spark/deploy/security/HBaseDelegationTokenProvider.scala
+++ b/core/src/main/scala/org/apache/spark/deploy/security/HBaseDelegationTokenProvider.scala
@@ -28,7 +28,7 @@ import org.apache.hadoop.security.token.{Token, TokenIdentifier}
import org.apache.spark.SparkConf
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.SERVICE_NAME
+import org.apache.spark.internal.LogKeys.SERVICE_NAME
import org.apache.spark.security.HadoopDelegationTokenProvider
import org.apache.spark.util.Utils
diff --git a/core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala b/core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala
index 0659c26fd15b..302da3473809 100755
--- a/core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala
+++ b/core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala
@@ -38,7 +38,7 @@ import org.apache.spark.deploy.StandaloneResourceUtils._
import org.apache.spark.deploy.master.{DriverState, Master}
import org.apache.spark.deploy.worker.ui.WorkerWebUI
import org.apache.spark.internal.{config, Logging, MDC}
-import org.apache.spark.internal.LogKey._
+import org.apache.spark.internal.LogKeys._
import org.apache.spark.internal.config.Tests.IS_TESTING
import org.apache.spark.internal.config.UI._
import org.apache.spark.internal.config.Worker._
diff --git a/core/src/main/scala/org/apache/spark/deploy/worker/WorkerWatcher.scala b/core/src/main/scala/org/apache/spark/deploy/worker/WorkerWatcher.scala
index 248899411274..be4adf15f9d6 100644
--- a/core/src/main/scala/org/apache/spark/deploy/worker/WorkerWatcher.scala
+++ b/core/src/main/scala/org/apache/spark/deploy/worker/WorkerWatcher.scala
@@ -20,7 +20,7 @@ package org.apache.spark.deploy.worker
import java.util.concurrent.atomic.AtomicBoolean
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.WORKER_URL
+import org.apache.spark.internal.LogKeys.WORKER_URL
import org.apache.spark.rpc._
/**
diff --git a/core/src/main/scala/org/apache/spark/deploy/worker/ui/LogPage.scala b/core/src/main/scala/org/apache/spark/deploy/worker/ui/LogPage.scala
index 006a388e98b5..defce5acc616 100644
--- a/core/src/main/scala/org/apache/spark/deploy/worker/ui/LogPage.scala
+++ b/core/src/main/scala/org/apache/spark/deploy/worker/ui/LogPage.scala
@@ -24,7 +24,7 @@ import scala.xml.{Node, Unparsed}
import jakarta.servlet.http.HttpServletRequest
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{LOG_TYPE, PATH}
+import org.apache.spark.internal.LogKeys.{LOG_TYPE, PATH}
import org.apache.spark.ui.{UIUtils, WebUIPage}
import org.apache.spark.util.Utils
import org.apache.spark.util.logging.RollingFileAppender
diff --git a/core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala b/core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala
index 8488333ec3ce..bead4d8dd2a5 100644
--- a/core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala
+++ b/core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala
@@ -32,7 +32,7 @@ import org.apache.spark.TaskState.TaskState
import org.apache.spark.deploy.SparkHadoopUtil
import org.apache.spark.deploy.worker.WorkerWatcher
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{CONFIG, CONFIG2, REASON}
+import org.apache.spark.internal.LogKeys.{CONFIG, CONFIG2, REASON}
import org.apache.spark.internal.config._
import org.apache.spark.network.netty.SparkTransportConf
import org.apache.spark.network.util.NettyUtils
diff --git a/core/src/main/scala/org/apache/spark/executor/Executor.scala b/core/src/main/scala/org/apache/spark/executor/Executor.scala
index a7657cd78cd9..ea24dd57b43f 100644
--- a/core/src/main/scala/org/apache/spark/executor/Executor.scala
+++ b/core/src/main/scala/org/apache/spark/executor/Executor.scala
@@ -41,7 +41,7 @@ import org.slf4j.MDC
import org.apache.spark._
import org.apache.spark.deploy.SparkHadoopUtil
import org.apache.spark.internal.{Logging, MDC => LogMDC}
-import org.apache.spark.internal.LogKey.{CLASS_NAME, ERROR, MAX_ATTEMPTS, TASK_ID, TASK_NAME, TIMEOUT}
+import org.apache.spark.internal.LogKeys.{CLASS_NAME, ERROR, MAX_ATTEMPTS, TASK_ID, TASK_NAME, TIMEOUT}
import org.apache.spark.internal.config._
import org.apache.spark.internal.plugin.PluginContainer
import org.apache.spark.memory.{SparkOutOfMemoryError, TaskMemoryManager}
diff --git a/core/src/main/scala/org/apache/spark/executor/ExecutorClassLoader.scala b/core/src/main/scala/org/apache/spark/executor/ExecutorClassLoader.scala
index c7047ddd278b..48d7f150ad9b 100644
--- a/core/src/main/scala/org/apache/spark/executor/ExecutorClassLoader.scala
+++ b/core/src/main/scala/org/apache/spark/executor/ExecutorClassLoader.scala
@@ -30,7 +30,7 @@ import org.apache.xbean.asm9.Opcodes._
import org.apache.spark.{SparkConf, SparkEnv}
import org.apache.spark.deploy.SparkHadoopUtil
-import org.apache.spark.internal.{Logging, LogKey, MDC}
+import org.apache.spark.internal.{Logging, LogKeys, MDC}
import org.apache.spark.util.ParentClassLoader
/**
@@ -183,8 +183,8 @@ class ExecutorClassLoader(
None
case e: Exception =>
// Something bad happened while checking if the class exists
- logError(log"Failed to check existence of class ${MDC(LogKey.CLASS_NAME, name)} " +
- log"on REPL class server at ${MDC(LogKey.URI, uri)}", e)
+ logError(log"Failed to check existence of class ${MDC(LogKeys.CLASS_NAME, name)} " +
+ log"on REPL class server at ${MDC(LogKeys.URI, uri)}", e)
if (userClassPathFirst) {
// Allow to try to load from "parentLoader"
None
diff --git a/core/src/main/scala/org/apache/spark/internal/io/SparkHadoopWriter.scala b/core/src/main/scala/org/apache/spark/internal/io/SparkHadoopWriter.scala
index 95ea814042d3..52529ff1d4e9 100644
--- a/core/src/main/scala/org/apache/spark/internal/io/SparkHadoopWriter.scala
+++ b/core/src/main/scala/org/apache/spark/internal/io/SparkHadoopWriter.scala
@@ -33,7 +33,7 @@ import org.apache.hadoop.mapreduce.task.{TaskAttemptContextImpl => NewTaskAttemp
import org.apache.spark.{SerializableWritable, SparkConf, SparkException, TaskContext}
import org.apache.spark.deploy.SparkHadoopUtil
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{JOB_ID, TASK_ATTEMPT_ID}
+import org.apache.spark.internal.LogKeys.{JOB_ID, TASK_ATTEMPT_ID}
import org.apache.spark.internal.io.FileCommitProtocol.TaskCommitMessage
import org.apache.spark.rdd.{HadoopRDD, RDD}
import org.apache.spark.util.{SerializableConfiguration, SerializableJobConf, Utils}
diff --git a/core/src/main/scala/org/apache/spark/mapred/SparkHadoopMapRedUtil.scala b/core/src/main/scala/org/apache/spark/mapred/SparkHadoopMapRedUtil.scala
index c68999f34079..a5395aa01a9b 100644
--- a/core/src/main/scala/org/apache/spark/mapred/SparkHadoopMapRedUtil.scala
+++ b/core/src/main/scala/org/apache/spark/mapred/SparkHadoopMapRedUtil.scala
@@ -25,7 +25,7 @@ import org.apache.hadoop.mapreduce.{OutputCommitter => MapReduceOutputCommitter}
import org.apache.spark.{SparkEnv, TaskContext}
import org.apache.spark.executor.CommitDeniedException
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.TASK_ATTEMPT_ID
+import org.apache.spark.internal.LogKeys.TASK_ATTEMPT_ID
import org.apache.spark.util.Utils
object SparkHadoopMapRedUtil extends Logging {
diff --git a/core/src/main/scala/org/apache/spark/metrics/MetricsConfig.scala b/core/src/main/scala/org/apache/spark/metrics/MetricsConfig.scala
index 12df40c3476a..a845feeb67ff 100644
--- a/core/src/main/scala/org/apache/spark/metrics/MetricsConfig.scala
+++ b/core/src/main/scala/org/apache/spark/metrics/MetricsConfig.scala
@@ -26,7 +26,7 @@ import scala.util.matching.Regex
import org.apache.spark.SparkConf
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.PATH
+import org.apache.spark.internal.LogKeys.PATH
import org.apache.spark.internal.config.METRICS_CONF
import org.apache.spark.util.Utils
diff --git a/core/src/main/scala/org/apache/spark/metrics/MetricsSystem.scala b/core/src/main/scala/org/apache/spark/metrics/MetricsSystem.scala
index 555083bb65d2..6f80a5cd6a06 100644
--- a/core/src/main/scala/org/apache/spark/metrics/MetricsSystem.scala
+++ b/core/src/main/scala/org/apache/spark/metrics/MetricsSystem.scala
@@ -27,7 +27,7 @@ import org.eclipse.jetty.servlet.ServletContextHandler
import org.apache.spark.{SecurityManager, SparkConf}
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.CLASS_NAME
+import org.apache.spark.internal.LogKeys.CLASS_NAME
import org.apache.spark.internal.config._
import org.apache.spark.metrics.sink.{MetricsServlet, PrometheusServlet, Sink}
import org.apache.spark.metrics.source.{Source, StaticSources}
diff --git a/core/src/main/scala/org/apache/spark/rdd/PipedRDD.scala b/core/src/main/scala/org/apache/spark/rdd/PipedRDD.scala
index 127bdf6d9181..a806b72766c6 100644
--- a/core/src/main/scala/org/apache/spark/rdd/PipedRDD.scala
+++ b/core/src/main/scala/org/apache/spark/rdd/PipedRDD.scala
@@ -34,7 +34,7 @@ import scala.reflect.ClassTag
import org.apache.spark.{Partition, TaskContext}
import org.apache.spark.errors.SparkCoreErrors
-import org.apache.spark.internal.LogKey.{COMMAND, ERROR, PATH}
+import org.apache.spark.internal.LogKeys.{COMMAND, ERROR, PATH}
import org.apache.spark.internal.MDC
import org.apache.spark.util.Utils
diff --git a/core/src/main/scala/org/apache/spark/rpc/netty/Inbox.scala b/core/src/main/scala/org/apache/spark/rpc/netty/Inbox.scala
index b503c5a0f808..be503d8e564f 100644
--- a/core/src/main/scala/org/apache/spark/rpc/netty/Inbox.scala
+++ b/core/src/main/scala/org/apache/spark/rpc/netty/Inbox.scala
@@ -23,7 +23,7 @@ import scala.util.control.NonFatal
import org.apache.spark.SparkException
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.END_POINT
+import org.apache.spark.internal.LogKeys.END_POINT
import org.apache.spark.rpc.{RpcAddress, RpcEndpoint, ThreadSafeRpcEndpoint}
diff --git a/core/src/main/scala/org/apache/spark/rpc/netty/MessageLoop.scala b/core/src/main/scala/org/apache/spark/rpc/netty/MessageLoop.scala
index 2d94ed5d05e1..2fd1c6d7fe71 100644
--- a/core/src/main/scala/org/apache/spark/rpc/netty/MessageLoop.scala
+++ b/core/src/main/scala/org/apache/spark/rpc/netty/MessageLoop.scala
@@ -23,7 +23,7 @@ import scala.util.control.NonFatal
import org.apache.spark.{SparkConf, SparkContext}
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.ERROR
+import org.apache.spark.internal.LogKeys.ERROR
import org.apache.spark.internal.config.EXECUTOR_ID
import org.apache.spark.internal.config.Network._
import org.apache.spark.rpc.{IsolatedRpcEndpoint, RpcEndpoint}
diff --git a/core/src/main/scala/org/apache/spark/scheduler/AsyncEventQueue.scala b/core/src/main/scala/org/apache/spark/scheduler/AsyncEventQueue.scala
index 271fc9ac92ba..5184a76847d8 100644
--- a/core/src/main/scala/org/apache/spark/scheduler/AsyncEventQueue.scala
+++ b/core/src/main/scala/org/apache/spark/scheduler/AsyncEventQueue.scala
@@ -24,7 +24,7 @@ import com.codahale.metrics.{Gauge, Timer}
import org.apache.spark.{SparkConf, SparkContext}
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.EVENT_QUEUE
+import org.apache.spark.internal.LogKeys.EVENT_QUEUE
import org.apache.spark.internal.config._
import org.apache.spark.util.Utils
diff --git a/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala b/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
index 41cbd795b7e5..bc64c297324c 100644
--- a/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
+++ b/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
@@ -37,7 +37,7 @@ import org.apache.spark.broadcast.Broadcast
import org.apache.spark.errors.SparkCoreErrors
import org.apache.spark.executor.{ExecutorMetrics, TaskMetrics}
import org.apache.spark.internal.{config, Logging, MDC}
-import org.apache.spark.internal.LogKey.{ACCUMULATOR_ID, CLASS_NAME, JOB_ID, PARTITION_ID, STAGE_ID, TASK_ID}
+import org.apache.spark.internal.LogKeys.{ACCUMULATOR_ID, CLASS_NAME, JOB_ID, PARTITION_ID, STAGE_ID, TASK_ID}
import org.apache.spark.internal.config.{LEGACY_ABORT_STAGE_AFTER_KILL_TASKS, RDD_CACHE_VISIBILITY_TRACKING_ENABLED}
import org.apache.spark.internal.config.Tests.TEST_NO_STAGE_RETRY
import org.apache.spark.network.shuffle.{BlockStoreClient, MergeFinalizerListener}
diff --git a/core/src/main/scala/org/apache/spark/scheduler/HealthTracker.scala b/core/src/main/scala/org/apache/spark/scheduler/HealthTracker.scala
index aaa9e5bdd9e1..45ae1df156e1 100644
--- a/core/src/main/scala/org/apache/spark/scheduler/HealthTracker.scala
+++ b/core/src/main/scala/org/apache/spark/scheduler/HealthTracker.scala
@@ -23,7 +23,7 @@ import scala.collection.mutable.{ArrayBuffer, HashMap, HashSet}
import org.apache.spark.{ExecutorAllocationClient, SparkConf, SparkContext}
import org.apache.spark.internal.{config, Logging, MDC}
-import org.apache.spark.internal.LogKey.HOST
+import org.apache.spark.internal.LogKeys.HOST
import org.apache.spark.util.{Clock, SystemClock, Utils}
/**
diff --git a/core/src/main/scala/org/apache/spark/scheduler/LiveListenerBus.scala b/core/src/main/scala/org/apache/spark/scheduler/LiveListenerBus.scala
index bd0bff18ff57..7251eb2c86ea 100644
--- a/core/src/main/scala/org/apache/spark/scheduler/LiveListenerBus.scala
+++ b/core/src/main/scala/org/apache/spark/scheduler/LiveListenerBus.scala
@@ -31,7 +31,7 @@ import com.codahale.metrics.{Counter, MetricRegistry, Timer}
import org.apache.spark.{SparkConf, SparkContext}
import org.apache.spark.errors.SparkCoreErrors
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{CLASS_NAME, MAX_SIZE}
+import org.apache.spark.internal.LogKeys.{CLASS_NAME, MAX_SIZE}
import org.apache.spark.internal.config._
import org.apache.spark.metrics.MetricsSystem
import org.apache.spark.metrics.source.Source
diff --git a/core/src/main/scala/org/apache/spark/scheduler/ReplayListenerBus.scala b/core/src/main/scala/org/apache/spark/scheduler/ReplayListenerBus.scala
index 24c25d237794..f0267ad35ad2 100644
--- a/core/src/main/scala/org/apache/spark/scheduler/ReplayListenerBus.scala
+++ b/core/src/main/scala/org/apache/spark/scheduler/ReplayListenerBus.scala
@@ -25,7 +25,7 @@ import com.fasterxml.jackson.core.JsonParseException
import com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{LINE, LINE_NUM, PATH}
+import org.apache.spark.internal.LogKeys.{LINE, LINE_NUM, PATH}
import org.apache.spark.scheduler.ReplayListenerBus._
import org.apache.spark.util.JsonProtocol
diff --git a/core/src/main/scala/org/apache/spark/scheduler/SchedulableBuilder.scala b/core/src/main/scala/org/apache/spark/scheduler/SchedulableBuilder.scala
index 7e61dad3c141..c14018960267 100644
--- a/core/src/main/scala/org/apache/spark/scheduler/SchedulableBuilder.scala
+++ b/core/src/main/scala/org/apache/spark/scheduler/SchedulableBuilder.scala
@@ -27,7 +27,7 @@ import org.apache.hadoop.fs.Path
import org.apache.spark.SparkContext
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.PATH
+import org.apache.spark.internal.LogKeys.PATH
import org.apache.spark.internal.config.{SCHEDULER_ALLOCATION_FILE, SCHEDULER_MODE}
import org.apache.spark.scheduler.SchedulingMode.SchedulingMode
import org.apache.spark.util.Utils
diff --git a/core/src/main/scala/org/apache/spark/scheduler/TaskResultGetter.scala b/core/src/main/scala/org/apache/spark/scheduler/TaskResultGetter.scala
index e93bc0747349..97c539bb05a5 100644
--- a/core/src/main/scala/org/apache/spark/scheduler/TaskResultGetter.scala
+++ b/core/src/main/scala/org/apache/spark/scheduler/TaskResultGetter.scala
@@ -26,7 +26,7 @@ import scala.util.control.NonFatal
import org.apache.spark._
import org.apache.spark.TaskState.TaskState
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.CLASS_LOADER
+import org.apache.spark.internal.LogKeys.CLASS_LOADER
import org.apache.spark.serializer.{SerializerHelper, SerializerInstance}
import org.apache.spark.util.{LongAccumulator, ThreadUtils, Utils}
diff --git a/core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala b/core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala
index 17c44926d626..bc2afd32b8a9 100644
--- a/core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala
+++ b/core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala
@@ -33,8 +33,7 @@ import org.apache.spark.InternalAccumulator.{input, shuffleRead}
import org.apache.spark.TaskState.TaskState
import org.apache.spark.errors.SparkCoreErrors
import org.apache.spark.executor.ExecutorMetrics
-import org.apache.spark.internal.{config, Logging, LogKey, MDC}
-import org.apache.spark.internal.LogKey.{REASON, TASK_SET_NAME, TASK_STATE, TID}
+import org.apache.spark.internal.{config, Logging, LogKeys, MDC}
import org.apache.spark.internal.config._
import org.apache.spark.resource.ResourceProfile
import org.apache.spark.rpc.RpcEndpoint
@@ -428,7 +427,8 @@ private[spark] class TaskSchedulerImpl(
} catch {
case e: TaskNotSerializableException =>
// scalastyle:off line.size.limit
- logError(log"Resource offer failed, task set ${MDC(TASK_SET_NAME, taskSet.name)} was not serializable")
+ logError(log"Resource offer failed, task set " +
+ log"${MDC(LogKeys.TASK_SET_NAME, taskSet.name)} was not serializable")
// scalastyle:on
// Do not offer resources for this task, but don't throw an error to allow other
// task sets to be submitted.
@@ -807,9 +807,10 @@ private[spark] class TaskSchedulerImpl(
taskSet.taskInfos(tid).launchSucceeded()
}
case None =>
- logError(
- log"Ignoring update with state ${MDC(TASK_STATE, state)} for TID ${MDC(TID, tid)} because its task set is gone (this is " +
- log"likely the result of receiving duplicate task finished status updates) or its executor has been marked as failed.")
+ logError(log"Ignoring update with state ${MDC(LogKeys.TASK_STATE, state)} for " +
+ log"TID ${MDC(LogKeys.TID, tid)} because its task set is gone (this is " +
+ log"likely the result of receiving duplicate task finished status updates) or its " +
+ log"executor has been marked as failed.")
}
} catch {
case e: Exception => logError("Exception in statusUpdate", e)
@@ -1024,7 +1025,8 @@ private[spark] class TaskSchedulerImpl(
// one may be triggered by a dropped connection from the worker while another may be a
// report of executor termination. We produce log messages for both so we
// eventually report the termination reason.
- logError(log"Lost an executor ${MDC(LogKey.EXECUTOR_ID, executorId)} (already removed): ${MDC(REASON, reason)}")
+ logError(log"Lost an executor ${MDC(LogKeys.EXECUTOR_ID, executorId)} " +
+ log"(already removed): ${MDC(LogKeys.REASON, reason)}")
}
}
}
@@ -1052,7 +1054,8 @@ private[spark] class TaskSchedulerImpl(
logInfo(s"Executor $executorId on $hostPort is decommissioned" +
s"${getDecommissionDuration(executorId)}.")
case _ =>
- logError(log"Lost executor ${MDC(LogKey.EXECUTOR_ID, executorId)} on ${MDC(LogKey.HOST, hostPort)}: ${MDC(REASON, reason)}")
+ logError(log"Lost executor ${MDC(LogKeys.EXECUTOR_ID, executorId)} on " +
+ log"${MDC(LogKeys.HOST, hostPort)}: ${MDC(LogKeys.REASON, reason)}")
}
// return decommission duration in string or "" if decommission startTime not exists
diff --git a/core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala b/core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala
index 1418901e3442..1d86ebc0a734 100644
--- a/core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala
+++ b/core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala
@@ -31,7 +31,7 @@ import org.apache.spark.InternalAccumulator.{input, shuffleRead}
import org.apache.spark.TaskState.TaskState
import org.apache.spark.errors.SparkCoreErrors
import org.apache.spark.internal.{config, Logging, MDC}
-import org.apache.spark.internal.LogKey._
+import org.apache.spark.internal.LogKeys._
import org.apache.spark.internal.config._
import org.apache.spark.scheduler.SchedulingMode._
import org.apache.spark.util.{AccumulatorV2, Clock, LongAccumulator, SystemClock, Utils}
diff --git a/core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala b/core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
index 06cfb53e2ded..9ddc63b27537 100644
--- a/core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
+++ b/core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
@@ -33,7 +33,7 @@ import org.apache.spark.deploy.security.HadoopDelegationTokenManager
import org.apache.spark.errors.SparkCoreErrors
import org.apache.spark.executor.ExecutorLogUrlHandler
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.ERROR
+import org.apache.spark.internal.LogKeys.ERROR
import org.apache.spark.internal.config._
import org.apache.spark.internal.config.Network._
import org.apache.spark.resource.ResourceProfile
diff --git a/core/src/main/scala/org/apache/spark/scheduler/cluster/StandaloneSchedulerBackend.scala b/core/src/main/scala/org/apache/spark/scheduler/cluster/StandaloneSchedulerBackend.scala
index f92756105977..144ea1b31ca3 100644
--- a/core/src/main/scala/org/apache/spark/scheduler/cluster/StandaloneSchedulerBackend.scala
+++ b/core/src/main/scala/org/apache/spark/scheduler/cluster/StandaloneSchedulerBackend.scala
@@ -28,7 +28,7 @@ import org.apache.spark.deploy.{ApplicationDescription, Command}
import org.apache.spark.deploy.client.{StandaloneAppClient, StandaloneAppClientListener}
import org.apache.spark.executor.ExecutorExitCode
import org.apache.spark.internal.{config, Logging, MDC}
-import org.apache.spark.internal.LogKey.REASON
+import org.apache.spark.internal.LogKeys.REASON
import org.apache.spark.internal.config.EXECUTOR_REMOVE_DELAY
import org.apache.spark.internal.config.Tests.IS_TESTING
import org.apache.spark.launcher.{LauncherBackend, SparkAppHandle}
diff --git a/core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala b/core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala
index 5a0b2ba3735c..2ca30cf4cd62 100644
--- a/core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala
+++ b/core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala
@@ -40,7 +40,7 @@ import org.roaringbitmap.RoaringBitmap
import org.apache.spark._
import org.apache.spark.api.python.PythonBroadcast
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.CLASS_NAME
+import org.apache.spark.internal.LogKeys.CLASS_NAME
import org.apache.spark.internal.config.Kryo._
import org.apache.spark.internal.io.FileCommitProtocol._
import org.apache.spark.network.util.ByteUnit
diff --git a/core/src/main/scala/org/apache/spark/shuffle/IndexShuffleBlockResolver.scala b/core/src/main/scala/org/apache/spark/shuffle/IndexShuffleBlockResolver.scala
index e7b9a7e2f0ee..a1a204dfc01b 100644
--- a/core/src/main/scala/org/apache/spark/shuffle/IndexShuffleBlockResolver.scala
+++ b/core/src/main/scala/org/apache/spark/shuffle/IndexShuffleBlockResolver.scala
@@ -29,7 +29,7 @@ import com.google.common.cache.CacheBuilder
import org.apache.spark.{SecurityManager, SparkConf, SparkEnv, SparkException}
import org.apache.spark.errors.SparkCoreErrors
import org.apache.spark.internal.{config, Logging, MDC}
-import org.apache.spark.internal.LogKey.PATH
+import org.apache.spark.internal.LogKeys.PATH
import org.apache.spark.io.NioBufferedFileInputStream
import org.apache.spark.network.buffer.{FileSegmentManagedBuffer, ManagedBuffer}
import org.apache.spark.network.client.StreamCallbackWithID
diff --git a/core/src/main/scala/org/apache/spark/status/AppStatusStore.scala b/core/src/main/scala/org/apache/spark/status/AppStatusStore.scala
index 109a9a2e3eb9..29e21e7f5ffe 100644
--- a/core/src/main/scala/org/apache/spark/status/AppStatusStore.scala
+++ b/core/src/main/scala/org/apache/spark/status/AppStatusStore.scala
@@ -26,7 +26,7 @@ import scala.jdk.CollectionConverters._
import org.apache.spark.{JobExecutionStatus, SparkConf, SparkContext}
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.PATH
+import org.apache.spark.internal.LogKeys.PATH
import org.apache.spark.internal.config.Status.LIVE_UI_LOCAL_STORE_DIR
import org.apache.spark.status.AppStatusUtils.getQuantilesValue
import org.apache.spark.status.api.v1
diff --git a/core/src/main/scala/org/apache/spark/storage/BlockManager.scala b/core/src/main/scala/org/apache/spark/storage/BlockManager.scala
index ebe782301044..740031d70752 100644
--- a/core/src/main/scala/org/apache/spark/storage/BlockManager.scala
+++ b/core/src/main/scala/org/apache/spark/storage/BlockManager.scala
@@ -42,7 +42,7 @@ import org.apache.spark._
import org.apache.spark.errors.SparkCoreErrors
import org.apache.spark.executor.DataReadMethod
import org.apache.spark.internal.{config, Logging, MDC}
-import org.apache.spark.internal.LogKey.{BLOCK_ID, COUNT, SLEEP_TIME}
+import org.apache.spark.internal.LogKeys.{BLOCK_ID, COUNT, SLEEP_TIME}
import org.apache.spark.internal.config.{Network, RDD_CACHE_VISIBILITY_TRACKING_ENABLED, Tests}
import org.apache.spark.memory.{MemoryManager, MemoryMode}
import org.apache.spark.metrics.source.Source
diff --git a/core/src/main/scala/org/apache/spark/storage/BlockManagerDecommissioner.scala b/core/src/main/scala/org/apache/spark/storage/BlockManagerDecommissioner.scala
index 5b4ecef233f8..ad4533166316 100644
--- a/core/src/main/scala/org/apache/spark/storage/BlockManagerDecommissioner.scala
+++ b/core/src/main/scala/org/apache/spark/storage/BlockManagerDecommissioner.scala
@@ -27,7 +27,7 @@ import scala.util.control.NonFatal
import org.apache.spark._
import org.apache.spark.errors.SparkCoreErrors
import org.apache.spark.internal.{config, Logging, MDC}
-import org.apache.spark.internal.LogKey.SHUFFLE_BLOCK_INFO
+import org.apache.spark.internal.LogKeys.SHUFFLE_BLOCK_INFO
import org.apache.spark.shuffle.ShuffleBlockInfo
import org.apache.spark.storage.BlockManagerMessages.ReplicateBlock
import org.apache.spark.util.{ThreadUtils, Utils}
diff --git a/core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala b/core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala
index 5bb4e096c029..cf3e755fc4f9 100644
--- a/core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala
+++ b/core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala
@@ -32,7 +32,7 @@ import com.google.common.cache.CacheBuilder
import org.apache.spark.{MapOutputTrackerMaster, SparkConf, SparkContext, SparkEnv}
import org.apache.spark.annotation.DeveloperApi
import org.apache.spark.internal.{config, Logging, MDC}
-import org.apache.spark.internal.LogKey.{BLOCK_MANAGER_ID, EXECUTOR_ID, OLD_BLOCK_MANAGER_ID}
+import org.apache.spark.internal.LogKeys.{BLOCK_MANAGER_ID, EXECUTOR_ID, OLD_BLOCK_MANAGER_ID}
import org.apache.spark.internal.config.RDD_CACHE_VISIBILITY_TRACKING_ENABLED
import org.apache.spark.network.shuffle.{ExternalBlockStoreClient, RemoteBlockPushResolver}
import org.apache.spark.rpc.{IsolatedThreadSafeRpcEndpoint, RpcCallContext, RpcEndpointRef, RpcEnv}
diff --git a/core/src/main/scala/org/apache/spark/storage/BlockManagerStorageEndpoint.scala b/core/src/main/scala/org/apache/spark/storage/BlockManagerStorageEndpoint.scala
index 1fccbd16ced5..686ac1eb786e 100644
--- a/core/src/main/scala/org/apache/spark/storage/BlockManagerStorageEndpoint.scala
+++ b/core/src/main/scala/org/apache/spark/storage/BlockManagerStorageEndpoint.scala
@@ -21,7 +21,7 @@ import scala.concurrent.{ExecutionContext, ExecutionContextExecutorService, Futu
import org.apache.spark.{MapOutputTracker, SparkEnv}
import org.apache.spark.internal.{Logging, MDC, MessageWithContext}
-import org.apache.spark.internal.LogKey.{BLOCK_ID, BROADCAST_ID, RDD_ID, SHUFFLE_ID}
+import org.apache.spark.internal.LogKeys.{BLOCK_ID, BROADCAST_ID, RDD_ID, SHUFFLE_ID}
import org.apache.spark.rpc.{IsolatedThreadSafeRpcEndpoint, RpcCallContext, RpcEnv}
import org.apache.spark.storage.BlockManagerMessages._
import org.apache.spark.util.{ThreadUtils, Utils}
diff --git a/core/src/main/scala/org/apache/spark/storage/DiskBlockManager.scala b/core/src/main/scala/org/apache/spark/storage/DiskBlockManager.scala
index 4c0b5f4a14f6..ad60b66b4f7a 100644
--- a/core/src/main/scala/org/apache/spark/storage/DiskBlockManager.scala
+++ b/core/src/main/scala/org/apache/spark/storage/DiskBlockManager.scala
@@ -31,7 +31,7 @@ import org.apache.spark.{SparkConf, SparkException}
import org.apache.spark.errors.SparkCoreErrors
import org.apache.spark.executor.ExecutorExitCode
import org.apache.spark.internal.{config, Logging, MDC}
-import org.apache.spark.internal.LogKey.{MERGE_DIR_NAME, PATH}
+import org.apache.spark.internal.LogKeys.{MERGE_DIR_NAME, PATH}
import org.apache.spark.network.shuffle.ExecutorDiskUtils
import org.apache.spark.storage.DiskBlockManager.ATTEMPT_ID_KEY
import org.apache.spark.storage.DiskBlockManager.MERGE_DIR_KEY
diff --git a/core/src/main/scala/org/apache/spark/storage/DiskBlockObjectWriter.scala b/core/src/main/scala/org/apache/spark/storage/DiskBlockObjectWriter.scala
index 0b6e33ff5fb3..d6707419d2fb 100644
--- a/core/src/main/scala/org/apache/spark/storage/DiskBlockObjectWriter.scala
+++ b/core/src/main/scala/org/apache/spark/storage/DiskBlockObjectWriter.scala
@@ -25,7 +25,7 @@ import java.util.zip.Checksum
import org.apache.spark.SparkException
import org.apache.spark.errors.SparkCoreErrors
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{ERROR, PATH}
+import org.apache.spark.internal.LogKeys.{ERROR, PATH}
import org.apache.spark.io.MutableCheckedOutputStream
import org.apache.spark.serializer.{SerializationStream, SerializerInstance, SerializerManager}
import org.apache.spark.shuffle.ShuffleWriteMetricsReporter
diff --git a/core/src/main/scala/org/apache/spark/storage/PushBasedFetchHelper.scala b/core/src/main/scala/org/apache/spark/storage/PushBasedFetchHelper.scala
index 31958af84e54..52056594e792 100644
--- a/core/src/main/scala/org/apache/spark/storage/PushBasedFetchHelper.scala
+++ b/core/src/main/scala/org/apache/spark/storage/PushBasedFetchHelper.scala
@@ -29,7 +29,7 @@ import org.roaringbitmap.RoaringBitmap
import org.apache.spark.MapOutputTracker
import org.apache.spark.MapOutputTracker.SHUFFLE_PUSH_MAP_ID
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{HOST, PORT, REDUCE_ID, SHUFFLE_ID, SHUFFLE_MERGE_ID}
+import org.apache.spark.internal.LogKeys.{HOST, PORT, REDUCE_ID, SHUFFLE_ID, SHUFFLE_MERGE_ID}
import org.apache.spark.network.shuffle.{BlockStoreClient, MergedBlockMeta, MergedBlocksMetaListener}
import org.apache.spark.shuffle.ShuffleReadMetricsReporter
import org.apache.spark.storage.BlockManagerId.SHUFFLE_MERGER_IDENTIFIER
diff --git a/core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala b/core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala
index d22ce3dbed77..96508649522c 100644
--- a/core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala
+++ b/core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala
@@ -37,7 +37,7 @@ import org.apache.spark.{MapOutputTracker, SparkException, TaskContext}
import org.apache.spark.MapOutputTracker.SHUFFLE_PUSH_MAP_ID
import org.apache.spark.errors.SparkCoreErrors
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{BLOCK_ID, ERROR, HOST, MAX_ATTEMPTS, PORT}
+import org.apache.spark.internal.LogKeys.{BLOCK_ID, ERROR, HOST, MAX_ATTEMPTS, PORT}
import org.apache.spark.network.buffer.{FileSegmentManagedBuffer, ManagedBuffer}
import org.apache.spark.network.shuffle._
import org.apache.spark.network.shuffle.checksum.{Cause, ShuffleChecksumHelper}
diff --git a/core/src/main/scala/org/apache/spark/ui/DriverLogPage.scala b/core/src/main/scala/org/apache/spark/ui/DriverLogPage.scala
index 310211515999..8b4eebc26b3b 100644
--- a/core/src/main/scala/org/apache/spark/ui/DriverLogPage.scala
+++ b/core/src/main/scala/org/apache/spark/ui/DriverLogPage.scala
@@ -22,7 +22,7 @@ import jakarta.servlet.http.HttpServletRequest
import org.apache.spark.SparkConf
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{LOG_TYPE, PATH}
+import org.apache.spark.internal.LogKeys.{LOG_TYPE, PATH}
import org.apache.spark.internal.config.DRIVER_LOG_LOCAL_DIR
import org.apache.spark.util.Utils
import org.apache.spark.util.logging.DriverLogger.DRIVER_LOG_FILE
diff --git a/core/src/main/scala/org/apache/spark/ui/SparkUI.scala b/core/src/main/scala/org/apache/spark/ui/SparkUI.scala
index ddf451c16f3a..ac76f74725e8 100644
--- a/core/src/main/scala/org/apache/spark/ui/SparkUI.scala
+++ b/core/src/main/scala/org/apache/spark/ui/SparkUI.scala
@@ -24,7 +24,7 @@ import org.eclipse.jetty.servlet.ServletContextHandler
import org.apache.spark.{SecurityManager, SparkConf, SparkContext}
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.CLASS_NAME
+import org.apache.spark.internal.LogKeys.CLASS_NAME
import org.apache.spark.internal.config.DRIVER_LOG_LOCAL_DIR
import org.apache.spark.internal.config.UI._
import org.apache.spark.scheduler._
diff --git a/core/src/main/scala/org/apache/spark/ui/WebUI.scala b/core/src/main/scala/org/apache/spark/ui/WebUI.scala
index baeed322e8ad..25d411bbf6b3 100644
--- a/core/src/main/scala/org/apache/spark/ui/WebUI.scala
+++ b/core/src/main/scala/org/apache/spark/ui/WebUI.scala
@@ -30,7 +30,7 @@ import org.json4s.JsonAST.{JNothing, JValue}
import org.apache.spark.{SecurityManager, SparkConf, SSLOptions}
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.CLASS_NAME
+import org.apache.spark.internal.LogKeys.CLASS_NAME
import org.apache.spark.internal.config._
import org.apache.spark.ui.JettyUtils._
import org.apache.spark.util.Utils
diff --git a/core/src/main/scala/org/apache/spark/util/EventLoop.scala b/core/src/main/scala/org/apache/spark/util/EventLoop.scala
index eaa9ef517294..b9de661b63c4 100644
--- a/core/src/main/scala/org/apache/spark/util/EventLoop.scala
+++ b/core/src/main/scala/org/apache/spark/util/EventLoop.scala
@@ -23,7 +23,7 @@ import java.util.concurrent.atomic.AtomicBoolean
import scala.util.control.NonFatal
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.EVENT_LOOP
+import org.apache.spark.internal.LogKeys.EVENT_LOOP
/**
* An event loop to receive events from the caller and process all events in the event thread. It
diff --git a/core/src/main/scala/org/apache/spark/util/ListenerBus.scala b/core/src/main/scala/org/apache/spark/util/ListenerBus.scala
index 814201d8c959..51499870a6a3 100644
--- a/core/src/main/scala/org/apache/spark/util/ListenerBus.scala
+++ b/core/src/main/scala/org/apache/spark/util/ListenerBus.scala
@@ -27,7 +27,7 @@ import com.codahale.metrics.Timer
import org.apache.spark.SparkEnv
import org.apache.spark.internal.{config, Logging, MDC}
-import org.apache.spark.internal.LogKey.LISTENER
+import org.apache.spark.internal.LogKeys.LISTENER
import org.apache.spark.scheduler.EventLoggingListener
import org.apache.spark.scheduler.SparkListenerEnvironmentUpdate
diff --git a/core/src/main/scala/org/apache/spark/util/ShutdownHookManager.scala b/core/src/main/scala/org/apache/spark/util/ShutdownHookManager.scala
index b9dece19f265..993352c6a637 100644
--- a/core/src/main/scala/org/apache/spark/util/ShutdownHookManager.scala
+++ b/core/src/main/scala/org/apache/spark/util/ShutdownHookManager.scala
@@ -27,7 +27,7 @@ import org.apache.hadoop.fs.FileSystem
import org.apache.spark.SparkConf
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.PATH
+import org.apache.spark.internal.LogKeys.PATH
import org.apache.spark.internal.config.SPARK_SHUTDOWN_TIMEOUT_MS
diff --git a/core/src/main/scala/org/apache/spark/util/SparkUncaughtExceptionHandler.scala b/core/src/main/scala/org/apache/spark/util/SparkUncaughtExceptionHandler.scala
index 74f1474f9cf7..730b762a3948 100644
--- a/core/src/main/scala/org/apache/spark/util/SparkUncaughtExceptionHandler.scala
+++ b/core/src/main/scala/org/apache/spark/util/SparkUncaughtExceptionHandler.scala
@@ -18,7 +18,7 @@
package org.apache.spark.util
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.THREAD
+import org.apache.spark.internal.LogKeys.THREAD
/**
* The default uncaught exception handler for Spark daemons. It terminates the whole process for
diff --git a/core/src/main/scala/org/apache/spark/util/Utils.scala b/core/src/main/scala/org/apache/spark/util/Utils.scala
index af91a4b32c6f..67ba72485277 100644
--- a/core/src/main/scala/org/apache/spark/util/Utils.scala
+++ b/core/src/main/scala/org/apache/spark/util/Utils.scala
@@ -69,7 +69,7 @@ import org.slf4j.Logger
import org.apache.spark._
import org.apache.spark.deploy.SparkHadoopUtil
import org.apache.spark.internal.{Logging, MDC, MessageWithContext}
-import org.apache.spark.internal.LogKey._
+import org.apache.spark.internal.LogKeys._
import org.apache.spark.internal.config._
import org.apache.spark.internal.config.Streaming._
import org.apache.spark.internal.config.Tests.IS_TESTING
diff --git a/core/src/main/scala/org/apache/spark/util/logging/FileAppender.scala b/core/src/main/scala/org/apache/spark/util/logging/FileAppender.scala
index 1dadf15da40f..23cd222f68ef 100644
--- a/core/src/main/scala/org/apache/spark/util/logging/FileAppender.scala
+++ b/core/src/main/scala/org/apache/spark/util/logging/FileAppender.scala
@@ -21,7 +21,7 @@ import java.io.{File, FileOutputStream, InputStream, IOException}
import org.apache.spark.SparkConf
import org.apache.spark.internal.{config, Logging, MDC}
-import org.apache.spark.internal.LogKey.PATH
+import org.apache.spark.internal.LogKeys.PATH
import org.apache.spark.util.{IntParam, Utils}
/**
diff --git a/core/src/main/scala/org/apache/spark/util/logging/RollingFileAppender.scala b/core/src/main/scala/org/apache/spark/util/logging/RollingFileAppender.scala
index f8f144f6e388..28ffdc850897 100644
--- a/core/src/main/scala/org/apache/spark/util/logging/RollingFileAppender.scala
+++ b/core/src/main/scala/org/apache/spark/util/logging/RollingFileAppender.scala
@@ -25,7 +25,7 @@ import org.apache.commons.io.IOUtils
import org.apache.spark.SparkConf
import org.apache.spark.internal.{config, MDC}
-import org.apache.spark.internal.LogKey.PATH
+import org.apache.spark.internal.LogKeys.PATH
import org.apache.spark.util.ArrayImplicits._
/**
diff --git a/graphx/src/main/scala/org/apache/spark/graphx/GraphLoader.scala b/graphx/src/main/scala/org/apache/spark/graphx/GraphLoader.scala
index 193d6551a366..045149922c8e 100644
--- a/graphx/src/main/scala/org/apache/spark/graphx/GraphLoader.scala
+++ b/graphx/src/main/scala/org/apache/spark/graphx/GraphLoader.scala
@@ -22,7 +22,7 @@ import java.util.concurrent.TimeUnit
import org.apache.spark.SparkContext
import org.apache.spark.graphx.impl.{EdgePartitionBuilder, GraphImpl}
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.TOTAL_TIME
+import org.apache.spark.internal.LogKeys.TOTAL_TIME
import org.apache.spark.storage.StorageLevel
/**
diff --git a/graphx/src/main/scala/org/apache/spark/graphx/Pregel.scala b/graphx/src/main/scala/org/apache/spark/graphx/Pregel.scala
index e1b9b2fe5ae6..1493d8114c69 100644
--- a/graphx/src/main/scala/org/apache/spark/graphx/Pregel.scala
+++ b/graphx/src/main/scala/org/apache/spark/graphx/Pregel.scala
@@ -21,7 +21,7 @@ import scala.reflect.ClassTag
import org.apache.spark.graphx.util.PeriodicGraphCheckpointer
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.NUM_ITERATIONS
+import org.apache.spark.internal.LogKeys.NUM_ITERATIONS
import org.apache.spark.rdd.RDD
import org.apache.spark.rdd.util.PeriodicRDDCheckpointer
diff --git a/graphx/src/main/scala/org/apache/spark/graphx/lib/PageRank.scala b/graphx/src/main/scala/org/apache/spark/graphx/lib/PageRank.scala
index aa5898fb585c..4fe010bfce78 100644
--- a/graphx/src/main/scala/org/apache/spark/graphx/lib/PageRank.scala
+++ b/graphx/src/main/scala/org/apache/spark/graphx/lib/PageRank.scala
@@ -23,7 +23,7 @@ import breeze.linalg.{Vector => BV}
import org.apache.spark.graphx._
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.NUM_ITERATIONS
+import org.apache.spark.internal.LogKeys.NUM_ITERATIONS
import org.apache.spark.ml.linalg.{Vector, Vectors}
/**
diff --git a/mllib/src/main/scala/org/apache/spark/ml/classification/LinearSVC.scala b/mllib/src/main/scala/org/apache/spark/ml/classification/LinearSVC.scala
index 024693ba06f2..297b28456fd6 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/classification/LinearSVC.scala
+++ b/mllib/src/main/scala/org/apache/spark/ml/classification/LinearSVC.scala
@@ -26,7 +26,7 @@ import org.apache.hadoop.fs.Path
import org.apache.spark.SparkException
import org.apache.spark.annotation.Since
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{COUNT, RANGE}
+import org.apache.spark.internal.LogKeys.{COUNT, RANGE}
import org.apache.spark.ml.feature._
import org.apache.spark.ml.linalg._
import org.apache.spark.ml.optim.aggregator._
diff --git a/mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala b/mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala
index 0d487377b931..03c3b247e9ee 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala
+++ b/mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala
@@ -28,7 +28,7 @@ import org.apache.hadoop.fs.Path
import org.apache.spark.SparkException
import org.apache.spark.annotation.Since
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{COUNT, RANGE}
+import org.apache.spark.internal.LogKeys.{COUNT, RANGE}
import org.apache.spark.ml.feature._
import org.apache.spark.ml.impl.Utils
import org.apache.spark.ml.linalg._
diff --git a/mllib/src/main/scala/org/apache/spark/ml/clustering/KMeans.scala b/mllib/src/main/scala/org/apache/spark/ml/clustering/KMeans.scala
index 7d1ade18986d..120c7c100dbc 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/clustering/KMeans.scala
+++ b/mllib/src/main/scala/org/apache/spark/ml/clustering/KMeans.scala
@@ -22,7 +22,7 @@ import scala.collection.mutable
import org.apache.hadoop.fs.Path
import org.apache.spark.annotation.Since
-import org.apache.spark.internal.LogKey.{COST, INIT_MODE, NUM_ITERATIONS, TOTAL_TIME}
+import org.apache.spark.internal.LogKeys.{COST, INIT_MODE, NUM_ITERATIONS, TOTAL_TIME}
import org.apache.spark.internal.MDC
import org.apache.spark.ml.{Estimator, Model, PipelineStage}
import org.apache.spark.ml.feature.{Instance, InstanceBlock}
diff --git a/mllib/src/main/scala/org/apache/spark/ml/optim/IterativelyReweightedLeastSquares.scala b/mllib/src/main/scala/org/apache/spark/ml/optim/IterativelyReweightedLeastSquares.scala
index 337d3fa03422..17adf61e46e9 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/optim/IterativelyReweightedLeastSquares.scala
+++ b/mllib/src/main/scala/org/apache/spark/ml/optim/IterativelyReweightedLeastSquares.scala
@@ -17,7 +17,7 @@
package org.apache.spark.ml.optim
-import org.apache.spark.internal.LogKey.{NUM_ITERATIONS, RELATIVE_TOLERANCE}
+import org.apache.spark.internal.LogKeys.{NUM_ITERATIONS, RELATIVE_TOLERANCE}
import org.apache.spark.internal.MDC
import org.apache.spark.ml.feature.{Instance, OffsetInstance}
import org.apache.spark.ml.linalg._
diff --git a/mllib/src/main/scala/org/apache/spark/ml/optim/WeightedLeastSquares.scala b/mllib/src/main/scala/org/apache/spark/ml/optim/WeightedLeastSquares.scala
index 718b21873bde..757630b0ad16 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/optim/WeightedLeastSquares.scala
+++ b/mllib/src/main/scala/org/apache/spark/ml/optim/WeightedLeastSquares.scala
@@ -17,7 +17,7 @@
package org.apache.spark.ml.optim
-import org.apache.spark.internal.LogKey.COUNT
+import org.apache.spark.internal.LogKeys.COUNT
import org.apache.spark.internal.MDC
import org.apache.spark.ml.feature.Instance
import org.apache.spark.ml.linalg._
diff --git a/mllib/src/main/scala/org/apache/spark/ml/r/RWrapperUtils.scala b/mllib/src/main/scala/org/apache/spark/ml/r/RWrapperUtils.scala
index a9a7ca336589..9e66647ef35f 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/r/RWrapperUtils.scala
+++ b/mllib/src/main/scala/org/apache/spark/ml/r/RWrapperUtils.scala
@@ -18,7 +18,7 @@
package org.apache.spark.ml.r
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{FEATURE_COLUMN, LABEL_COLUMN, NEW_FEATURE_COLUMN_NAME, NEW_LABEL_COLUMN_NAME}
+import org.apache.spark.internal.LogKeys.{FEATURE_COLUMN, LABEL_COLUMN, NEW_FEATURE_COLUMN_NAME, NEW_LABEL_COLUMN_NAME}
import org.apache.spark.ml.attribute.{Attribute, AttributeGroup, NominalAttribute}
import org.apache.spark.ml.feature.{RFormula, RFormulaModel}
import org.apache.spark.ml.util.Identifiable
diff --git a/mllib/src/main/scala/org/apache/spark/ml/tree/impl/GradientBoostedTrees.scala b/mllib/src/main/scala/org/apache/spark/ml/tree/impl/GradientBoostedTrees.scala
index 36d52d07a6e0..5184109bd3a5 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/tree/impl/GradientBoostedTrees.scala
+++ b/mllib/src/main/scala/org/apache/spark/ml/tree/impl/GradientBoostedTrees.scala
@@ -19,7 +19,7 @@ package org.apache.spark.ml.tree.impl
import org.apache.spark.broadcast.Broadcast
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.TIMER
+import org.apache.spark.internal.LogKeys.TIMER
import org.apache.spark.ml.feature.Instance
import org.apache.spark.ml.linalg.Vector
import org.apache.spark.ml.regression.DecisionTreeRegressionModel
diff --git a/mllib/src/main/scala/org/apache/spark/ml/tree/impl/RandomForest.scala b/mllib/src/main/scala/org/apache/spark/ml/tree/impl/RandomForest.scala
index 0719f5dfc84b..c6783f755604 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/tree/impl/RandomForest.scala
+++ b/mllib/src/main/scala/org/apache/spark/ml/tree/impl/RandomForest.scala
@@ -22,7 +22,7 @@ import scala.util.Random
import org.apache.spark.broadcast.Broadcast
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{NUM_CLASSES, NUM_EXAMPLES, NUM_FEATURES, TIMER, WEIGHTED_NUM}
+import org.apache.spark.internal.LogKeys.{NUM_CLASSES, NUM_EXAMPLES, NUM_FEATURES, TIMER, WEIGHTED_NUM}
import org.apache.spark.ml.classification.DecisionTreeClassificationModel
import org.apache.spark.ml.feature.Instance
import org.apache.spark.ml.impl.Utils
diff --git a/mllib/src/main/scala/org/apache/spark/ml/tuning/CrossValidator.scala b/mllib/src/main/scala/org/apache/spark/ml/tuning/CrossValidator.scala
index 6bc4ecfb687b..55b78d4bf3bb 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/tuning/CrossValidator.scala
+++ b/mllib/src/main/scala/org/apache/spark/ml/tuning/CrossValidator.scala
@@ -28,7 +28,7 @@ import org.json4s.DefaultFormats
import org.apache.spark.annotation.Since
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{CROSS_VALIDATION_METRIC, CROSS_VALIDATION_METRICS, ESTIMATOR_PARAMETER_MAP}
+import org.apache.spark.internal.LogKeys.{CROSS_VALIDATION_METRIC, CROSS_VALIDATION_METRICS, ESTIMATOR_PARAMETER_MAP}
import org.apache.spark.ml.{Estimator, Model}
import org.apache.spark.ml.evaluation.Evaluator
import org.apache.spark.ml.param.{IntParam, Param, ParamMap, ParamValidators}
diff --git a/mllib/src/main/scala/org/apache/spark/ml/tuning/TrainValidationSplit.scala b/mllib/src/main/scala/org/apache/spark/ml/tuning/TrainValidationSplit.scala
index 692a68cb3565..fa4bcc32de95 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/tuning/TrainValidationSplit.scala
+++ b/mllib/src/main/scala/org/apache/spark/ml/tuning/TrainValidationSplit.scala
@@ -29,7 +29,7 @@ import org.json4s.DefaultFormats
import org.apache.spark.annotation.Since
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{ESTIMATOR_PARAMETER_MAP, TRAIN_VALIDATION_SPLIT_METRIC, TRAIN_VALIDATION_SPLIT_METRICS}
+import org.apache.spark.internal.LogKeys.{ESTIMATOR_PARAMETER_MAP, TRAIN_VALIDATION_SPLIT_METRIC, TRAIN_VALIDATION_SPLIT_METRICS}
import org.apache.spark.ml.{Estimator, Model}
import org.apache.spark.ml.evaluation.Evaluator
import org.apache.spark.ml.param.{DoubleParam, ParamMap, ParamValidators}
diff --git a/mllib/src/main/scala/org/apache/spark/ml/util/DatasetUtils.scala b/mllib/src/main/scala/org/apache/spark/ml/util/DatasetUtils.scala
index 41d28ac55c62..d5b6396e5ba8 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/util/DatasetUtils.scala
+++ b/mllib/src/main/scala/org/apache/spark/ml/util/DatasetUtils.scala
@@ -19,7 +19,7 @@ package org.apache.spark.ml.util
import org.apache.spark.SparkException
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{CLASS_NAME, LABEL_COLUMN, NUM_CLASSES}
+import org.apache.spark.internal.LogKeys.{CLASS_NAME, LABEL_COLUMN, NUM_CLASSES}
import org.apache.spark.ml.PredictorParams
import org.apache.spark.ml.classification.ClassifierParams
import org.apache.spark.ml.feature.Instance
diff --git a/mllib/src/main/scala/org/apache/spark/ml/util/Instrumentation.scala b/mllib/src/main/scala/org/apache/spark/ml/util/Instrumentation.scala
index e71addd9b311..acb6bf831239 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/util/Instrumentation.scala
+++ b/mllib/src/main/scala/org/apache/spark/ml/util/Instrumentation.scala
@@ -28,7 +28,7 @@ import org.json4s.JsonDSL._
import org.json4s.jackson.JsonMethods._
import org.apache.spark.internal.{LogEntry, Logging, MDC}
-import org.apache.spark.internal.LogKey.{CLASS_NAME, NUM_PARTITION, PIPELINE_STAGE_UID, STORAGE_LEVEL}
+import org.apache.spark.internal.LogKeys.{CLASS_NAME, NUM_PARTITION, PIPELINE_STAGE_UID, STORAGE_LEVEL}
import org.apache.spark.ml.{MLEvents, PipelineStage}
import org.apache.spark.ml.param.{Param, Params}
import org.apache.spark.rdd.RDD
diff --git a/mllib/src/main/scala/org/apache/spark/ml/util/ReadWrite.scala b/mllib/src/main/scala/org/apache/spark/ml/util/ReadWrite.scala
index 2a9cd006e9dc..9b26d0a911ac 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/util/ReadWrite.scala
+++ b/mllib/src/main/scala/org/apache/spark/ml/util/ReadWrite.scala
@@ -33,7 +33,7 @@ import org.json4s.jackson.JsonMethods._
import org.apache.spark.{SparkContext, SparkException}
import org.apache.spark.annotation.{Since, Unstable}
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.PATH
+import org.apache.spark.internal.LogKeys.PATH
import org.apache.spark.ml._
import org.apache.spark.ml.classification.{OneVsRest, OneVsRestModel}
import org.apache.spark.ml.feature.RFormulaModel
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/clustering/BisectingKMeans.scala b/mllib/src/main/scala/org/apache/spark/mllib/clustering/BisectingKMeans.scala
index 2921e6bfedf8..c826654f0893 100644
--- a/mllib/src/main/scala/org/apache/spark/mllib/clustering/BisectingKMeans.scala
+++ b/mllib/src/main/scala/org/apache/spark/mllib/clustering/BisectingKMeans.scala
@@ -25,7 +25,7 @@ import scala.collection.mutable
import org.apache.spark.annotation.Since
import org.apache.spark.api.java.JavaRDD
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{CLUSTER_LEVEL, COST, DIVISIBLE_CLUSTER_INDICES_SIZE, FEATURE_DIMENSION, MIN_POINT_PER_CLUSTER, NUM_POINT}
+import org.apache.spark.internal.LogKeys.{CLUSTER_LEVEL, COST, DIVISIBLE_CLUSTER_INDICES_SIZE, FEATURE_DIMENSION, MIN_POINT_PER_CLUSTER, NUM_POINT}
import org.apache.spark.ml.util.Instrumentation
import org.apache.spark.mllib.linalg.{Vector, Vectors}
import org.apache.spark.mllib.linalg.BLAS.axpy
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/clustering/KMeans.scala b/mllib/src/main/scala/org/apache/spark/mllib/clustering/KMeans.scala
index 336bfd19e514..52fbc7a5a47f 100644
--- a/mllib/src/main/scala/org/apache/spark/mllib/clustering/KMeans.scala
+++ b/mllib/src/main/scala/org/apache/spark/mllib/clustering/KMeans.scala
@@ -22,7 +22,7 @@ import scala.collection.mutable.ArrayBuffer
import org.apache.spark.annotation.Since
import org.apache.spark.broadcast.Broadcast
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{COST, INIT_MODE, NUM_ITERATIONS, TOTAL_TIME}
+import org.apache.spark.internal.LogKeys.{COST, INIT_MODE, NUM_ITERATIONS, TOTAL_TIME}
import org.apache.spark.ml.util.Instrumentation
import org.apache.spark.mllib.linalg.{Vector, Vectors}
import org.apache.spark.mllib.linalg.BLAS.axpy
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/clustering/LocalKMeans.scala b/mllib/src/main/scala/org/apache/spark/mllib/clustering/LocalKMeans.scala
index ea602fd22d81..11af5a3368fa 100644
--- a/mllib/src/main/scala/org/apache/spark/mllib/clustering/LocalKMeans.scala
+++ b/mllib/src/main/scala/org/apache/spark/mllib/clustering/LocalKMeans.scala
@@ -20,7 +20,7 @@ package org.apache.spark.mllib.clustering
import scala.util.Random
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.NUM_ITERATIONS
+import org.apache.spark.internal.LogKeys.NUM_ITERATIONS
import org.apache.spark.mllib.linalg.BLAS.{axpy, scal}
import org.apache.spark.mllib.linalg.Vectors
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/clustering/PowerIterationClustering.scala b/mllib/src/main/scala/org/apache/spark/mllib/clustering/PowerIterationClustering.scala
index cc80b676b0df..9150bb305876 100644
--- a/mllib/src/main/scala/org/apache/spark/mllib/clustering/PowerIterationClustering.scala
+++ b/mllib/src/main/scala/org/apache/spark/mllib/clustering/PowerIterationClustering.scala
@@ -26,7 +26,7 @@ import org.apache.spark.annotation.Since
import org.apache.spark.api.java.JavaRDD
import org.apache.spark.graphx._
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{DELTA, DIFF_DELTA, NORM, NUM_ITERATIONS}
+import org.apache.spark.internal.LogKeys.{DELTA, DIFF_DELTA, NORM, NUM_ITERATIONS}
import org.apache.spark.mllib.linalg.Vectors
import org.apache.spark.mllib.util.{Loader, MLUtils, Saveable}
import org.apache.spark.rdd.RDD
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/clustering/StreamingKMeans.scala b/mllib/src/main/scala/org/apache/spark/mllib/clustering/StreamingKMeans.scala
index d6d590caa74a..85a735007810 100644
--- a/mllib/src/main/scala/org/apache/spark/mllib/clustering/StreamingKMeans.scala
+++ b/mllib/src/main/scala/org/apache/spark/mllib/clustering/StreamingKMeans.scala
@@ -22,7 +22,7 @@ import scala.reflect.ClassTag
import org.apache.spark.annotation.Since
import org.apache.spark.api.java.JavaSparkContext._
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{CLUSTER_CENTROIDS, CLUSTER_LABEL, CLUSTER_WEIGHT, LARGEST_CLUSTER_INDEX, SMALLEST_CLUSTER_INDEX}
+import org.apache.spark.internal.LogKeys.{CLUSTER_CENTROIDS, CLUSTER_LABEL, CLUSTER_WEIGHT, LARGEST_CLUSTER_INDEX, SMALLEST_CLUSTER_INDEX}
import org.apache.spark.mllib.linalg.{BLAS, Vector, Vectors}
import org.apache.spark.rdd.RDD
import org.apache.spark.streaming.api.java.{JavaDStream, JavaPairDStream}
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/evaluation/BinaryClassificationMetrics.scala b/mllib/src/main/scala/org/apache/spark/mllib/evaluation/BinaryClassificationMetrics.scala
index 3d7e3740c61e..a203d3fc7353 100644
--- a/mllib/src/main/scala/org/apache/spark/mllib/evaluation/BinaryClassificationMetrics.scala
+++ b/mllib/src/main/scala/org/apache/spark/mllib/evaluation/BinaryClassificationMetrics.scala
@@ -19,7 +19,7 @@ package org.apache.spark.mllib.evaluation
import org.apache.spark.annotation.Since
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{COUNT, NUM_BIN}
+import org.apache.spark.internal.LogKeys.{COUNT, NUM_BIN}
import org.apache.spark.mllib.evaluation.binary._
import org.apache.spark.rdd.RDD
import org.apache.spark.sql.{DataFrame, Row}
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala b/mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala
index fc28258188cc..d2fc78a7ff7f 100644
--- a/mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala
+++ b/mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala
@@ -32,7 +32,7 @@ import org.apache.spark.annotation.Since
import org.apache.spark.api.java.JavaRDD
import org.apache.spark.broadcast.Broadcast
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{ALPHA, COUNT, TRAIN_WORD_COUNT, VOCAB_SIZE}
+import org.apache.spark.internal.LogKeys.{ALPHA, COUNT, TRAIN_WORD_COUNT, VOCAB_SIZE}
import org.apache.spark.internal.config.Kryo.KRYO_SERIALIZER_MAX_BUFFER_SIZE
import org.apache.spark.ml.linalg.BLAS
import org.apache.spark.mllib.linalg.{Vector, Vectors}
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/fpm/PrefixSpan.scala b/mllib/src/main/scala/org/apache/spark/mllib/fpm/PrefixSpan.scala
index 2bded0e23e8b..539e93867bc2 100644
--- a/mllib/src/main/scala/org/apache/spark/mllib/fpm/PrefixSpan.scala
+++ b/mllib/src/main/scala/org/apache/spark/mllib/fpm/PrefixSpan.scala
@@ -34,7 +34,7 @@ import org.apache.spark.annotation.Since
import org.apache.spark.api.java.JavaRDD
import org.apache.spark.api.java.JavaSparkContext.fakeClassTag
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{MIN_FREQUENT_PATTERN_COUNT, NUM_FREQUENT_ITEMS, NUM_LOCAL_FREQUENT_PATTERN, NUM_PREFIXES, NUM_SEQUENCES}
+import org.apache.spark.internal.LogKeys.{MIN_FREQUENT_PATTERN_COUNT, NUM_FREQUENT_ITEMS, NUM_LOCAL_FREQUENT_PATTERN, NUM_PREFIXES, NUM_SEQUENCES}
import org.apache.spark.mllib.util.{Loader, Saveable}
import org.apache.spark.rdd.RDD
import org.apache.spark.sql.{DataFrame, Row, SparkSession}
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/regression/StreamingLinearAlgorithm.scala b/mllib/src/main/scala/org/apache/spark/mllib/regression/StreamingLinearAlgorithm.scala
index 5e4811da0f9c..2fa1339bc72a 100644
--- a/mllib/src/main/scala/org/apache/spark/mllib/regression/StreamingLinearAlgorithm.scala
+++ b/mllib/src/main/scala/org/apache/spark/mllib/regression/StreamingLinearAlgorithm.scala
@@ -22,7 +22,7 @@ import scala.reflect.ClassTag
import org.apache.spark.annotation.Since
import org.apache.spark.api.java.JavaSparkContext.fakeClassTag
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{MODEL_WEIGHTS, TIME}
+import org.apache.spark.internal.LogKeys.{MODEL_WEIGHTS, TIME}
import org.apache.spark.mllib.linalg.Vector
import org.apache.spark.streaming.api.java.{JavaDStream, JavaPairDStream}
import org.apache.spark.streaming.dstream.DStream
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/util/DataValidators.scala b/mllib/src/main/scala/org/apache/spark/mllib/util/DataValidators.scala
index d8c0f8711cab..4857c9b00f42 100644
--- a/mllib/src/main/scala/org/apache/spark/mllib/util/DataValidators.scala
+++ b/mllib/src/main/scala/org/apache/spark/mllib/util/DataValidators.scala
@@ -19,7 +19,7 @@ package org.apache.spark.mllib.util
import org.apache.spark.annotation.Since
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{COUNT, RANGE}
+import org.apache.spark.internal.LogKeys.{COUNT, RANGE}
import org.apache.spark.mllib.regression.LabeledPoint
import org.apache.spark.rdd.RDD
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/util/MLUtils.scala b/mllib/src/main/scala/org/apache/spark/mllib/util/MLUtils.scala
index 10adf10690b7..e23423e4c004 100644
--- a/mllib/src/main/scala/org/apache/spark/mllib/util/MLUtils.scala
+++ b/mllib/src/main/scala/org/apache/spark/mllib/util/MLUtils.scala
@@ -23,7 +23,7 @@ import scala.reflect.ClassTag
import org.apache.spark.{SparkContext, SparkException}
import org.apache.spark.annotation.Since
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.OPTIMIZER_CLASS_NAME
+import org.apache.spark.internal.LogKeys.OPTIMIZER_CLASS_NAME
import org.apache.spark.ml.linalg.{MatrixUDT => MLMatrixUDT, VectorUDT => MLVectorUDT}
import org.apache.spark.ml.util.Instrumentation
import org.apache.spark.mllib.linalg._
diff --git a/mllib/src/test/scala/org/apache/spark/ml/feature/VectorIndexerSuite.scala b/mllib/src/test/scala/org/apache/spark/ml/feature/VectorIndexerSuite.scala
index 0a9347b87977..384fcf6ceb85 100644
--- a/mllib/src/test/scala/org/apache/spark/ml/feature/VectorIndexerSuite.scala
+++ b/mllib/src/test/scala/org/apache/spark/ml/feature/VectorIndexerSuite.scala
@@ -19,7 +19,7 @@ package org.apache.spark.ml.feature
import org.apache.spark.SparkException
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{CATEGORICAL_FEATURES, MAX_CATEGORIES}
+import org.apache.spark.internal.LogKeys.{CATEGORICAL_FEATURES, MAX_CATEGORIES}
import org.apache.spark.ml.attribute._
import org.apache.spark.ml.linalg.{SparseVector, Vector, Vectors}
import org.apache.spark.ml.param.ParamsSuite
diff --git a/mllib/src/test/scala/org/apache/spark/ml/recommendation/ALSSuite.scala b/mllib/src/test/scala/org/apache/spark/ml/recommendation/ALSSuite.scala
index ff94f7beaef7..94abeaf0804e 100644
--- a/mllib/src/test/scala/org/apache/spark/ml/recommendation/ALSSuite.scala
+++ b/mllib/src/test/scala/org/apache/spark/ml/recommendation/ALSSuite.scala
@@ -29,7 +29,7 @@ import org.apache.commons.io.filefilter.TrueFileFilter
import org.apache.spark._
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{RMSE, TEST_SIZE, TRAINING_SIZE}
+import org.apache.spark.internal.LogKeys.{RMSE, TEST_SIZE, TRAINING_SIZE}
import org.apache.spark.ml.linalg.{BLAS, Vectors}
import org.apache.spark.ml.recommendation.ALS._
import org.apache.spark.ml.util.{DefaultReadWriteTest, MLTest, MLTestingUtils}
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/linalg/VectorsSuite.scala b/mllib/src/test/scala/org/apache/spark/mllib/linalg/VectorsSuite.scala
index aa2a91321436..6301638c9a46 100644
--- a/mllib/src/test/scala/org/apache/spark/mllib/linalg/VectorsSuite.scala
+++ b/mllib/src/test/scala/org/apache/spark/mllib/linalg/VectorsSuite.scala
@@ -25,7 +25,7 @@ import breeze.linalg.{DenseMatrix => BDM}
import org.json4s.jackson.JsonMethods.{parse => parseJson}
import org.apache.spark.{SparkConf, SparkException, SparkFunSuite}
-import org.apache.spark.internal.LogKey.MALFORMATTED_STIRNG
+import org.apache.spark.internal.LogKeys.MALFORMATTED_STIRNG
import org.apache.spark.internal.MDC
import org.apache.spark.internal.config.Kryo._
import org.apache.spark.ml.{linalg => newlinalg}
diff --git a/mllib/src/test/scala/org/apache/spark/mllib/tree/GradientBoostedTreesSuite.scala b/mllib/src/test/scala/org/apache/spark/mllib/tree/GradientBoostedTreesSuite.scala
index f5c6abfc66f2..e654eac83649 100644
--- a/mllib/src/test/scala/org/apache/spark/mllib/tree/GradientBoostedTreesSuite.scala
+++ b/mllib/src/test/scala/org/apache/spark/mllib/tree/GradientBoostedTreesSuite.scala
@@ -19,7 +19,7 @@ package org.apache.spark.mllib.tree
import org.apache.spark.SparkFunSuite
import org.apache.spark.internal.{MDC, MessageWithContext}
-import org.apache.spark.internal.LogKey.{LEARNING_RATE, NUM_ITERATIONS, SUBSAMPLING_RATE}
+import org.apache.spark.internal.LogKeys.{LEARNING_RATE, NUM_ITERATIONS, SUBSAMPLING_RATE}
import org.apache.spark.mllib.regression.LabeledPoint
import org.apache.spark.mllib.tree.configuration.{BoostingStrategy, Strategy}
import org.apache.spark.mllib.tree.configuration.Algo._
diff --git a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesConf.scala b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesConf.scala
index 9fdd9518d2d8..a1ef04f4e311 100644
--- a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesConf.scala
+++ b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesConf.scala
@@ -26,7 +26,7 @@ import org.apache.spark.deploy.k8s.Config._
import org.apache.spark.deploy.k8s.Constants._
import org.apache.spark.deploy.k8s.submit._
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{CONFIG, EXECUTOR_ENV_REGEX}
+import org.apache.spark.internal.LogKeys.{CONFIG, EXECUTOR_ENV_REGEX}
import org.apache.spark.internal.config.ConfigEntry
import org.apache.spark.resource.ResourceProfile.DEFAULT_RESOURCE_PROFILE_ID
import org.apache.spark.util.Utils
diff --git a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala
index 50ecefdb6a5d..c2fcfe179bbe 100644
--- a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala
+++ b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala
@@ -33,7 +33,7 @@ import org.apache.spark.annotation.{DeveloperApi, Since, Unstable}
import org.apache.spark.deploy.SparkHadoopUtil
import org.apache.spark.deploy.k8s.Config.KUBERNETES_FILE_UPLOAD_PATH
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.POD_ID
+import org.apache.spark.internal.LogKeys.POD_ID
import org.apache.spark.launcher.SparkLauncher
import org.apache.spark.resource.ResourceUtils
import org.apache.spark.util.{Clock, SystemClock, Utils}
diff --git a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/SparkKubernetesClientFactory.scala b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/SparkKubernetesClientFactory.scala
index abb8a43e3732..79f76e96474e 100644
--- a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/SparkKubernetesClientFactory.scala
+++ b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/SparkKubernetesClientFactory.scala
@@ -33,7 +33,7 @@ import org.apache.spark.SparkConf
import org.apache.spark.annotation.{DeveloperApi, Since, Stable}
import org.apache.spark.deploy.k8s.Config._
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.K8S_CONTEXT
+import org.apache.spark.internal.LogKeys.K8S_CONTEXT
import org.apache.spark.internal.config.ConfigEntry
import org.apache.spark.util.ThreadUtils
diff --git a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/DriverCommandFeatureStep.scala b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/DriverCommandFeatureStep.scala
index de15bf9b24d9..a8706370eead 100644
--- a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/DriverCommandFeatureStep.scala
+++ b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/DriverCommandFeatureStep.scala
@@ -25,7 +25,7 @@ import org.apache.spark.deploy.k8s.Config._
import org.apache.spark.deploy.k8s.Constants._
import org.apache.spark.deploy.k8s.submit._
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{CONFIG, CONFIG2, CONFIG3, CONFIG4, CONFIG5}
+import org.apache.spark.internal.LogKeys.{CONFIG, CONFIG2, CONFIG3, CONFIG4, CONFIG5}
import org.apache.spark.internal.config.{PYSPARK_DRIVER_PYTHON, PYSPARK_PYTHON}
import org.apache.spark.launcher.SparkLauncher
diff --git a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/KubernetesClientApplication.scala b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/KubernetesClientApplication.scala
index c1cef2b84c71..042f984933a7 100644
--- a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/KubernetesClientApplication.scala
+++ b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/KubernetesClientApplication.scala
@@ -32,7 +32,7 @@ import org.apache.spark.deploy.k8s.Config._
import org.apache.spark.deploy.k8s.Constants._
import org.apache.spark.deploy.k8s.KubernetesUtils.addOwnerReference
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{APP_ID, APP_NAME, SUBMISSION_ID}
+import org.apache.spark.internal.LogKeys.{APP_ID, APP_NAME, SUBMISSION_ID}
import org.apache.spark.util.Utils
/**
diff --git a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/KubernetesClientUtils.scala b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/KubernetesClientUtils.scala
index 51f6a2079bf7..930588fb0077 100644
--- a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/KubernetesClientUtils.scala
+++ b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/KubernetesClientUtils.scala
@@ -32,7 +32,7 @@ import org.apache.spark.deploy.k8s.{Config, Constants, KubernetesUtils}
import org.apache.spark.deploy.k8s.Config.{KUBERNETES_DNS_SUBDOMAIN_NAME_MAX_LENGTH, KUBERNETES_NAMESPACE}
import org.apache.spark.deploy.k8s.Constants.ENV_SPARK_CONF_DIR
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{CONFIG, PATH, PATHS}
+import org.apache.spark.internal.LogKeys.{CONFIG, PATH, PATHS}
import org.apache.spark.util.ArrayImplicits._
private[spark] object KubernetesClientUtils extends Logging {
diff --git a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/LoggingPodStatusWatcher.scala b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/LoggingPodStatusWatcher.scala
index 2c6fb8079a17..465c5e605b8c 100644
--- a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/LoggingPodStatusWatcher.scala
+++ b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/LoggingPodStatusWatcher.scala
@@ -24,7 +24,7 @@ import org.apache.spark.deploy.k8s.Config._
import org.apache.spark.deploy.k8s.KubernetesDriverConf
import org.apache.spark.deploy.k8s.KubernetesUtils._
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{APP_ID, APP_NAME, POD_PHASE, POD_STATE, STATUS, SUBMISSION_ID}
+import org.apache.spark.internal.LogKeys.{APP_ID, APP_NAME, POD_PHASE, POD_STATE, STATUS, SUBMISSION_ID}
private[k8s] trait LoggingPodStatusWatcher extends Watcher[Pod] {
def watchOrStop(submissionId: String): Boolean
diff --git a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsAllocator.scala b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsAllocator.scala
index eb70a509d1e7..38303b314082 100644
--- a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsAllocator.scala
+++ b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsAllocator.scala
@@ -33,8 +33,7 @@ import org.apache.spark.deploy.k8s.Config._
import org.apache.spark.deploy.k8s.Constants._
import org.apache.spark.deploy.k8s.KubernetesConf
import org.apache.spark.deploy.k8s.KubernetesUtils.addOwnerReference
-import org.apache.spark.internal.{Logging, LogKey, MDC}
-import org.apache.spark.internal.LogKey.{COUNT, EXECUTOR_IDS, TIMEOUT}
+import org.apache.spark.internal.{Logging, LogKeys, MDC}
import org.apache.spark.internal.config._
import org.apache.spark.resource.ResourceProfile
import org.apache.spark.scheduler.cluster.SchedulerBackendUtils.DEFAULT_NUMBER_EXECUTORS
@@ -145,7 +144,7 @@ class ExecutorPodsAllocator(
onNewSnapshots(applicationId, schedulerBackend, executorPodsSnapshot)
if (failureTracker.numFailedExecutors > maxNumExecutorFailures) {
logError(log"Max number of executor failures " +
- log"(${MDC(LogKey.MAX_EXECUTOR_FAILURES, maxNumExecutorFailures)}) reached")
+ log"(${MDC(LogKeys.MAX_EXECUTOR_FAILURES, maxNumExecutorFailures)}) reached")
stopApplication(EXCEED_MAX_EXECUTOR_FAILURES)
}
}
@@ -211,10 +210,11 @@ class ExecutorPodsAllocator(
}
if (timedOut.nonEmpty) {
- logWarning(log"Executors with ids ${MDC(EXECUTOR_IDS, timedOut.mkString(","))}} were not " +
- log"detected in the Kubernetes cluster after ${MDC(TIMEOUT, podCreationTimeout)} ms " +
- log"despite the fact that a previous allocation attempt tried to create them. " +
- log"The executors may have been deleted but the application missed the deletion event.")
+ logWarning(log"Executors with ids ${MDC(LogKeys.EXECUTOR_IDS, timedOut.mkString(","))}} " +
+ log"were not detected in the Kubernetes cluster after " +
+ log"${MDC(LogKeys.TIMEOUT, podCreationTimeout)} ms despite the fact that a previous " +
+ log"allocation attempt tried to create them. The executors may have been deleted but the " +
+ log"application missed the deletion event.")
newlyCreatedExecutors --= timedOut
if (shouldDeleteExecutors) {
@@ -283,7 +283,7 @@ class ExecutorPodsAllocator(
val newFailedExecutorIds = currentFailedExecutorIds.diff(failedExecutorIds)
if (newFailedExecutorIds.nonEmpty) {
- logWarning(log"${MDC(COUNT, newFailedExecutorIds.size)} new failed executors.")
+ logWarning(log"${MDC(LogKeys.COUNT, newFailedExecutorIds.size)} new failed executors.")
newFailedExecutorIds.foreach { _ => failureTracker.registerExecutorFailure() }
}
failedExecutorIds = failedExecutorIds ++ currentFailedExecutorIds
@@ -343,8 +343,8 @@ class ExecutorPodsAllocator(
val toDelete = newlyCreatedToDelete ++ pendingToDelete
if (toDelete.nonEmpty) {
- logInfo(log"Deleting ${MDC(LogKey.COUNT, toDelete.size)} excess pod requests " +
- log"(${MDC(LogKey.RESOURCE_PROFILE_IDS, toDelete.mkString(","))}).")
+ logInfo(log"Deleting ${MDC(LogKeys.COUNT, toDelete.size)} excess pod requests " +
+ log"(${MDC(LogKeys.RESOURCE_PROFILE_IDS, toDelete.mkString(","))}).")
_deletedExecutorIds = _deletedExecutorIds ++ toDelete
Utils.tryLogNonFatalError {
@@ -398,11 +398,11 @@ class ExecutorPodsAllocator(
val numMissingPodsForRpId = targetNum - podCountForRpId
val numExecutorsToAllocate =
math.min(math.min(numMissingPodsForRpId, podAllocationSize), sharedSlotFromPendingPods)
- logInfo(log"Going to request ${MDC(LogKey.COUNT, numExecutorsToAllocate)} executors from " +
- log"Kubernetes for ResourceProfile Id: ${MDC(LogKey.RESOURCE_PROFILE_ID, rpId)}, " +
- log"target: ${MDC(LogKey.POD_TARGET_COUNT, targetNum)}, " +
- log"known: ${MDC(LogKey.POD_COUNT, podCountForRpId)}, sharedSlotFromPendingPods: " +
- log"${MDC(LogKey.POD_SHARED_SLOT_COUNT, sharedSlotFromPendingPods)}.")
+ logInfo(log"Going to request ${MDC(LogKeys.COUNT, numExecutorsToAllocate)} executors from" +
+ log" Kubernetes for ResourceProfile Id: ${MDC(LogKeys.RESOURCE_PROFILE_ID, rpId)}, " +
+ log"target: ${MDC(LogKeys.POD_TARGET_COUNT, targetNum)}, " +
+ log"known: ${MDC(LogKeys.POD_COUNT, podCountForRpId)}, sharedSlotFromPendingPods: " +
+ log"${MDC(LogKeys.POD_SHARED_SLOT_COUNT, sharedSlotFromPendingPods)}.")
requestNewExecutors(numExecutorsToAllocate, applicationId, rpId, k8sKnownPVCNames)
}
}
@@ -431,8 +431,8 @@ class ExecutorPodsAllocator(
.filterNot(pvc => pvcsInUse.contains(pvc.getMetadata.getName))
.filter(pvc => now - Instant.parse(pvc.getMetadata.getCreationTimestamp).toEpochMilli
> podAllocationDelay)
- logInfo(log"Found ${MDC(LogKey.COUNT, reusablePVCs.size)} reusable PVCs from " +
- log"${MDC(LogKey.TOTAL, createdPVCs.size)} PVCs")
+ logInfo(log"Found ${MDC(LogKeys.COUNT, reusablePVCs.size)} reusable PVCs from " +
+ log"${MDC(LogKeys.TOTAL, createdPVCs.size)} PVCs")
reusablePVCs
} catch {
case _: KubernetesClientException =>
@@ -454,7 +454,7 @@ class ExecutorPodsAllocator(
for ( _ <- 0 until numExecutorsToAllocate) {
if (reusablePVCs.isEmpty && podAllocOnPVC && maxPVCs <= PVC_COUNTER.get()) {
logInfo(
- log"Wait to reuse one of the existing ${MDC(LogKey.COUNT, PVC_COUNTER.get())} PVCs.")
+ log"Wait to reuse one of the existing ${MDC(LogKeys.COUNT, PVC_COUNTER.get())} PVCs.")
return
}
val newExecutorId = EXECUTOR_ID_COUNTER.incrementAndGet()
@@ -486,8 +486,8 @@ class ExecutorPodsAllocator(
}
val pvc = resource.asInstanceOf[PersistentVolumeClaim]
logInfo(log"Trying to create PersistentVolumeClaim " +
- log"${MDC(LogKey.PVC_METADATA_NAME, pvc.getMetadata.getName)} with " +
- log"StorageClass ${MDC(LogKey.CLASS_NAME, pvc.getSpec.getStorageClassName)}")
+ log"${MDC(LogKeys.PVC_METADATA_NAME, pvc.getMetadata.getName)} with " +
+ log"StorageClass ${MDC(LogKeys.CLASS_NAME, pvc.getSpec.getStorageClassName)}")
kubernetesClient.persistentVolumeClaims().inNamespace(namespace).resource(pvc).create()
PVC_COUNTER.incrementAndGet()
}
@@ -526,7 +526,7 @@ class ExecutorPodsAllocator(
val matchedPVC = reusablePVCs.remove(index)
replacedResources.add(pvc)
logInfo(log"Reuse PersistentVolumeClaim " +
- log"${MDC(LogKey.PVC_METADATA_NAME, matchedPVC.getMetadata.getName)}")
+ log"${MDC(LogKeys.PVC_METADATA_NAME, matchedPVC.getMetadata.getName)}")
volume.get.getPersistentVolumeClaim.setClaimName(matchedPVC.getMetadata.getName)
}
}
@@ -542,7 +542,7 @@ class ExecutorPodsAllocator(
} catch {
case e: Exception =>
logError(log"Cannot get the creationTimestamp of the pod: " +
- log"${MDC(LogKey.POD_ID, state.pod)}", e)
+ log"${MDC(LogKeys.POD_ID, state.pod)}", e)
true
}
}
diff --git a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsLifecycleManager.scala b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsLifecycleManager.scala
index 212659d48be8..0d79efa06e49 100644
--- a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsLifecycleManager.scala
+++ b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsLifecycleManager.scala
@@ -31,7 +31,7 @@ import org.apache.spark.deploy.k8s.Config._
import org.apache.spark.deploy.k8s.Constants._
import org.apache.spark.deploy.k8s.KubernetesUtils._
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.EXECUTOR_ID
+import org.apache.spark.internal.LogKeys.EXECUTOR_ID
import org.apache.spark.scheduler.ExecutorExited
import org.apache.spark.util.Utils
diff --git a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsSnapshot.scala b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsSnapshot.scala
index 99cef671d2e4..44daf57d76eb 100644
--- a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsSnapshot.scala
+++ b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsSnapshot.scala
@@ -25,7 +25,7 @@ import io.fabric8.kubernetes.api.model.Pod
import org.apache.spark.deploy.k8s.Constants._
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{POD_NAME, POD_NAMESPACE, POD_PHASE}
+import org.apache.spark.internal.LogKeys.{POD_NAME, POD_NAMESPACE, POD_PHASE}
/**
* An immutable view of the current executor pods that are running in the cluster.
diff --git a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorRollPlugin.scala b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorRollPlugin.scala
index 2a09aebbeb20..e84aae1f27b3 100644
--- a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorRollPlugin.scala
+++ b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorRollPlugin.scala
@@ -27,7 +27,7 @@ import org.apache.spark.api.plugin.{DriverPlugin, ExecutorPlugin, PluginContext,
import org.apache.spark.deploy.k8s.Config.{EXECUTOR_ROLL_INTERVAL, EXECUTOR_ROLL_POLICY, ExecutorRollPolicy, MINIMUM_TASKS_PER_EXECUTOR_BEFORE_ROLLING}
import org.apache.spark.executor.ExecutorMetrics
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{CLASS_NAME, CONFIG, EXECUTOR_ID, INTERVAL}
+import org.apache.spark.internal.LogKeys.{CLASS_NAME, CONFIG, EXECUTOR_ID, INTERVAL}
import org.apache.spark.internal.config.DECOMMISSION_ENABLED
import org.apache.spark.scheduler.ExecutorDecommissionInfo
import org.apache.spark.sql.catalyst.util.DateTimeConstants.MILLIS_PER_SECOND
diff --git a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterManager.scala b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterManager.scala
index 26355b749d54..ddcdc2cf663a 100644
--- a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterManager.scala
+++ b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterManager.scala
@@ -26,7 +26,7 @@ import org.apache.spark.deploy.k8s.{KubernetesConf, KubernetesUtils, SparkKubern
import org.apache.spark.deploy.k8s.Config._
import org.apache.spark.deploy.k8s.Constants.DEFAULT_EXECUTOR_CONTAINER_NAME
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.MASTER_URL
+import org.apache.spark.internal.LogKeys.MASTER_URL
import org.apache.spark.internal.config.TASK_MAX_FAILURES
import org.apache.spark.scheduler.{ExternalClusterManager, SchedulerBackend, TaskScheduler, TaskSchedulerImpl}
import org.apache.spark.scheduler.local.LocalSchedulerBackend
diff --git a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala
index 8a8278b9d076..4e4634504a0f 100644
--- a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala
+++ b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala
@@ -32,7 +32,7 @@ import org.apache.spark.deploy.k8s.Config._
import org.apache.spark.deploy.k8s.Constants._
import org.apache.spark.deploy.k8s.submit.KubernetesClientUtils
import org.apache.spark.deploy.security.HadoopDelegationTokenManager
-import org.apache.spark.internal.LogKey.{COUNT, HOST_PORT, TOTAL}
+import org.apache.spark.internal.LogKeys.{COUNT, HOST_PORT, TOTAL}
import org.apache.spark.internal.MDC
import org.apache.spark.internal.config.SCHEDULER_MIN_REGISTERED_RESOURCES_RATIO
import org.apache.spark.resource.ResourceProfile
diff --git a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/shuffle/KubernetesLocalDiskShuffleExecutorComponents.scala b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/shuffle/KubernetesLocalDiskShuffleExecutorComponents.scala
index 8acd2b5f2aba..7a6fdef7411b 100644
--- a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/shuffle/KubernetesLocalDiskShuffleExecutorComponents.scala
+++ b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/shuffle/KubernetesLocalDiskShuffleExecutorComponents.scala
@@ -27,7 +27,7 @@ import org.apache.commons.io.FileExistsException
import org.apache.spark.{SparkConf, SparkEnv}
import org.apache.spark.deploy.k8s.Config.KUBERNETES_DRIVER_REUSE_PVC
-import org.apache.spark.internal.{Logging, LogKey, MDC}
+import org.apache.spark.internal.{Logging, LogKeys, MDC}
import org.apache.spark.internal.config.{SHUFFLE_CHECKSUM_ALGORITHM, SHUFFLE_CHECKSUM_ENABLED}
import org.apache.spark.shuffle.ShuffleChecksumUtils.{compareChecksums, getChecksumFileName}
import org.apache.spark.shuffle.api.{ShuffleExecutorComponents, ShuffleMapOutputWriter, SingleSpillShuffleMapOutputWriter}
@@ -54,7 +54,7 @@ class KubernetesLocalDiskShuffleExecutorComponents(sparkConf: SparkConf)
KubernetesLocalDiskShuffleExecutorComponents.recoverDiskStore(sparkConf, blockManager)
}
} else {
- logInfo(log"Skip recovery because ${MDC(LogKey.CONFIG, KUBERNETES_DRIVER_REUSE_PVC.key)} " +
+ logInfo(log"Skip recovery because ${MDC(LogKeys.CONFIG, KUBERNETES_DRIVER_REUSE_PVC.key)} " +
log"is disabled.")
}
}
@@ -95,23 +95,23 @@ object KubernetesLocalDiskShuffleExecutorComponents extends Logging {
.partition(_.getName.contains(".checksum"))
val (indexFiles, dataFiles) = files.partition(_.getName.endsWith(".index"))
- logInfo(log"Found ${MDC(LogKey.DATA_FILE_NUM, dataFiles.length)} data files, " +
- log"${MDC(LogKey.INDEX_FILE_NUM, indexFiles.length)} index files, " +
- log"and ${MDC(LogKey.CHECKSUM_FILE_NUM, checksumFiles.length)} checksum files.")
+ logInfo(log"Found ${MDC(LogKeys.DATA_FILE_NUM, dataFiles.length)} data files, " +
+ log"${MDC(LogKeys.INDEX_FILE_NUM, indexFiles.length)} index files, " +
+ log"and ${MDC(LogKeys.CHECKSUM_FILE_NUM, checksumFiles.length)} checksum files.")
// Build a hashmap with checksum file name as a key
val checksumFileMap = new mutable.HashMap[String, File]()
val algorithm = conf.get(SHUFFLE_CHECKSUM_ALGORITHM)
checksumFiles.foreach { f =>
- logInfo(log"${MDC(LogKey.FILE_NAME, f.getName)} -> " +
- log"${MDC(LogKey.FILE_ABSOLUTE_PATH, f.getAbsolutePath)}")
+ logInfo(log"${MDC(LogKeys.FILE_NAME, f.getName)} -> " +
+ log"${MDC(LogKeys.FILE_ABSOLUTE_PATH, f.getAbsolutePath)}")
checksumFileMap.put(f.getName, f)
}
// Build a hashmap with shuffle data file name as a key
val indexFileMap = new mutable.HashMap[String, File]()
indexFiles.foreach { f =>
- logInfo(log"${MDC(LogKey.FILE_NAME, f.getName.replace(".index", ".data"))} -> " +
- log"${MDC(LogKey.FILE_ABSOLUTE_PATH, f.getAbsolutePath)}")
+ logInfo(log"${MDC(LogKeys.FILE_NAME, f.getName.replace(".index", ".data"))} -> " +
+ log"${MDC(LogKeys.FILE_ABSOLUTE_PATH, f.getAbsolutePath)}")
indexFileMap.put(f.getName.replace(".index", ".data"), f)
}
@@ -120,7 +120,7 @@ object KubernetesLocalDiskShuffleExecutorComponents extends Logging {
val level = StorageLevel.DISK_ONLY
val checksumDisabled = !conf.get(SHUFFLE_CHECKSUM_ENABLED)
(dataFiles ++ indexFiles).foreach { f =>
- logInfo(log"Try to recover ${MDC(LogKey.FILE_ABSOLUTE_PATH, f.getAbsolutePath)}")
+ logInfo(log"Try to recover ${MDC(LogKeys.FILE_ABSOLUTE_PATH, f.getAbsolutePath)}")
try {
val id = BlockId(f.getName)
// To make it sure to handle only shuffle blocks
@@ -133,7 +133,7 @@ object KubernetesLocalDiskShuffleExecutorComponents extends Logging {
val decryptedSize = f.length()
bm.TempFileBasedBlockStoreUpdater(id, level, classTag, f, decryptedSize).save()
} else {
- logInfo(log"Ignore ${MDC(LogKey.FILE_ABSOLUTE_PATH, f.getAbsolutePath)} " +
+ logInfo(log"Ignore ${MDC(LogKeys.FILE_ABSOLUTE_PATH, f.getAbsolutePath)} " +
log"due to the verification failure.")
}
} else {
diff --git a/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/DepsTestsSuite.scala b/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/DepsTestsSuite.scala
index 5f95b8daa66c..c0f5e0fe265d 100644
--- a/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/DepsTestsSuite.scala
+++ b/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/DepsTestsSuite.scala
@@ -38,7 +38,7 @@ import org.apache.spark.deploy.k8s.integrationtest.DepsTestsSuite.{DEPS_TIMEOUT,
import org.apache.spark.deploy.k8s.integrationtest.KubernetesSuite._
import org.apache.spark.deploy.k8s.integrationtest.Utils.getExamplesJarName
import org.apache.spark.deploy.k8s.integrationtest.backend.minikube.Minikube
-import org.apache.spark.internal.{LogKey, MDC}
+import org.apache.spark.internal.{LogKeys, MDC}
import org.apache.spark.internal.config.{ARCHIVES, PYSPARK_DRIVER_PYTHON, PYSPARK_PYTHON}
private[spark] trait DepsTestsSuite { k8sSuite: KubernetesSuite =>
@@ -327,7 +327,7 @@ private[spark] trait DepsTestsSuite { k8sSuite: KubernetesSuite =>
s3client.createBucket(createBucketRequest)
} catch {
case e: Exception =>
- logError(log"Failed to create bucket ${MDC(LogKey.BUCKET, BUCKET)}", e)
+ logError(log"Failed to create bucket ${MDC(LogKeys.BUCKET, BUCKET)}", e)
throw new SparkException(s"Failed to create bucket $BUCKET.", e)
}
}
diff --git a/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala b/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
index 19e918c4ff2a..dffb05e196d7 100644
--- a/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
+++ b/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
@@ -43,7 +43,7 @@ import org.apache.spark.deploy.{ExecutorFailureTracker, SparkHadoopUtil}
import org.apache.spark.deploy.history.HistoryServer
import org.apache.spark.deploy.security.HadoopDelegationTokenManager
import org.apache.spark.deploy.yarn.config._
-import org.apache.spark.internal.{Logging, LogKey, MDC}
+import org.apache.spark.internal.{Logging, LogKeys, MDC}
import org.apache.spark.internal.config._
import org.apache.spark.internal.config.UI._
import org.apache.spark.metrics.{MetricsSystem, MetricsSystemInstances}
@@ -219,7 +219,7 @@ private[spark] class ApplicationMaster(
"APPMASTER", sparkConf.get(APP_CALLER_CONTEXT),
Option(appAttemptId.getApplicationId.toString), attemptID).setCurrentContext()
- logInfo(log"ApplicationAttemptId: ${MDC(LogKey.APP_ATTEMPT_ID, appAttemptId)}")
+ logInfo(log"ApplicationAttemptId: ${MDC(LogKeys.APP_ATTEMPT_ID, appAttemptId)}")
// During shutdown, we may not be able to create an FileSystem object. So, pre-create here.
val stagingDirPath = new Path(System.getenv("SPARK_YARN_STAGING_DIR"))
@@ -367,9 +367,9 @@ private[spark] class ApplicationMaster(
final def unregister(status: FinalApplicationStatus, diagnostics: String = null): Unit = {
synchronized {
if (registered && !unregistered) {
- logInfo(log"Unregistering ApplicationMaster with ${MDC(LogKey.APP_STATE, status)}" +
+ logInfo(log"Unregistering ApplicationMaster with ${MDC(LogKeys.APP_STATE, status)}" +
Option(diagnostics).map(
- msg => log" (diag message: ${MDC(LogKey.MESSAGE, msg)})").getOrElse(log""))
+ msg => log" (diag message: ${MDC(LogKeys.MESSAGE, msg)})").getOrElse(log""))
unregistered = true
client.unregister(status, Option(diagnostics).getOrElse(""))
}
@@ -387,9 +387,9 @@ private[spark] class ApplicationMaster(
finalStatus = FinalApplicationStatus.FAILED
exitCode = ApplicationMaster.EXIT_SC_NOT_INITED
}
- logInfo(log"Final app status: ${MDC(LogKey.APP_STATE, finalStatus)}, " +
- log"exitCode: ${MDC(LogKey.EXIT_CODE, exitCode)}" +
- Option(msg).map(msg => log", (reason: ${MDC(LogKey.REASON, msg)})").getOrElse(log""))
+ logInfo(log"Final app status: ${MDC(LogKeys.APP_STATE, finalStatus)}, " +
+ log"exitCode: ${MDC(LogKeys.EXIT_CODE, exitCode)}" +
+ Option(msg).map(msg => log", (reason: ${MDC(LogKeys.REASON, msg)})").getOrElse(log""))
finalMsg = ComStrUtils.abbreviate(msg, sparkConf.get(AM_FINAL_MSG_LIMIT).toInt)
finished = true
if (!inShutdown && Thread.currentThread() != reporterThread && reporterThread != null) {
@@ -483,7 +483,7 @@ private[spark] class ApplicationMaster(
rpcEnv.setupEndpoint("YarnAM", new AMEndpoint(rpcEnv, driverRef))
if (_sparkConf.get(SHUFFLE_SERVICE_ENABLED)) {
logInfo(log"Initializing service data for shuffle service using name '" +
- log"${MDC(LogKey.SHUFFLE_SERVICE_NAME, _sparkConf.get(SHUFFLE_SERVICE_NAME))}'")
+ log"${MDC(LogKeys.SHUFFLE_SERVICE_NAME, _sparkConf.get(SHUFFLE_SERVICE_NAME))}'")
}
allocator.allocateResources()
val ms = MetricsSystem.createMetricsSystem(MetricsSystemInstances.APPLICATION_MASTER, sparkConf)
@@ -598,8 +598,8 @@ private[spark] class ApplicationMaster(
ApplicationMaster.EXIT_REPORTER_FAILURE, "Exception was thrown " +
s"$failureCount time(s) from Reporter thread.")
} else {
- logWarning(
- log"Reporter thread fails ${MDC(LogKey.FAILURES, failureCount)} time(s) in a row.", e)
+ logWarning(log"Reporter thread fails ${MDC(LogKeys.FAILURES, failureCount)} " +
+ log"time(s) in a row.", e)
}
}
try {
@@ -658,8 +658,8 @@ private[spark] class ApplicationMaster(
t.setName("Reporter")
t.start()
logInfo(log"Started progress reporter thread with " +
- log"(heartbeat: ${MDC(LogKey.HEARTBEAT_INTERVAL, heartbeatInterval)}, initial allocation: " +
- log"${MDC(LogKey.INITIAL_HEARTBEAT_INTERVAL, initialAllocationInterval)}) intervals")
+ log"(heartbeat: ${MDC(LogKeys.HEARTBEAT_INTERVAL, heartbeatInterval)}, initial allocation: " +
+ log"${MDC(LogKeys.INITIAL_HEARTBEAT_INTERVAL, initialAllocationInterval)}) intervals")
t
}
@@ -685,7 +685,7 @@ private[spark] class ApplicationMaster(
try {
val preserveFiles = sparkConf.get(PRESERVE_STAGING_FILES)
if (!preserveFiles) {
- logInfo(log"Deleting staging directory ${MDC(LogKey.PATH, stagingDirPath)}")
+ logInfo(log"Deleting staging directory ${MDC(LogKeys.PATH, stagingDirPath)}")
fs.delete(stagingDirPath, true)
}
} catch {
@@ -750,7 +750,7 @@ private[spark] class ApplicationMaster(
// Reporter thread can interrupt to stop user class
case SparkUserAppException(exitCode) =>
val msg = log"User application exited with status " +
- log"${MDC(LogKey.EXIT_CODE, exitCode)}"
+ log"${MDC(LogKeys.EXIT_CODE, exitCode)}"
logError(msg)
finish(FinalApplicationStatus.FAILED, exitCode, msg.message)
case cause: Throwable =>
@@ -834,7 +834,7 @@ private[spark] class ApplicationMaster(
case KillExecutors(executorIds) =>
logInfo(log"Driver requested to kill executor(s) " +
- log"${MDC(LogKey.EXECUTOR_IDS, executorIds.mkString(", "))}.")
+ log"${MDC(LogKeys.EXECUTOR_IDS, executorIds.mkString(", "))}.")
Option(allocator) match {
case Some(a) => executorIds.foreach(a.killExecutor)
case None => logWarning("Container allocator is not ready to kill executors yet.")
@@ -858,11 +858,11 @@ private[spark] class ApplicationMaster(
if (shutdown || !clientModeTreatDisconnectAsFailed) {
if (exitCode == 0) {
logInfo(log"Driver terminated or disconnected! Shutting down. " +
- log"${MDC(LogKey.HOST_PORT, remoteAddress)}")
+ log"${MDC(LogKeys.HOST_PORT, remoteAddress)}")
finish(FinalApplicationStatus.SUCCEEDED, ApplicationMaster.EXIT_SUCCESS)
} else {
- logError(log"Driver terminated with exit code ${MDC(LogKey.EXIT_CODE, exitCode)}! " +
- log"Shutting down. ${MDC(LogKey.HOST_PORT, remoteAddress)}")
+ logError(log"Driver terminated with exit code ${MDC(LogKeys.EXIT_CODE, exitCode)}! " +
+ log"Shutting down. ${MDC(LogKeys.HOST_PORT, remoteAddress)}")
finish(FinalApplicationStatus.FAILED, exitCode)
}
} else {
diff --git a/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala b/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala
index 8d6a014d16bf..0567d8efb85e 100644
--- a/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala
+++ b/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala
@@ -55,7 +55,7 @@ import org.apache.spark.deploy.security.HadoopDelegationTokenManager
import org.apache.spark.deploy.yarn.ResourceRequestHelper._
import org.apache.spark.deploy.yarn.YarnSparkHadoopUtil._
import org.apache.spark.deploy.yarn.config._
-import org.apache.spark.internal.{Logging, LogKey, MDC}
+import org.apache.spark.internal.{Logging, LogKeys, MDC}
import org.apache.spark.internal.config._
import org.apache.spark.internal.config.Python._
import org.apache.spark.launcher.{JavaModuleOptions, LauncherBackend, SparkAppHandle, YarnCommandBuilderUtils}
@@ -141,8 +141,8 @@ private[spark] class Client(
val principal = sparkConf.get(PRINCIPAL).orNull
require((principal == null) == (keytab == null),
"Both principal and keytab must be defined, or neither.")
- logInfo(log"Kerberos credentials: principal = ${MDC(LogKey.PRINCIPAL, principal)}, " +
- log"keytab = ${MDC(LogKey.KEYTAB, keytab)}")
+ logInfo(log"Kerberos credentials: principal = ${MDC(LogKeys.PRINCIPAL, principal)}, " +
+ log"keytab = ${MDC(LogKeys.KEYTAB, keytab)}")
// Generate a file name that can be used for the keytab file, that does not conflict
// with any user file.
Some(new File(keytab).getName() + "-" + UUID.randomUUID().toString)
@@ -229,7 +229,7 @@ private[spark] class Client(
val appContext = createApplicationSubmissionContext(newApp, containerContext)
// Finally, submit and monitor the application
- logInfo(log"Submitting application ${MDC(LogKey.APP_ID, appId)} to ResourceManager")
+ logInfo(log"Submitting application ${MDC(LogKeys.APP_ID, appId)} to ResourceManager")
yarnClient.submitApplication(appContext)
launcherBackend.setAppId(appId.toString)
reportLauncherState(SparkAppHandle.State.SUBMITTED)
@@ -254,11 +254,11 @@ private[spark] class Client(
try {
val fs = stagingDirPath.getFileSystem(hadoopConf)
if (fs.delete(stagingDirPath, true)) {
- logInfo(log"Deleted staging directory ${MDC(LogKey.PATH, stagingDirPath)}")
+ logInfo(log"Deleted staging directory ${MDC(LogKeys.PATH, stagingDirPath)}")
}
} catch {
case ioe: IOException =>
- logWarning(log"Failed to cleanup staging dir ${MDC(LogKey.PATH, stagingDirPath)}", ioe)
+ logWarning(log"Failed to cleanup staging dir ${MDC(LogKeys.PATH, stagingDirPath)}", ioe)
}
}
@@ -332,7 +332,7 @@ private[spark] class Client(
appContext.setLogAggregationContext(logAggregationContext)
} catch {
case NonFatal(e) =>
- logWarning(log"Ignoring ${MDC(LogKey.CONFIG, ROLLED_LOG_INCLUDE_PATTERN.key)}} " +
+ logWarning(log"Ignoring ${MDC(LogKeys.CONFIG, ROLLED_LOG_INCLUDE_PATTERN.key)}} " +
log"because the version of YARN does not support it", e)
}
}
@@ -372,15 +372,15 @@ private[spark] class Client(
// for fetching delegation tokens. See YARN-5910 for more details.
sparkConf.get(config.AM_TOKEN_CONF_REGEX).foreach { regex =>
logInfo(log"Processing token conf (spark.yarn.am.tokenConfRegex) with " +
- log"regex ${MDC(LogKey.TOKEN_REGEX, regex)}")
+ log"regex ${MDC(LogKeys.TOKEN_REGEX, regex)}")
val dob = new DataOutputBuffer()
val copy = new Configuration(false)
copy.clear()
hadoopConf.asScala.foreach { entry =>
if (entry.getKey.matches(regex)) {
copy.set(entry.getKey, entry.getValue)
- logInfo(log"Captured key: ${MDC(LogKey.KEY, entry.getKey)} -> " +
- log"value: ${MDC(LogKey.VALUE, entry.getValue)}")
+ logInfo(log"Captured key: ${MDC(LogKeys.KEY, entry.getKey)} -> " +
+ log"value: ${MDC(LogKeys.VALUE, entry.getValue)}")
}
}
copy.write(dob);
@@ -406,7 +406,7 @@ private[spark] class Client(
private def verifyClusterResources(newAppResponse: GetNewApplicationResponse): Unit = {
val maxMem = newAppResponse.getMaximumResourceCapability.getMemorySize
logInfo(log"Verifying our application has not requested more than the maximum memory " +
- log"capability of the cluster (${MDC(LogKey.MAX_MEMORY_SIZE, maxMem)} MB per container)")
+ log"capability of the cluster (${MDC(LogKeys.MAX_MEMORY_SIZE, maxMem)} MB per container)")
val executorMem =
executorMemory + executorOffHeapMemory + executorMemoryOverhead + pysparkWorkerMemory
if (executorMem > maxMem) {
@@ -423,8 +423,8 @@ private[spark] class Client(
"Please check the values of 'yarn.scheduler.maximum-allocation-mb' and/or " +
"'yarn.nodemanager.resource.memory-mb'.")
}
- logInfo(log"Will allocate AM container, with ${MDC(LogKey.MEMORY_SIZE, amMem)} MB memory " +
- log"including ${MDC(LogKey.OVERHEAD_MEMORY_SIZE, amMemoryOverhead)} MB overhead")
+ logInfo(log"Will allocate AM container, with ${MDC(LogKeys.MEMORY_SIZE, amMem)} MB memory " +
+ log"including ${MDC(LogKeys.OVERHEAD_MEMORY_SIZE, amMemoryOverhead)} MB overhead")
// We could add checks to make sure the entire cluster has enough resources but that involves
// getting all the node reports and computing ourselves.
@@ -448,8 +448,8 @@ private[spark] class Client(
var destPath = srcPath
if (force || !compareFs(srcFs, destFs) || "file".equals(srcFs.getScheme)) {
destPath = new Path(destDir, destName.getOrElse(srcPath.getName()))
- logInfo(log"Uploading resource ${MDC(LogKey.SRC_PATH, srcPath)} -> " +
- log"${MDC(LogKey.TARGET_PATH, destPath)}")
+ logInfo(log"Uploading resource ${MDC(LogKeys.SRC_PATH, srcPath)} -> " +
+ log"${MDC(LogKeys.TARGET_PATH, destPath)}")
try {
FileUtil.copy(srcFs, srcPath, destFs, destPath, false, hadoopConf)
} catch {
@@ -461,7 +461,7 @@ private[spark] class Client(
destFs.setPermission(destPath, new FsPermission(APP_FILE_PERMISSION))
} else {
logInfo(log"Source and destination file systems are the same. " +
- log"Not copying ${MDC(LogKey.SRC_PATH, srcPath)}")
+ log"Not copying ${MDC(LogKeys.SRC_PATH, srcPath)}")
}
// Resolve any symlinks in the URI path so using a "current" symlink to point to a specific
// version shows the specific version in the distributed cache configuration
@@ -561,11 +561,11 @@ private[spark] class Client(
val uriStr = uri.toString()
val fileName = new File(uri.getPath).getName
if (distributedUris.contains(uriStr)) {
- logWarning(log"Same path resource ${MDC(LogKey.URI, uri)} added multiple times " +
+ logWarning(log"Same path resource ${MDC(LogKeys.URI, uri)} added multiple times " +
log"to distributed cache.")
false
} else if (distributedNames.contains(fileName)) {
- logWarning(log"Same name resource ${MDC(LogKey.URI, uri)} added multiple times " +
+ logWarning(log"Same name resource ${MDC(LogKeys.URI, uri)} added multiple times " +
log"to distributed cache")
false
} else {
@@ -704,8 +704,8 @@ private[spark] class Client(
case None =>
// No configuration, so fall back to uploading local jar files.
logWarning(
- log"Neither ${MDC(LogKey.CONFIG, SPARK_JARS.key)} nor " +
- log"${MDC(LogKey.CONFIG2, SPARK_ARCHIVE.key)}} is set, falling back to uploading " +
+ log"Neither ${MDC(LogKeys.CONFIG, SPARK_JARS.key)} nor " +
+ log"${MDC(LogKeys.CONFIG2, SPARK_ARCHIVE.key)}} is set, falling back to uploading " +
log"libraries under SPARK_HOME.")
val jarsDir = new File(YarnCommandBuilderUtils.findJarsDir(
sparkConf.getenv("SPARK_HOME")))
@@ -885,7 +885,7 @@ private[spark] class Client(
if (dir.isDirectory()) {
val files = dir.listFiles()
if (files == null) {
- logWarning(log"Failed to list files under directory ${MDC(LogKey.PATH, dir)}")
+ logWarning(log"Failed to list files under directory ${MDC(LogKeys.PATH, dir)}")
} else {
files.foreach { file =>
if (file.isFile && !hadoopConfFiles.contains(file.getName())) {
@@ -1074,7 +1074,7 @@ private[spark] class Client(
sparkConf))
}
if (sparkConf.get(AM_JAVA_OPTIONS).isDefined) {
- logWarning(log"${MDC(LogKey.CONFIG, AM_JAVA_OPTIONS.key)} will not take effect " +
+ logWarning(log"${MDC(LogKeys.CONFIG, AM_JAVA_OPTIONS.key)} will not take effect " +
log"in cluster mode")
}
} else {
@@ -1207,7 +1207,7 @@ private[spark] class Client(
getApplicationReport()
} catch {
case e: ApplicationNotFoundException =>
- logError(log"Application ${MDC(LogKey.APP_ID, appId)} not found.")
+ logError(log"Application ${MDC(LogKeys.APP_ID, appId)} not found.")
cleanupStagingDir()
return YarnAppReport(YarnApplicationState.KILLED, FinalApplicationStatus.KILLED, None)
case NonFatal(e) if !e.isInstanceOf[InterruptedIOException] =>
@@ -1221,8 +1221,8 @@ private[spark] class Client(
reportsSinceLastLog += 1
if (logApplicationReport) {
if (lastState != state || reportsSinceLastLog >= reportsTillNextLog) {
- logInfo(log"Application report for ${MDC(LogKey.APP_ID, appId)} " +
- log"(state: ${MDC(LogKey.APP_STATE, state)})")
+ logInfo(log"Application report for ${MDC(LogKeys.APP_ID, appId)} " +
+ log"(state: ${MDC(LogKeys.APP_STATE, state)})")
reportsSinceLastLog = 0
}
@@ -1231,7 +1231,7 @@ private[spark] class Client(
if (log.isDebugEnabled) {
logDebug(formatReportDetails(report, getDriverLogsLink(report)))
} else if (lastState != state) {
- logInfo(log"${MDC(LogKey.REPORT_DETAILS,
+ logInfo(log"${MDC(LogKeys.REPORT_DETAILS,
formatReportDetails(report, getDriverLogsLink(report)))}")
}
}
@@ -1354,7 +1354,7 @@ private[spark] class Client(
.getOrElse(IMap.empty)
} catch {
case e: Exception =>
- logWarning(log"Unable to get driver log links for ${MDC(LogKey.APP_ID, appId)}: ", e)
+ logWarning(log"Unable to get driver log links for ${MDC(LogKeys.APP_ID, appId)}: ", e)
// Include the full stack trace only at DEBUG level to reduce verbosity
logDebug(s"Unable to get driver log links for $appId", e)
IMap.empty
@@ -1374,9 +1374,9 @@ private[spark] class Client(
if (!launcherBackend.isConnected() && fireAndForget) {
val report = getApplicationReport()
val state = report.getYarnApplicationState
- logInfo(log"Application report for ${MDC(LogKey.APP_ID, appId)} " +
- log"(state: ${MDC(LogKey.APP_STATE, state)})")
- logInfo(log"${MDC(LogKey.REPORT_DETAILS,
+ logInfo(log"Application report for ${MDC(LogKeys.APP_ID, appId)} " +
+ log"(state: ${MDC(LogKeys.APP_STATE, state)})")
+ logInfo(log"${MDC(LogKeys.REPORT_DETAILS,
formatReportDetails(report, getDriverLogsLink(report)))}")
if (state == YarnApplicationState.FAILED || state == YarnApplicationState.KILLED) {
throw new SparkException(s"Application $appId finished with status: $state")
diff --git a/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala b/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala
index 81588b2fcdda..62753f35ae76 100644
--- a/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala
+++ b/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala
@@ -20,7 +20,7 @@ package org.apache.spark.deploy.yarn
import scala.collection.mutable.ArrayBuffer
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.ARGS
+import org.apache.spark.internal.LogKeys.ARGS
// TODO: Add code and support for ensuring that yarn resource 'tasks' are location aware !
private[spark] class ClientArguments(args: Array[String]) extends Logging {
diff --git a/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala b/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala
index 035f02f16fbe..983ab5b4341b 100644
--- a/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala
+++ b/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala
@@ -39,7 +39,7 @@ import org.apache.hadoop.yarn.util.Records
import org.apache.spark.{SecurityManager, SparkConf, SparkException}
import org.apache.spark.internal.{Logging, MDC, MessageWithContext}
-import org.apache.spark.internal.LogKey.{EXECUTOR_ENVS, EXECUTOR_LAUNCH_COMMANDS, EXECUTOR_RESOURCES}
+import org.apache.spark.internal.LogKeys.{EXECUTOR_ENVS, EXECUTOR_LAUNCH_COMMANDS, EXECUTOR_RESOURCES}
import org.apache.spark.internal.config._
import org.apache.spark.network.util.JavaUtils
import org.apache.spark.util.Utils
diff --git a/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ResourceRequestHelper.scala b/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ResourceRequestHelper.scala
index 755a69520ce4..a747f99f1b85 100644
--- a/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ResourceRequestHelper.scala
+++ b/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ResourceRequestHelper.scala
@@ -26,7 +26,7 @@ import org.apache.hadoop.yarn.exceptions.ResourceNotFoundException
import org.apache.spark.{SparkConf, SparkException}
import org.apache.spark.deploy.yarn.config._
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{ERROR, RESOURCE_NAME}
+import org.apache.spark.internal.LogKeys.{ERROR, RESOURCE_NAME}
import org.apache.spark.internal.config._
import org.apache.spark.resource.ResourceID
import org.apache.spark.resource.ResourceUtils.{AMOUNT, FPGA, GPU}
diff --git a/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/SparkRackResolver.scala b/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/SparkRackResolver.scala
index 16f7581c7af4..618f0dc8a4da 100644
--- a/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/SparkRackResolver.scala
+++ b/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/SparkRackResolver.scala
@@ -30,7 +30,7 @@ import org.apache.logging.log4j.{Level, LogManager}
import org.apache.logging.log4j.core.Logger
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.NODE_LOCATION
+import org.apache.spark.internal.LogKeys.NODE_LOCATION
/**
* Re-implement YARN's [[RackResolver]] for hadoop releases without YARN-9332.
diff --git a/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala b/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala
index b410ea2eb44d..29cf8ccc20f9 100644
--- a/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala
+++ b/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala
@@ -41,7 +41,7 @@ import org.apache.spark.deploy.yarn.YarnSparkHadoopUtil._
import org.apache.spark.deploy.yarn.config._
import org.apache.spark.executor.ExecutorExitCode
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey
+import org.apache.spark.internal.LogKeys
import org.apache.spark.internal.config._
import org.apache.spark.resource.ResourceProfile
import org.apache.spark.resource.ResourceProfile.DEFAULT_RESOURCE_PROFILE_ID
@@ -195,7 +195,7 @@ private[yarn] class YarnAllocator(
case (true, false) => true
case (true, true) =>
logWarning(log"Yarn Executor Decommissioning is supported only " +
- log"when ${MDC(LogKey.CONFIG, SHUFFLE_SERVICE_ENABLED.key)} is set to false. " +
+ log"when ${MDC(LogKeys.CONFIG, SHUFFLE_SERVICE_ENABLED.key)} is set to false. " +
log"See: SPARK-39018.")
false
case (false, _) => false
@@ -314,7 +314,7 @@ private[yarn] class YarnAllocator(
if (!rpIdToYarnResource.containsKey(rp.id)) {
// track the resource profile if not already there
getOrUpdateRunningExecutorForRPId(rp.id)
- logInfo(log"Resource profile ${MDC(LogKey.RESOURCE_PROFILE_ID, rp.id)} doesn't exist, " +
+ logInfo(log"Resource profile ${MDC(LogKeys.RESOURCE_PROFILE_ID, rp.id)} doesn't exist, " +
log"adding it")
val resourcesWithDefaults =
@@ -401,8 +401,8 @@ private[yarn] class YarnAllocator(
val res = resourceProfileToTotalExecs.map { case (rp, numExecs) =>
createYarnResourceForResourceProfile(rp)
if (numExecs != getOrUpdateTargetNumExecutorsForRPId(rp.id)) {
- logInfo(log"Driver requested a total number of ${MDC(LogKey.COUNT, numExecs)} " +
- log"executor(s) for resource profile id: ${MDC(LogKey.RESOURCE_PROFILE_ID, rp.id)}.")
+ logInfo(log"Driver requested a total number of ${MDC(LogKeys.COUNT, numExecs)} " +
+ log"executor(s) for resource profile id: ${MDC(LogKeys.RESOURCE_PROFILE_ID, rp.id)}.")
targetNumExecutorsPerResourceProfileId(rp.id) = numExecs
allocatorNodeHealthTracker.setSchedulerExcludedNodes(excludedNodes)
true
@@ -424,7 +424,7 @@ private[yarn] class YarnAllocator(
internalReleaseContainer(container)
getOrUpdateRunningExecutorForRPId(rpId).remove(executorId)
case _ => logWarning(log"Attempted to kill unknown executor " +
- log"${MDC(LogKey.EXECUTOR_ID, executorId)}!")
+ log"${MDC(LogKeys.EXECUTOR_ID, executorId)}!")
}
}
@@ -523,13 +523,13 @@ private[yarn] class YarnAllocator(
if (missing > 0) {
val resource = rpIdToYarnResource.get(rpId)
if (log.isInfoEnabled()) {
- var requestContainerMessage = log"Will request ${MDC(LogKey.COUNT, missing)} executor " +
- log"container(s) for ResourceProfile Id: ${MDC(LogKey.RESOURCE_PROFILE_ID, rpId)}, " +
- log"each with ${MDC(LogKey.VIRTUAL_CORES, resource.getVirtualCores)} core(s) and " +
- log"${MDC(LogKey.MEMORY_SIZE, resource.getMemorySize)} MB memory."
+ var requestContainerMessage = log"Will request ${MDC(LogKeys.COUNT, missing)} executor " +
+ log"container(s) for ResourceProfile Id: ${MDC(LogKeys.RESOURCE_PROFILE_ID, rpId)}, " +
+ log"each with ${MDC(LogKeys.VIRTUAL_CORES, resource.getVirtualCores)} core(s) and " +
+ log"${MDC(LogKeys.MEMORY_SIZE, resource.getMemorySize)} MB memory."
if (resource.getResources.nonEmpty) {
requestContainerMessage = requestContainerMessage +
- log" with custom resources: ${MDC(LogKey.RESOURCE, resource)}"
+ log" with custom resources: ${MDC(LogKeys.RESOURCE, resource)}"
}
logInfo(requestContainerMessage)
}
@@ -540,7 +540,7 @@ private[yarn] class YarnAllocator(
}
val cancelledContainers = staleRequests.size
if (cancelledContainers > 0) {
- logInfo(log"Canceled ${MDC(LogKey.COUNT, cancelledContainers)} container request(s) " +
+ logInfo(log"Canceled ${MDC(LogKeys.COUNT, cancelledContainers)} container request(s) " +
log"(locality no longer needed)")
}
@@ -575,7 +575,7 @@ private[yarn] class YarnAllocator(
amClient.removeContainerRequest(nonLocal)
}
if (numToCancel > 0) {
- logInfo(log"Canceled ${MDC(LogKey.COUNT, numToCancel)} unlocalized container " +
+ logInfo(log"Canceled ${MDC(LogKeys.COUNT, numToCancel)} unlocalized container " +
log"requests to resubmit with locality")
}
}
@@ -587,19 +587,19 @@ private[yarn] class YarnAllocator(
if (log.isInfoEnabled()) {
val (localized, anyHost) = newLocalityRequests.partition(_.getNodes() != null)
if (anyHost.nonEmpty) {
- logInfo(log"Submitted ${MDC(LogKey.COUNT, anyHost.size)}} unlocalized container " +
+ logInfo(log"Submitted ${MDC(LogKeys.COUNT, anyHost.size)}} unlocalized container " +
log"requests.")
}
localized.foreach { request =>
logInfo(log"Submitted container request for host " +
- log"${MDC(LogKey.HOST, hostStr(request))}.")
+ log"${MDC(LogKeys.HOST, hostStr(request))}.")
}
}
} else if (numPendingAllocate > 0 && missing < 0) {
val numToCancel = math.min(numPendingAllocate, -missing)
- logInfo(log"Canceling requests for ${MDC(LogKey.COUNT, numToCancel)} executor " +
+ logInfo(log"Canceling requests for ${MDC(LogKeys.COUNT, numToCancel)} executor " +
log"container(s) to have a new desired total " +
- log"${MDC(LogKey.EXECUTOR_DESIRED_COUNT,
+ log"${MDC(LogKeys.EXECUTOR_DESIRED_COUNT,
getOrUpdateTargetNumExecutorsForRPId(rpId))} executors.")
// cancel pending allocate requests by taking locality preference into account
val cancelRequests = (staleRequests ++ anyHostRequests ++ localRequests).take(numToCancel)
@@ -706,8 +706,8 @@ private[yarn] class YarnAllocator(
runAllocatedContainers(containersToUse)
- logInfo(log"Received ${MDC(LogKey.COUNT, allocatedContainers.size)} containers from YARN, " +
- log"launching executors on ${MDC(LogKey.EXECUTOR_LAUNCH_COUNT, containersToUse.size)} " +
+ logInfo(log"Received ${MDC(LogKeys.COUNT, allocatedContainers.size)} containers from YARN, " +
+ log"launching executors on ${MDC(LogKeys.EXECUTOR_LAUNCH_COUNT, containersToUse.size)} " +
log"of them.")
}
@@ -761,10 +761,10 @@ private[yarn] class YarnAllocator(
val executorId = executorIdCounter.toString
val yarnResourceForRpId = rpIdToYarnResource.get(rpId)
assert(container.getResource.getMemorySize >= yarnResourceForRpId.getMemorySize)
- logInfo(log"Launching container ${MDC(LogKey.CONTAINER_ID, containerId)} " +
- log"on host ${MDC(LogKey.HOST, executorHostname)} for " +
- log"executor with ID ${MDC(LogKey.EXECUTOR_ID, executorId)} for " +
- log"ResourceProfile Id ${MDC(LogKey.RESOURCE_PROFILE_ID, rpId)}")
+ logInfo(log"Launching container ${MDC(LogKeys.CONTAINER_ID, containerId)} " +
+ log"on host ${MDC(LogKeys.HOST, executorHostname)} for " +
+ log"executor with ID ${MDC(LogKeys.EXECUTOR_ID, executorId)} for " +
+ log"ResourceProfile Id ${MDC(LogKeys.RESOURCE_PROFILE_ID, rpId)}")
val rp = rpIdToResourceProfile(rpId)
val defaultResources = ResourceProfile.getDefaultProfileExecutorResources(sparkConf)
@@ -802,8 +802,8 @@ private[yarn] class YarnAllocator(
getOrUpdateNumExecutorsStartingForRPId(rpId).decrementAndGet()
launchingExecutorContainerIds.remove(containerId)
if (NonFatal(e)) {
- logError(log"Failed to launch executor ${MDC(LogKey.EXECUTOR_ID, executorId)} " +
- log"on container ${MDC(LogKey.CONTAINER_ID, containerId)}", e)
+ logError(log"Failed to launch executor ${MDC(LogKeys.EXECUTOR_ID, executorId)} " +
+ log"on container ${MDC(LogKeys.CONTAINER_ID, containerId)}", e)
// Assigned container should be released immediately
// to avoid unnecessary resource occupation.
amClient.releaseAssignedContainer(containerId)
@@ -818,8 +818,8 @@ private[yarn] class YarnAllocator(
}
} else {
logInfo(log"Skip launching executorRunnable as running executors count: " +
- log"${MDC(LogKey.COUNT, rpRunningExecs)} reached target executors count: " +
- log"${MDC(LogKey.EXECUTOR_TARGET_COUNT, getOrUpdateTargetNumExecutorsForRPId(rpId))}.")
+ log"${MDC(LogKeys.COUNT, rpRunningExecs)} reached target executors count: " +
+ log"${MDC(LogKeys.EXECUTOR_TARGET_COUNT, getOrUpdateTargetNumExecutorsForRPId(rpId))}.")
}
}
}
@@ -861,38 +861,38 @@ private[yarn] class YarnAllocator(
case Some((executorId, _)) =>
getOrUpdateRunningExecutorForRPId(rpId).remove(executorId)
case None => logWarning(log"Cannot find executorId for container: " +
- log"${MDC(LogKey.CONTAINER_ID, containerId)}")
+ log"${MDC(LogKeys.CONTAINER_ID, containerId)}")
}
- logInfo(log"Completed container ${MDC(LogKey.CONTAINER_ID, containerId)}" +
- log"${MDC(LogKey.HOST, onHostStr)} " +
- log"(state: ${MDC(LogKey.CONTAINER_STATE, completedContainer.getState)}, " +
- log"exit status: ${MDC(LogKey.EXIT_CODE, completedContainer.getExitStatus)}")
+ logInfo(log"Completed container ${MDC(LogKeys.CONTAINER_ID, containerId)}" +
+ log"${MDC(LogKeys.HOST, onHostStr)} " +
+ log"(state: ${MDC(LogKeys.CONTAINER_STATE, completedContainer.getState)}, " +
+ log"exit status: ${MDC(LogKeys.EXIT_CODE, completedContainer.getExitStatus)}")
val exitStatus = completedContainer.getExitStatus
val (exitCausedByApp, containerExitReason) = exitStatus match {
case _ if shutdown =>
- (false, log"Executor for container ${MDC(LogKey.CONTAINER_ID, containerId)} " +
+ (false, log"Executor for container ${MDC(LogKeys.CONTAINER_ID, containerId)} " +
log"exited after Application shutdown.")
case ContainerExitStatus.SUCCESS =>
- (false, log"Executor for container ${MDC(LogKey.CONTAINER_ID, containerId)} " +
+ (false, log"Executor for container ${MDC(LogKeys.CONTAINER_ID, containerId)} " +
log"exited because of a YARN event (e.g., preemption) and not because of an " +
log"error in the running job.")
case ContainerExitStatus.PREEMPTED =>
// Preemption is not the fault of the running tasks, since YARN preempts containers
// merely to do resource sharing, and tasks that fail due to preempted executors could
// just as easily finish on any other executor. See SPARK-8167.
- (false, log"Container ${MDC(LogKey.CONTAINER_ID, containerId)}" +
- log"${MDC(LogKey.HOST, onHostStr)} was preempted.")
+ (false, log"Container ${MDC(LogKeys.CONTAINER_ID, containerId)}" +
+ log"${MDC(LogKeys.HOST, onHostStr)} was preempted.")
// Should probably still count memory exceeded exit codes towards task failures
case ContainerExitStatus.KILLED_EXCEEDED_VMEM =>
val vmemExceededPattern = raw"$MEM_REGEX of $MEM_REGEX virtual memory used".r
val diag = vmemExceededPattern.findFirstIn(completedContainer.getDiagnostics)
.map(_.concat(".")).getOrElse("")
val message = log"Container killed by YARN for exceeding virtual memory limits. " +
- log"${MDC(LogKey.ERROR, diag)} Consider boosting " +
- log"${MDC(LogKey.CONFIG, EXECUTOR_MEMORY_OVERHEAD.key)} or boosting " +
- log"${MDC(LogKey.CONFIG2, YarnConfiguration.NM_VMEM_PMEM_RATIO)} or disabling " +
- log"${MDC(LogKey.CONFIG3, YarnConfiguration.NM_VMEM_CHECK_ENABLED)} " +
+ log"${MDC(LogKeys.ERROR, diag)} Consider boosting " +
+ log"${MDC(LogKeys.CONFIG, EXECUTOR_MEMORY_OVERHEAD.key)} or boosting " +
+ log"${MDC(LogKeys.CONFIG2, YarnConfiguration.NM_VMEM_PMEM_RATIO)} or disabling " +
+ log"${MDC(LogKeys.CONFIG3, YarnConfiguration.NM_VMEM_CHECK_ENABLED)} " +
log"because of YARN-4714."
(true, message)
case ContainerExitStatus.KILLED_EXCEEDED_PMEM =>
@@ -900,8 +900,8 @@ private[yarn] class YarnAllocator(
val diag = pmemExceededPattern.findFirstIn(completedContainer.getDiagnostics)
.map(_.concat(".")).getOrElse("")
val message = log"Container killed by YARN for exceeding physical memory limits. " +
- log"${MDC(LogKey.ERROR, diag)} Consider boosting " +
- log"${MDC(LogKey.CONFIG, EXECUTOR_MEMORY_OVERHEAD.key)}."
+ log"${MDC(LogKeys.ERROR, diag)} Consider boosting " +
+ log"${MDC(LogKeys.CONFIG, EXECUTOR_MEMORY_OVERHEAD.key)}."
(true, message)
case other_exit_status =>
val exitStatus = completedContainer.getExitStatus
@@ -912,19 +912,19 @@ private[yarn] class YarnAllocator(
// SPARK-26269: follow YARN's behaviour, see details in
// org.apache.hadoop.yarn.util.Apps#shouldCountTowardsNodeBlacklisting
if (NOT_APP_AND_SYSTEM_FAULT_EXIT_STATUS.contains(other_exit_status)) {
- (false, log"Container marked as failed: ${MDC(LogKey.CONTAINER_ID, containerId)}" +
- log"${MDC(LogKey.HOST, onHostStr)}. " +
- log"Exit status: ${MDC(LogKey.EXIT_CODE, exitStatus)}. " +
- log"Possible causes: ${MDC(LogKey.REASON, sparkExitCodeReason)} " +
- log"Diagnostics: ${MDC(LogKey.ERROR, completedContainer.getDiagnostics)}.")
+ (false, log"Container marked as failed: ${MDC(LogKeys.CONTAINER_ID, containerId)}" +
+ log"${MDC(LogKeys.HOST, onHostStr)}. " +
+ log"Exit status: ${MDC(LogKeys.EXIT_CODE, exitStatus)}. " +
+ log"Possible causes: ${MDC(LogKeys.REASON, sparkExitCodeReason)} " +
+ log"Diagnostics: ${MDC(LogKeys.ERROR, completedContainer.getDiagnostics)}.")
} else {
// completed container from a bad node
allocatorNodeHealthTracker.handleResourceAllocationFailure(hostOpt)
- (true, log"Container from a bad node: ${MDC(LogKey.CONTAINER_ID, containerId)}" +
- log"${MDC(LogKey.HOST, onHostStr)}. " +
- log"Exit status: ${MDC(LogKey.EXIT_CODE, exitStatus)}. " +
- log"Possible causes: ${MDC(LogKey.REASON, sparkExitCodeReason)} " +
- log"Diagnostics: ${MDC(LogKey.ERROR, completedContainer.getDiagnostics)}.")
+ (true, log"Container from a bad node: ${MDC(LogKeys.CONTAINER_ID, containerId)}" +
+ log"${MDC(LogKeys.HOST, onHostStr)}. " +
+ log"Exit status: ${MDC(LogKeys.EXIT_CODE, exitStatus)}. " +
+ log"Possible causes: ${MDC(LogKeys.REASON, sparkExitCodeReason)} " +
+ log"Diagnostics: ${MDC(LogKeys.ERROR, completedContainer.getDiagnostics)}.")
}
}
if (exitCausedByApp) {
@@ -995,7 +995,7 @@ private[yarn] class YarnAllocator(
context.reply(releasedExecutorLossReasons.remove(eid).get)
} else {
logWarning(log"Tried to get the loss reason for non-existent executor " +
- log"${MDC(LogKey.EXECUTOR_ID, eid)}")
+ log"${MDC(LogKeys.EXECUTOR_ID, eid)}")
context.sendFailure(
new SparkException(s"Fail to find loss reason for non-existent executor $eid"))
}
diff --git a/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocatorNodeHealthTracker.scala b/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocatorNodeHealthTracker.scala
index 3ba6687b0224..6938c0d7f802 100644
--- a/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocatorNodeHealthTracker.scala
+++ b/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocatorNodeHealthTracker.scala
@@ -26,7 +26,7 @@ import org.apache.spark.SparkConf
import org.apache.spark.deploy.ExecutorFailureTracker
import org.apache.spark.deploy.yarn.config._
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{FAILURES, HOST, NODES}
+import org.apache.spark.internal.LogKeys.{FAILURES, HOST, NODES}
import org.apache.spark.internal.config._
import org.apache.spark.scheduler.HealthTracker
diff --git a/resource-managers/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerBackend.scala b/resource-managers/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerBackend.scala
index c7defac98b90..26be1ff89314 100644
--- a/resource-managers/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerBackend.scala
+++ b/resource-managers/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerBackend.scala
@@ -27,7 +27,7 @@ import org.apache.spark.{SparkContext, SparkException}
import org.apache.spark.deploy.yarn.{Client, ClientArguments, YarnAppReport}
import org.apache.spark.deploy.yarn.config._
import org.apache.spark.internal.{config, Logging, MDC}
-import org.apache.spark.internal.LogKey.{APP_ID, APP_STATE}
+import org.apache.spark.internal.LogKeys.{APP_ID, APP_STATE}
import org.apache.spark.launcher.SparkAppHandle
import org.apache.spark.scheduler.TaskSchedulerImpl
import org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages._
diff --git a/resource-managers/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackend.scala b/resource-managers/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackend.scala
index c6976caf3a77..e21f1c3268ae 100644
--- a/resource-managers/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackend.scala
+++ b/resource-managers/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackend.scala
@@ -30,7 +30,7 @@ import org.apache.hadoop.yarn.api.records.{ApplicationAttemptId, ApplicationId}
import org.apache.spark.SparkContext
import org.apache.spark.deploy.security.HadoopDelegationTokenManager
import org.apache.spark.internal.{config, Logging, MDC}
-import org.apache.spark.internal.LogKey
+import org.apache.spark.internal.LogKeys
import org.apache.spark.internal.config.UI._
import org.apache.spark.resource.ResourceProfile
import org.apache.spark.rpc._
@@ -221,9 +221,9 @@ private[spark] abstract class YarnSchedulerBackend(
if (hasFilter) {
// SPARK-26255: Append user provided filters(spark.ui.filters) with yarn filter.
val allFilters = Seq(filterName) ++ conf.get(UI_FILTERS)
- logInfo(log"Add WebUI Filter. ${MDC(LogKey.UI_FILTER, filterName)}, " +
- log"${MDC(LogKey.UI_FILTER_PARAMS, filterParams)}, " +
- log"${MDC(LogKey.UI_PROXY_BASE, proxyBase)}")
+ logInfo(log"Add WebUI Filter. ${MDC(LogKeys.UI_FILTER, filterName)}, " +
+ log"${MDC(LogKeys.UI_FILTER_PARAMS, filterParams)}, " +
+ log"${MDC(LogKeys.UI_PROXY_BASE, proxyBase)}")
// For already installed handlers, prepend the filter.
scheduler.sc.ui.foreach { ui =>
@@ -308,8 +308,8 @@ private[spark] abstract class YarnSchedulerBackend(
.recover {
case NonFatal(e) =>
logWarning(log"Attempted to get executor loss reason for executor id " +
- log"${MDC(LogKey.EXECUTOR_ID, executorId)} at RPC address " +
- log"${MDC(LogKey.HOST_PORT, executorRpcAddress)}, but got no response. " +
+ log"${MDC(LogKeys.EXECUTOR_ID, executorId)} at RPC address " +
+ log"${MDC(LogKeys.HOST_PORT, executorRpcAddress)}, but got no response. " +
log"Marking as agent lost.", e)
RemoveExecutor(executorId, ExecutorProcessLost())
}(ThreadUtils.sameThread)
@@ -334,7 +334,7 @@ private[spark] abstract class YarnSchedulerBackend(
override def receive: PartialFunction[Any, Unit] = {
case RegisterClusterManager(am) =>
- logInfo(log"ApplicationMaster registered as ${MDC(LogKey.RPC_ENDPOINT_REF, am)}")
+ logInfo(log"ApplicationMaster registered as ${MDC(LogKeys.RPC_ENDPOINT_REF, am)}")
amEndpoint = Option(am)
reset()
@@ -348,7 +348,7 @@ private[spark] abstract class YarnSchedulerBackend(
case r @ RemoveExecutor(executorId, reason) =>
if (!stopped.get) {
logWarning(log"Requesting driver to remove executor " +
- log"${MDC(LogKey.EXECUTOR_ID, executorId)} for reason ${MDC(LogKey.REASON, reason)}")
+ log"${MDC(LogKeys.EXECUTOR_ID, executorId)} for reason ${MDC(LogKeys.REASON, reason)}")
driverEndpoint.send(r)
}
@@ -398,7 +398,7 @@ private[spark] abstract class YarnSchedulerBackend(
override def onDisconnected(remoteAddress: RpcAddress): Unit = {
if (amEndpoint.exists(_.address == remoteAddress)) {
logWarning(log"ApplicationMaster has disassociated: " +
- log"${MDC(LogKey.HOST_PORT, remoteAddress)}")
+ log"${MDC(LogKeys.HOST_PORT, remoteAddress)}")
amEndpoint = None
}
}
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala
index d9e2154487c7..6565591b7952 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala
@@ -25,7 +25,7 @@ import scala.reflect.ClassTag
import org.apache.spark.SparkUnsupportedOperationException
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.FUNCTION_NAME
+import org.apache.spark.internal.LogKeys.FUNCTION_NAME
import org.apache.spark.sql.AnalysisException
import org.apache.spark.sql.catalyst.FunctionIdentifier
import org.apache.spark.sql.catalyst.expressions._
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/HintErrorLogger.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/HintErrorLogger.scala
index 7338ef21a713..68c6ae9c03e3 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/HintErrorLogger.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/HintErrorLogger.scala
@@ -18,7 +18,7 @@
package org.apache.spark.sql.catalyst.analysis
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{QUERY_HINT, RELATION_NAME, UNSUPPORTED_HINT_REASON}
+import org.apache.spark.internal.LogKeys.{QUERY_HINT, RELATION_NAME, UNSUPPORTED_HINT_REASON}
import org.apache.spark.sql.catalyst.plans.logical.{HintErrorHandler, HintInfo}
/**
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/StreamingJoinHelper.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/StreamingJoinHelper.scala
index ee285f7ca860..ef425be42f98 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/StreamingJoinHelper.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/StreamingJoinHelper.scala
@@ -20,7 +20,7 @@ package org.apache.spark.sql.catalyst.analysis
import scala.util.control.NonFatal
import org.apache.spark.internal.Logging
-import org.apache.spark.internal.LogKey._
+import org.apache.spark.internal.LogKeys._
import org.apache.spark.internal.MDC
import org.apache.spark.sql.catalyst.expressions._
import org.apache.spark.sql.catalyst.planning.ExtractEquiJoinKeys
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/UnsupportedOperationChecker.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/UnsupportedOperationChecker.scala
index e39ec267fa61..2366dc4c0eb8 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/UnsupportedOperationChecker.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/UnsupportedOperationChecker.scala
@@ -18,7 +18,7 @@
package org.apache.spark.sql.catalyst.analysis
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{ANALYSIS_ERROR, QUERY_PLAN}
+import org.apache.spark.internal.LogKeys.{ANALYSIS_ERROR, QUERY_PLAN}
import org.apache.spark.sql.AnalysisException
import org.apache.spark.sql.catalyst.ExtendedAnalysisException
import org.apache.spark.sql.catalyst.expressions.{Attribute, AttributeReference, CurrentDate, CurrentTimestampLike, Expression, GroupingSets, LocalTimestamp, MonotonicallyIncreasingID, SessionWindow, WindowExpression}
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala
index 4807d886c9f9..5147680c7524 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala
@@ -33,7 +33,7 @@ import org.json4s.jackson.JsonMethods._
import org.apache.spark.SparkException
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey._
+import org.apache.spark.internal.LogKeys._
import org.apache.spark.sql.AnalysisException
import org.apache.spark.sql.catalyst.{CurrentUserContext, FunctionIdentifier, InternalRow, SQLConfHelper, TableIdentifier}
import org.apache.spark.sql.catalyst.analysis.{MultiInstanceRelation, Resolver, UnresolvedLeafNode}
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/CSVHeaderChecker.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/CSVHeaderChecker.scala
index e69a4552ebff..47e2e288357e 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/CSVHeaderChecker.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/CSVHeaderChecker.scala
@@ -22,7 +22,7 @@ import com.univocity.parsers.csv.{CsvParser, CsvParserSettings}
import org.apache.spark.SparkIllegalArgumentException
import org.apache.spark.internal.{Logging, MDC, MessageWithContext}
-import org.apache.spark.internal.LogKey.{CSV_HEADER_COLUMN_NAME, CSV_HEADER_COLUMN_NAMES, CSV_HEADER_LENGTH, CSV_SCHEMA_FIELD_NAME, CSV_SCHEMA_FIELD_NAMES, CSV_SOURCE, NUM_COLUMNS}
+import org.apache.spark.internal.LogKeys.{CSV_HEADER_COLUMN_NAME, CSV_HEADER_COLUMN_NAMES, CSV_HEADER_LENGTH, CSV_SCHEMA_FIELD_NAME, CSV_SCHEMA_FIELD_NAMES, CSV_SOURCE, NUM_COLUMNS}
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.types.StructType
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/V2ExpressionUtils.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/V2ExpressionUtils.scala
index c6cfccb74c16..ea43bebe8ecb 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/V2ExpressionUtils.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/V2ExpressionUtils.scala
@@ -20,7 +20,7 @@ package org.apache.spark.sql.catalyst.expressions
import java.lang.reflect.{Method, Modifier}
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{FUNCTION_NAME, FUNCTION_PARAMETER}
+import org.apache.spark.internal.LogKeys.{FUNCTION_NAME, FUNCTION_PARAMETER}
import org.apache.spark.sql.AnalysisException
import org.apache.spark.sql.catalyst.{InternalRow, SQLConfHelper}
import org.apache.spark.sql.catalyst.analysis.NoSuchFunctionException
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
index f14e364f1c15..ba3f1df22bc4 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
@@ -33,7 +33,7 @@ import org.codehaus.janino.util.ClassFile
import org.apache.spark.{SparkException, SparkIllegalArgumentException, TaskContext, TaskKilledException}
import org.apache.spark.executor.InputMetrics
import org.apache.spark.internal.Logging
-import org.apache.spark.internal.LogKey._
+import org.apache.spark.internal.LogKeys._
import org.apache.spark.internal.MDC
import org.apache.spark.metrics.source.CodegenMetrics
import org.apache.spark.sql.catalyst.InternalRow
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
index e4548ee69dc4..bdcbe36a3346 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
@@ -20,7 +20,7 @@ package org.apache.spark.sql.catalyst.optimizer
import scala.collection.mutable
import org.apache.spark.SparkException
-import org.apache.spark.internal.LogKey._
+import org.apache.spark.internal.LogKeys._
import org.apache.spark.internal.MDC
import org.apache.spark.sql.catalyst.SQLConfHelper
import org.apache.spark.sql.catalyst.analysis._
@@ -28,7 +28,7 @@ import org.apache.spark.sql.catalyst.catalog.{InMemoryCatalog, SessionCatalog}
import org.apache.spark.sql.catalyst.expressions._
import org.apache.spark.sql.catalyst.expressions.aggregate._
import org.apache.spark.sql.catalyst.plans._
-import org.apache.spark.sql.catalyst.plans.logical.{RepartitionOperation, _}
+import org.apache.spark.sql.catalyst.plans.logical._
import org.apache.spark.sql.catalyst.rules._
import org.apache.spark.sql.catalyst.trees.AlwaysProcess
import org.apache.spark.sql.catalyst.trees.TreePattern._
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/ReplaceNullWithFalseInPredicate.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/ReplaceNullWithFalseInPredicate.scala
index ed3f0cc5a80c..2cda1142299a 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/ReplaceNullWithFalseInPredicate.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/ReplaceNullWithFalseInPredicate.scala
@@ -18,7 +18,7 @@
package org.apache.spark.sql.catalyst.optimizer
import org.apache.spark.SparkIllegalArgumentException
-import org.apache.spark.internal.LogKey.{SQL_TEXT, UNSUPPORTED_EXPR}
+import org.apache.spark.internal.LogKeys.{SQL_TEXT, UNSUPPORTED_EXPR}
import org.apache.spark.internal.MDC
import org.apache.spark.sql.catalyst.expressions.{And, ArrayExists, ArrayFilter, CaseWhen, EqualNullSafe, Expression, If, In, InSet, LambdaFunction, Literal, MapFilter, Not, Or}
import org.apache.spark.sql.catalyst.expressions.Literal.{FalseLiteral, TrueLiteral}
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala
index 655b7c3455b1..2b4ee033b088 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala
@@ -20,7 +20,7 @@ package org.apache.spark.sql.catalyst.optimizer
import scala.annotation.tailrec
import scala.util.control.NonFatal
-import org.apache.spark.internal.LogKey.JOIN_CONDITION
+import org.apache.spark.internal.LogKeys.JOIN_CONDITION
import org.apache.spark.internal.MDC
import org.apache.spark.sql.catalyst.expressions._
import org.apache.spark.sql.catalyst.expressions.aggregate.AggregateExpression
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
index 28327c9dd3be..7d2355b2f08d 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
@@ -32,7 +32,7 @@ import org.apache.commons.codec.binary.Hex
import org.apache.spark.{SparkArithmeticException, SparkException, SparkIllegalArgumentException, SparkThrowable}
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.PARTITION_SPECIFICATION
+import org.apache.spark.internal.LogKeys.PARTITION_SPECIFICATION
import org.apache.spark.sql.catalyst.{FunctionIdentifier, SQLConfHelper, TableIdentifier}
import org.apache.spark.sql.catalyst.analysis._
import org.apache.spark.sql.catalyst.catalog.{BucketSpec, CatalogStorageFormat, ClusterBySpec}
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/rules/RuleExecutor.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/rules/RuleExecutor.scala
index 476ace2662f8..3a5153f46865 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/rules/RuleExecutor.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/rules/RuleExecutor.scala
@@ -19,7 +19,7 @@ package org.apache.spark.sql.catalyst.rules
import org.apache.spark.SparkException
import org.apache.spark.internal.{Logging, MessageWithContext}
-import org.apache.spark.internal.LogKey._
+import org.apache.spark.internal.LogKeys._
import org.apache.spark.internal.MDC
import org.apache.spark.sql.catalyst.QueryPlanningTracker
import org.apache.spark.sql.catalyst.trees.TreeNode
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/CharVarcharUtils.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/CharVarcharUtils.scala
index 06a88b5d7b51..011e385043d3 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/CharVarcharUtils.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/CharVarcharUtils.scala
@@ -20,7 +20,7 @@ package org.apache.spark.sql.catalyst.util
import scala.collection.mutable
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey._
+import org.apache.spark.internal.LogKeys._
import org.apache.spark.sql.catalyst.expressions._
import org.apache.spark.sql.catalyst.expressions.objects.StaticInvoke
import org.apache.spark.sql.catalyst.parser.CatalystSqlParser
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/ParseMode.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/ParseMode.scala
index dd1e466d1b38..cc1a01083af4 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/ParseMode.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/ParseMode.scala
@@ -20,7 +20,7 @@ package org.apache.spark.sql.catalyst.util
import java.util.Locale
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.PARSE_MODE
+import org.apache.spark.internal.LogKeys.PARSE_MODE
sealed trait ParseMode {
/**
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/ResolveDefaultColumnsUtil.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/ResolveDefaultColumnsUtil.scala
index db9adef8ef3b..7b00349a4f27 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/ResolveDefaultColumnsUtil.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/ResolveDefaultColumnsUtil.scala
@@ -21,7 +21,7 @@ import scala.collection.mutable.ArrayBuffer
import org.apache.spark.{SparkThrowable, SparkUnsupportedOperationException}
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey._
+import org.apache.spark.internal.LogKeys._
import org.apache.spark.sql.AnalysisException
import org.apache.spark.sql.catalyst.{InternalRow, SQLConfHelper}
import org.apache.spark.sql.catalyst.analysis._
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/StringUtils.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/StringUtils.scala
index 04df3635d475..2fecd9a23759 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/StringUtils.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/StringUtils.scala
@@ -22,7 +22,7 @@ import java.util.regex.{Pattern, PatternSyntaxException}
import org.apache.commons.text.similarity.LevenshteinDistance
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey._
+import org.apache.spark.internal.LogKeys._
import org.apache.spark.sql.catalyst.analysis.UnresolvedAttribute
import org.apache.spark.sql.errors.QueryCompilationErrors
import org.apache.spark.sql.internal.SQLConf
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/ValidatorUtil.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/ValidatorUtil.scala
index a49de687a27d..8a5291d0bac7 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/ValidatorUtil.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/ValidatorUtil.scala
@@ -27,7 +27,7 @@ import org.apache.hadoop.fs.Path
import org.apache.spark.SparkFiles
import org.apache.spark.deploy.SparkHadoopUtil
import org.apache.spark.internal.Logging
-import org.apache.spark.internal.LogKey._
+import org.apache.spark.internal.LogKeys._
import org.apache.spark.internal.MDC
/**
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/Column.scala b/sql/core/src/main/scala/org/apache/spark/sql/Column.scala
index 1cf315d45f65..2f383f45f1f2 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/Column.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/Column.scala
@@ -21,7 +21,7 @@ import scala.jdk.CollectionConverters._
import org.apache.spark.annotation.Stable
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{LEFT_EXPR, RIGHT_EXPR}
+import org.apache.spark.internal.LogKeys.{LEFT_EXPR, RIGHT_EXPR}
import org.apache.spark.sql.catalyst.analysis._
import org.apache.spark.sql.catalyst.encoders.{encoderFor, ExpressionEncoder}
import org.apache.spark.sql.catalyst.expressions._
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala b/sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala
index b3a5751fce04..5d85f070fbbe 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala
@@ -30,7 +30,7 @@ import org.apache.spark.{SPARK_VERSION, SparkConf, SparkContext, SparkException,
import org.apache.spark.annotation.{DeveloperApi, Experimental, Stable, Unstable}
import org.apache.spark.api.java.JavaRDD
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{CALL_SITE_LONG_FORM, CLASS_NAME}
+import org.apache.spark.internal.LogKeys.{CALL_SITE_LONG_FORM, CLASS_NAME}
import org.apache.spark.internal.config.{ConfigEntry, EXECUTOR_ALLOW_SPARK_CONTEXT}
import org.apache.spark.rdd.RDD
import org.apache.spark.scheduler.{SparkListener, SparkListenerApplicationEnd}
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/api/python/PythonSQLUtils.scala b/sql/core/src/main/scala/org/apache/spark/sql/api/python/PythonSQLUtils.scala
index 0464432d1f7a..1e42e6a5adaa 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/api/python/PythonSQLUtils.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/api/python/PythonSQLUtils.scala
@@ -27,7 +27,7 @@ import net.razorvine.pickle.{Pickler, Unpickler}
import org.apache.spark.SparkException
import org.apache.spark.api.python.DechunkedInputStream
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.CLASS_LOADER
+import org.apache.spark.internal.LogKeys.CLASS_LOADER
import org.apache.spark.security.SocketAuthServer
import org.apache.spark.sql.{Column, DataFrame, Row, SparkSession}
import org.apache.spark.sql.catalyst.{CatalystTypeConverters, InternalRow}
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/api/r/SQLUtils.scala b/sql/core/src/main/scala/org/apache/spark/sql/api/r/SQLUtils.scala
index 655d4193f684..ecbc57f25ad4 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/api/r/SQLUtils.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/api/r/SQLUtils.scala
@@ -28,7 +28,7 @@ import org.apache.spark.api.java.{JavaRDD, JavaSparkContext}
import org.apache.spark.api.r.SerDe
import org.apache.spark.broadcast.Broadcast
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.CONFIG
+import org.apache.spark.internal.LogKeys.CONFIG
import org.apache.spark.rdd.RDD
import org.apache.spark.sql._
import org.apache.spark.sql.catalyst.expressions.{ExprUtils, GenericRowWithSchema, Literal}
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveSessionCatalog.scala b/sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveSessionCatalog.scala
index 1f141766f437..9afbdb011cb8 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveSessionCatalog.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveSessionCatalog.scala
@@ -20,7 +20,7 @@ package org.apache.spark.sql.catalyst.analysis
import org.apache.commons.lang3.StringUtils
import org.apache.spark.SparkException
-import org.apache.spark.internal.LogKey.CONFIG
+import org.apache.spark.internal.LogKeys.CONFIG
import org.apache.spark.internal.MDC
import org.apache.spark.sql.SaveMode
import org.apache.spark.sql.catalyst.{FunctionIdentifier, TableIdentifier}
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/BaseScriptTransformationExec.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/BaseScriptTransformationExec.scala
index 6e54bde46942..af3a8d67e3c2 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/BaseScriptTransformationExec.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/BaseScriptTransformationExec.scala
@@ -28,7 +28,7 @@ import org.apache.hadoop.conf.Configuration
import org.apache.spark.{SparkFiles, TaskContext}
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey._
+import org.apache.spark.internal.LogKeys._
import org.apache.spark.rdd.RDD
import org.apache.spark.sql.catalyst.{CatalystTypeConverters, InternalRow}
import org.apache.spark.sql.catalyst.expressions.{Attribute, AttributeSet, Cast, Expression, GenericInternalRow, JsonToStructs, Literal, StructsToJson, UnsafeProjection}
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/CacheManager.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/CacheManager.scala
index b6471cf4c965..ae99873a9f77 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/CacheManager.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/CacheManager.scala
@@ -20,7 +20,7 @@ package org.apache.spark.sql.execution
import org.apache.hadoop.fs.{FileSystem, Path}
import org.apache.spark.internal.{LogEntry, Logging, MDC}
-import org.apache.spark.internal.LogKey._
+import org.apache.spark.internal.LogKeys._
import org.apache.spark.sql.{Dataset, SparkSession}
import org.apache.spark.sql.catalyst.catalog.HiveTableRelation
import org.apache.spark.sql.catalyst.expressions.{Attribute, SubqueryExpression}
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/ExistingRDD.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/ExistingRDD.scala
index 2c77b7f0a05f..8c7ed7b88d45 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/ExistingRDD.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/ExistingRDD.scala
@@ -18,7 +18,7 @@
package org.apache.spark.sql.execution
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{LOGICAL_PLAN_COLUMNS, OPTIMIZED_PLAN_COLUMNS}
+import org.apache.spark.internal.LogKeys.{LOGICAL_PLAN_COLUMNS, OPTIMIZED_PLAN_COLUMNS}
import org.apache.spark.rdd.RDD
import org.apache.spark.sql.{Dataset, Encoder, SparkSession}
import org.apache.spark.sql.catalyst.InternalRow
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala
index c3775b9d14d5..d04d8dc2cd7f 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala
@@ -27,7 +27,7 @@ import org.apache.hadoop.fs.Path
import org.apache.spark.SparkException
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.EXTENDED_EXPLAIN_GENERATOR
+import org.apache.spark.internal.LogKeys.EXTENDED_EXPLAIN_GENERATOR
import org.apache.spark.rdd.RDD
import org.apache.spark.sql.{AnalysisException, ExtendedExplainGenerator, Row, SparkSession}
import org.apache.spark.sql.catalyst.{InternalRow, QueryPlanningTracker}
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala
index c2ba38ac4a85..41793c2df017 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala
@@ -20,7 +20,7 @@ package org.apache.spark.sql.execution
import java.util.Locale
import org.apache.spark.{SparkException, SparkUnsupportedOperationException}
-import org.apache.spark.internal.LogKey.HASH_JOIN_KEYS
+import org.apache.spark.internal.LogKeys.HASH_JOIN_KEYS
import org.apache.spark.internal.MDC
import org.apache.spark.rdd.RDD
import org.apache.spark.sql.{execution, AnalysisException, Strategy}
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/WholeStageCodegenExec.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/WholeStageCodegenExec.scala
index 32d6555aa9d3..eb83fe16d808 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/WholeStageCodegenExec.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/WholeStageCodegenExec.scala
@@ -24,7 +24,7 @@ import scala.collection.mutable
import scala.util.control.NonFatal
import org.apache.spark.{broadcast, SparkException, SparkUnsupportedOperationException}
-import org.apache.spark.internal.LogKey.{CODEGEN_STAGE_ID, ERROR, TREE_NODE}
+import org.apache.spark.internal.LogKeys.{CODEGEN_STAGE_ID, ERROR, TREE_NODE}
import org.apache.spark.internal.MDC
import org.apache.spark.rdd.RDD
import org.apache.spark.sql.catalyst.InternalRow
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala
index e5bfb01d51f5..f30ffaf51566 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala
@@ -29,7 +29,7 @@ import scala.util.control.NonFatal
import org.apache.spark.SparkException
import org.apache.spark.broadcast
import org.apache.spark.internal.{MDC, MessageWithContext}
-import org.apache.spark.internal.LogKey._
+import org.apache.spark.internal.LogKeys._
import org.apache.spark.rdd.RDD
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.catalyst.InternalRow
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/InsertAdaptiveSparkPlan.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/InsertAdaptiveSparkPlan.scala
index ae18de16c23e..8517911d7026 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/InsertAdaptiveSparkPlan.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/InsertAdaptiveSparkPlan.scala
@@ -19,7 +19,7 @@ package org.apache.spark.sql.execution.adaptive
import scala.collection.mutable
-import org.apache.spark.internal.LogKey.{CONFIG, SUB_QUERY}
+import org.apache.spark.internal.LogKeys.{CONFIG, SUB_QUERY}
import org.apache.spark.internal.MDC
import org.apache.spark.sql.catalyst.expressions
import org.apache.spark.sql.catalyst.expressions.{DynamicPruningSubquery, ListQuery, SubqueryExpression}
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/command/AnalyzeTablesCommand.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/command/AnalyzeTablesCommand.scala
index a41706b1f99e..1650af74bc24 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/command/AnalyzeTablesCommand.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/command/AnalyzeTablesCommand.scala
@@ -19,7 +19,7 @@ package org.apache.spark.sql.execution.command
import scala.util.control.NonFatal
-import org.apache.spark.internal.LogKey.{DATABASE_NAME, ERROR, TABLE_NAME}
+import org.apache.spark.internal.LogKeys.{DATABASE_NAME, ERROR, TABLE_NAME}
import org.apache.spark.internal.MDC
import org.apache.spark.sql.{Row, SparkSession}
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/command/CommandUtils.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/command/CommandUtils.scala
index 463f0ca23afc..2cc09a61e224 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/command/CommandUtils.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/command/CommandUtils.scala
@@ -25,7 +25,7 @@ import scala.util.control.NonFatal
import org.apache.hadoop.fs.{FileStatus, FileSystem, Path, PathFilter}
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{DATABASE_NAME, ERROR, TABLE_NAME}
+import org.apache.spark.internal.LogKeys.{DATABASE_NAME, ERROR, TABLE_NAME}
import org.apache.spark.sql.{Column, SparkSession}
import org.apache.spark.sql.catalyst.{InternalRow, TableIdentifier}
import org.apache.spark.sql.catalyst.analysis.{ResolvedIdentifier, UnresolvedAttribute}
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/command/InsertIntoDataSourceDirCommand.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/command/InsertIntoDataSourceDirCommand.scala
index af894157c1ba..7c690c8ccc08 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/command/InsertIntoDataSourceDirCommand.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/command/InsertIntoDataSourceDirCommand.scala
@@ -17,7 +17,7 @@
package org.apache.spark.sql.execution.command
-import org.apache.spark.internal.LogKey._
+import org.apache.spark.internal.LogKeys._
import org.apache.spark.internal.MDC
import org.apache.spark.sql._
import org.apache.spark.sql.catalyst.catalog._
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/command/SetCommand.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/command/SetCommand.scala
index 6c57d9b2fe26..4e513fc3e8c1 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/command/SetCommand.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/command/SetCommand.scala
@@ -18,7 +18,7 @@
package org.apache.spark.sql.execution.command
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{CONFIG, CONFIG2, KEY, VALUE}
+import org.apache.spark.internal.LogKeys.{CONFIG, CONFIG2, KEY, VALUE}
import org.apache.spark.sql.{AnalysisException, Row, SparkSession}
import org.apache.spark.sql.catalyst.expressions.Attribute
import org.apache.spark.sql.catalyst.parser.ParseException
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/command/createDataSourceTables.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/command/createDataSourceTables.scala
index 1bf5fb26a1f6..9f5c240818fd 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/command/createDataSourceTables.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/command/createDataSourceTables.scala
@@ -19,7 +19,7 @@ package org.apache.spark.sql.execution.command
import java.net.URI
-import org.apache.spark.internal.LogKey._
+import org.apache.spark.internal.LogKeys._
import org.apache.spark.internal.MDC
import org.apache.spark.sql._
import org.apache.spark.sql.catalyst.catalog._
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala
index 3993cebd5eef..6f36802bef59 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala
@@ -29,7 +29,7 @@ import org.apache.hadoop.fs._
import org.apache.hadoop.mapred.{FileInputFormat, JobConf}
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey._
+import org.apache.spark.internal.LogKeys._
import org.apache.spark.internal.config.RDD_PARALLEL_LISTING_THRESHOLD
import org.apache.spark.sql.{Row, SparkSession}
import org.apache.spark.sql.catalyst.TableIdentifier
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/BasicWriteStatsTracker.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/BasicWriteStatsTracker.scala
index a34b235d2d12..2f097f78443b 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/BasicWriteStatsTracker.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/BasicWriteStatsTracker.scala
@@ -27,7 +27,7 @@ import org.apache.hadoop.fs.{FileSystem, Path}
import org.apache.spark.{SparkContext, TaskContext}
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{ACTUAL_NUM_FILES, EXPECTED_NUM_FILES}
+import org.apache.spark.internal.LogKeys.{ACTUAL_NUM_FILES, EXPECTED_NUM_FILES}
import org.apache.spark.sql.catalyst.InternalRow
import org.apache.spark.sql.execution.SQLExecution
import org.apache.spark.sql.execution.datasources.BasicWriteJobStatsTracker._
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala
index 1936167ecba6..4b337798df37 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala
@@ -28,7 +28,7 @@ import org.apache.hadoop.fs.Path
import org.apache.spark.SparkException
import org.apache.spark.deploy.SparkHadoopUtil
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{CLASS_NAME, DATA_SOURCE_PROVIDER, DATA_SOURCES, PATHS}
+import org.apache.spark.internal.LogKeys.{CLASS_NAME, DATA_SOURCE_PROVIDER, DATA_SOURCES, PATHS}
import org.apache.spark.sql._
import org.apache.spark.sql.catalyst.analysis.UnresolvedAttribute
import org.apache.spark.sql.catalyst.catalog.{BucketSpec, CatalogStorageFormat, CatalogTable, CatalogUtils}
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceManager.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceManager.scala
index 2912cc965510..93fc6cf367cf 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceManager.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceManager.scala
@@ -22,7 +22,7 @@ import java.util.concurrent.ConcurrentHashMap
import org.apache.spark.api.python.PythonUtils
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.DATA_SOURCE
+import org.apache.spark.internal.LogKeys.DATA_SOURCE
import org.apache.spark.sql.errors.QueryCompilationErrors
import org.apache.spark.sql.execution.datasources.v2.python.UserDefinedPythonDataSource
import org.apache.spark.util.Utils
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatWriter.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatWriter.scala
index 65a73261e833..d842e23cdf40 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatWriter.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatWriter.scala
@@ -27,7 +27,7 @@ import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
import org.apache.spark._
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey._
+import org.apache.spark.internal.LogKeys._
import org.apache.spark.internal.io.{FileCommitProtocol, SparkHadoopWriterUtils}
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.catalyst.InternalRow
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FilePartition.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FilePartition.scala
index 10c77607c311..10cae0e4966c 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FilePartition.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FilePartition.scala
@@ -22,7 +22,7 @@ import scala.math.BigDecimal.RoundingMode
import org.apache.spark.Partition
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{CONFIG, DESIRED_PARTITIONS_SIZE, MAX_PARTITIONS_SIZE, PARTITIONS_SIZE}
+import org.apache.spark.internal.LogKeys.{CONFIG, DESIRED_PARTITIONS_SIZE, MAX_PARTITIONS_SIZE, PARTITIONS_SIZE}
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.connector.read.InputPartition
import org.apache.spark.sql.execution.ScanFileListing
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileScanRDD.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileScanRDD.scala
index 28700fa4f37f..7c1c4fe2f137 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileScanRDD.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileScanRDD.scala
@@ -24,7 +24,7 @@ import org.apache.hadoop.fs.Path
import org.apache.spark.{Partition => RDDPartition, TaskContext}
import org.apache.spark.deploy.SparkHadoopUtil
-import org.apache.spark.internal.LogKey.PATH
+import org.apache.spark.internal.LogKeys.PATH
import org.apache.spark.internal.MDC
import org.apache.spark.paths.SparkPath
import org.apache.spark.rdd.{InputFileBlockHolder, RDD}
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileStatusCache.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileStatusCache.scala
index 29706f9cd7f4..2bb8476a9f0e 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileStatusCache.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileStatusCache.scala
@@ -26,7 +26,7 @@ import com.google.common.cache._
import org.apache.hadoop.fs.{FileStatus, Path}
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{CACHED_TABLE_PARTITION_METADATA_SIZE, MAX_TABLE_PARTITION_METADATA_SIZE}
+import org.apache.spark.internal.LogKeys.{CACHED_TABLE_PARTITION_METADATA_SIZE, MAX_TABLE_PARTITION_METADATA_SIZE}
import org.apache.spark.sql.SparkSession
import org.apache.spark.util.SizeEstimator
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVDataSource.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVDataSource.scala
index 52b5810362ce..9381d7c82459 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVDataSource.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVDataSource.scala
@@ -29,7 +29,7 @@ import org.apache.hadoop.mapreduce.lib.input.FileInputFormat
import org.apache.spark.TaskContext
import org.apache.spark.input.{PortableDataStream, StreamInputFormat}
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.PATH
+import org.apache.spark.internal.LogKeys.PATH
import org.apache.spark.rdd.{BinaryFileRDD, RDD}
import org.apache.spark.sql.{Dataset, Encoders, SparkSession}
import org.apache.spark.sql.catalyst.InternalRow
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRelation.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRelation.scala
index ca2d59a7672f..599cc369d51c 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRelation.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRelation.scala
@@ -22,7 +22,7 @@ import scala.math.BigDecimal.RoundingMode
import org.apache.spark.Partition
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{LOWER_BOUND, NEW_VALUE, OLD_VALUE, UPPER_BOUND}
+import org.apache.spark.internal.LogKeys.{LOWER_BOUND, NEW_VALUE, OLD_VALUE, UPPER_BOUND}
import org.apache.spark.rdd.RDD
import org.apache.spark.sql.{DataFrame, Row, SaveMode, SparkSession, SQLContext}
import org.apache.spark.sql.catalyst.analysis._
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala
index 7f18ecd8f07e..b4d005e6bb80 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala
@@ -33,7 +33,7 @@ import scala.util.control.NonFatal
import org.apache.spark.{SparkThrowable, SparkUnsupportedOperationException, TaskContext}
import org.apache.spark.executor.InputMetrics
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{DEFAULT_ISOLATION_LEVEL, ISOLATION_LEVEL}
+import org.apache.spark.internal.LogKeys.{DEFAULT_ISOLATION_LEVEL, ISOLATION_LEVEL}
import org.apache.spark.sql.{DataFrame, Row}
import org.apache.spark.sql.catalyst.{InternalRow, SQLConfHelper}
import org.apache.spark.sql.catalyst.analysis.{DecimalPrecision, Resolver}
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonOutputWriter.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonOutputWriter.scala
index 34046b075395..5727c502a709 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonOutputWriter.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonOutputWriter.scala
@@ -22,7 +22,7 @@ import org.apache.hadoop.fs.Path
import org.apache.hadoop.mapreduce.TaskAttemptContext
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{ENCODING, PATH}
+import org.apache.spark.internal.LogKeys.{ENCODING, PATH}
import org.apache.spark.sql.catalyst.InternalRow
import org.apache.spark.sql.catalyst.json.{JacksonGenerator, JSONOptions, JSONOptionsInRead}
import org.apache.spark.sql.execution.datasources.{CodecStreams, OutputWriter}
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcUtils.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcUtils.scala
index 4d1354d15aba..50c28c783b4c 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcUtils.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcUtils.scala
@@ -32,7 +32,7 @@ import org.apache.orc.{BooleanColumnStatistics, ColumnStatistics, DateColumnStat
import org.apache.spark.{SPARK_VERSION_SHORT, SparkException}
import org.apache.spark.deploy.SparkHadoopUtil
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.PATH
+import org.apache.spark.internal.LogKeys.PATH
import org.apache.spark.sql.{SPARK_VERSION_METADATA_KEY, SparkSession}
import org.apache.spark.sql.catalyst.{FileSourceOptions, InternalRow}
import org.apache.spark.sql.catalyst.analysis.caseSensitiveResolution
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala
index 3fdbc38bba7c..e5fbf8be1f0c 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala
@@ -33,7 +33,7 @@ import org.apache.parquet.hadoop._
import org.apache.spark.TaskContext
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{PATH, SCHEMA}
+import org.apache.spark.internal.LogKeys.{PATH, SCHEMA}
import org.apache.spark.sql._
import org.apache.spark.sql.catalyst.InternalRow
import org.apache.spark.sql.catalyst.expressions._
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetUtils.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetUtils.scala
index 1c1c46542b61..0549d99acc68 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetUtils.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetUtils.scala
@@ -34,7 +34,7 @@ import org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName
import org.apache.spark.{SparkException, SparkUnsupportedOperationException}
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{CLASS_NAME, CONFIG}
+import org.apache.spark.internal.LogKeys.{CLASS_NAME, CONFIG}
import org.apache.spark.sql.Row
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.catalyst.InternalRow
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/CacheTableExec.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/CacheTableExec.scala
index 925246320c66..fc8a40f88545 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/CacheTableExec.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/CacheTableExec.scala
@@ -19,7 +19,7 @@ package org.apache.spark.sql.execution.datasources.v2
import java.util.Locale
-import org.apache.spark.internal.LogKey.OPTIONS
+import org.apache.spark.internal.LogKeys.OPTIONS
import org.apache.spark.internal.MDC
import org.apache.spark.sql.{DataFrame, Dataset}
import org.apache.spark.sql.catalyst.{InternalRow, TableIdentifier}
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/CreateIndexExec.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/CreateIndexExec.scala
index df922af9ca3f..60d44101da3b 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/CreateIndexExec.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/CreateIndexExec.scala
@@ -21,7 +21,7 @@ import java.util
import scala.jdk.CollectionConverters._
-import org.apache.spark.internal.LogKey.{INDEX_NAME, TABLE_NAME}
+import org.apache.spark.internal.LogKeys.{INDEX_NAME, TABLE_NAME}
import org.apache.spark.internal.MDC
import org.apache.spark.sql.catalyst.InternalRow
import org.apache.spark.sql.catalyst.analysis.IndexAlreadyExistsException
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/CreateNamespaceExec.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/CreateNamespaceExec.scala
index 65fec4b243d0..76ba53ef99a0 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/CreateNamespaceExec.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/CreateNamespaceExec.scala
@@ -19,7 +19,7 @@ package org.apache.spark.sql.execution.datasources.v2
import scala.jdk.CollectionConverters.MapHasAsJava
-import org.apache.spark.internal.LogKey.NAMESPACE
+import org.apache.spark.internal.LogKeys.NAMESPACE
import org.apache.spark.internal.MDC
import org.apache.spark.sql.catalyst.InternalRow
import org.apache.spark.sql.catalyst.analysis.NamespaceAlreadyExistsException
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/CreateTableExec.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/CreateTableExec.scala
index b981fa3a2a05..f55fbafe11dd 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/CreateTableExec.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/CreateTableExec.scala
@@ -19,7 +19,7 @@ package org.apache.spark.sql.execution.datasources.v2
import scala.jdk.CollectionConverters._
-import org.apache.spark.internal.LogKey.TABLE_NAME
+import org.apache.spark.internal.LogKeys.TABLE_NAME
import org.apache.spark.internal.MDC
import org.apache.spark.sql.catalyst.InternalRow
import org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala
index a8eecb699ce9..0d926dcd99c4 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala
@@ -23,7 +23,7 @@ import org.apache.commons.lang3.StringUtils
import org.apache.spark.SparkException
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.EXPR
+import org.apache.spark.internal.LogKeys.EXPR
import org.apache.spark.sql.{SparkSession, Strategy}
import org.apache.spark.sql.catalyst.analysis.{ResolvedIdentifier, ResolvedNamespace, ResolvedPartitionSpec, ResolvedTable}
import org.apache.spark.sql.catalyst.catalog.CatalogUtils
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DropIndexExec.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DropIndexExec.scala
index a77d39fb696b..4fe6c3cd4a0e 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DropIndexExec.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DropIndexExec.scala
@@ -17,7 +17,7 @@
package org.apache.spark.sql.execution.datasources.v2
-import org.apache.spark.internal.LogKey.INDEX_NAME
+import org.apache.spark.internal.LogKeys.INDEX_NAME
import org.apache.spark.internal.MDC
import org.apache.spark.sql.catalyst.InternalRow
import org.apache.spark.sql.catalyst.analysis.NoSuchIndexException
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/FilePartitionReader.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/FilePartitionReader.scala
index 1863b4b445d4..b71b077f845e 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/FilePartitionReader.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/FilePartitionReader.scala
@@ -19,7 +19,7 @@ package org.apache.spark.sql.execution.datasources.v2
import java.io.{FileNotFoundException, IOException}
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.PARTITIONED_FILE_READER
+import org.apache.spark.internal.LogKeys.PARTITIONED_FILE_READER
import org.apache.spark.rdd.InputFileBlockHolder
import org.apache.spark.sql.catalyst.FileSourceOptions
import org.apache.spark.sql.connector.read.PartitionReader
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/FileScan.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/FileScan.scala
index 4816e3164d4b..d890107277d6 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/FileScan.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/FileScan.scala
@@ -22,7 +22,7 @@ import org.apache.commons.lang3.StringUtils
import org.apache.hadoop.fs.Path
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{PATH, REASON}
+import org.apache.spark.internal.LogKeys.{PATH, REASON}
import org.apache.spark.internal.config.IO_WARNING_LARGEFILETHRESHOLD
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.catalyst.expressions.{AttributeSet, Expression, ExpressionSet}
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/V2ScanPartitioningAndOrdering.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/V2ScanPartitioningAndOrdering.scala
index b6eda80e7ef8..cfab28bbd15e 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/V2ScanPartitioningAndOrdering.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/V2ScanPartitioningAndOrdering.scala
@@ -17,7 +17,7 @@
package org.apache.spark.sql.execution.datasources.v2
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.CLASS_NAME
+import org.apache.spark.internal.LogKeys.CLASS_NAME
import org.apache.spark.sql.catalyst.SQLConfHelper
import org.apache.spark.sql.catalyst.expressions.V2ExpressionUtils
import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/WriteToDataSourceV2Exec.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/WriteToDataSourceV2Exec.scala
index b8aff58c1268..07ea9ce2bcd6 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/WriteToDataSourceV2Exec.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/WriteToDataSourceV2Exec.scala
@@ -21,7 +21,7 @@ import scala.jdk.CollectionConverters._
import org.apache.spark.{SparkEnv, SparkException, TaskContext}
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey._
+import org.apache.spark.internal.LogKeys._
import org.apache.spark.rdd.RDD
import org.apache.spark.sql.catalyst.InternalRow
import org.apache.spark.sql.catalyst.analysis.NoSuchTableException
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/BroadcastExchangeExec.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/BroadcastExchangeExec.scala
index a760632b3fc6..60a98c67163e 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/BroadcastExchangeExec.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/BroadcastExchangeExec.scala
@@ -25,7 +25,7 @@ import scala.concurrent.duration.NANOSECONDS
import scala.util.control.NonFatal
import org.apache.spark.{broadcast, SparkException}
-import org.apache.spark.internal.LogKey._
+import org.apache.spark.internal.LogKeys._
import org.apache.spark.internal.MDC
import org.apache.spark.rdd.RDD
import org.apache.spark.sql.catalyst.InternalRow
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/python/ApplyInPandasWithStatePythonRunner.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/python/ApplyInPandasWithStatePythonRunner.scala
index 5b65c5fe7519..ae982f2f87f2 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/python/ApplyInPandasWithStatePythonRunner.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/python/ApplyInPandasWithStatePythonRunner.scala
@@ -27,7 +27,7 @@ import org.json4s._
import org.json4s.jackson.JsonMethods._
import org.apache.spark.api.python._
-import org.apache.spark.internal.LogKey.CONFIG
+import org.apache.spark.internal.LogKeys.CONFIG
import org.apache.spark.internal.MDC
import org.apache.spark.sql.Row
import org.apache.spark.sql.api.python.PythonSQLUtils
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/python/AttachDistributedSequenceExec.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/python/AttachDistributedSequenceExec.scala
index 62a64bbb3289..a8a6fa97c52a 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/python/AttachDistributedSequenceExec.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/python/AttachDistributedSequenceExec.scala
@@ -17,7 +17,7 @@
package org.apache.spark.sql.execution.python
-import org.apache.spark.internal.LogKey.{RDD_ID, SPARK_PLAN_ID}
+import org.apache.spark.internal.LogKeys.{RDD_ID, SPARK_PLAN_ID}
import org.apache.spark.internal.MDC
import org.apache.spark.rdd.RDD
import org.apache.spark.sql.catalyst.InternalRow
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/AsyncProgressTrackingMicroBatchExecution.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/AsyncProgressTrackingMicroBatchExecution.scala
index 75c7f9dbe861..ce11e9100782 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/AsyncProgressTrackingMicroBatchExecution.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/AsyncProgressTrackingMicroBatchExecution.scala
@@ -20,7 +20,7 @@ package org.apache.spark.sql.execution.streaming
import java.util.concurrent._
import java.util.concurrent.atomic.AtomicLong
-import org.apache.spark.internal.LogKey.PRETTY_ID_STRING
+import org.apache.spark.internal.LogKeys.PRETTY_ID_STRING
import org.apache.spark.internal.MDC
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.catalyst.streaming.WriteToStream
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/AvailableNowDataStreamWrapper.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/AvailableNowDataStreamWrapper.scala
index 37f75742214c..f42250c3c702 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/AvailableNowDataStreamWrapper.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/AvailableNowDataStreamWrapper.scala
@@ -18,7 +18,7 @@
package org.apache.spark.sql.execution.streaming
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{DELEGATE, READ_LIMIT}
+import org.apache.spark.internal.LogKeys.{DELEGATE, READ_LIMIT}
import org.apache.spark.sql.connector.read.streaming.{MicroBatchStream, ReadLimit, SparkDataStream, SupportsAdmissionControl, SupportsTriggerAvailableNow}
import org.apache.spark.sql.connector.read.streaming
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/CheckpointFileManager.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/CheckpointFileManager.scala
index e9307ec971ac..982cc13c4086 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/CheckpointFileManager.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/CheckpointFileManager.scala
@@ -27,7 +27,7 @@ import org.apache.hadoop.fs.local.{LocalFs, RawLocalFs}
import org.apache.hadoop.fs.permission.FsPermission
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey._
+import org.apache.spark.internal.LogKeys.{FINAL_PATH, PATH, TEMP_PATH}
import org.apache.spark.sql.errors.QueryExecutionErrors
import org.apache.spark.sql.execution.streaming.CheckpointFileManager.RenameHelperMethods
import org.apache.spark.sql.internal.SQLConf
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/CompactibleFileStreamLog.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/CompactibleFileStreamLog.scala
index 908a9c662669..d6770452e71f 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/CompactibleFileStreamLog.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/CompactibleFileStreamLog.scala
@@ -27,8 +27,7 @@ import org.apache.hadoop.fs.Path
import org.json4s.{Formats, NoTypeHints}
import org.json4s.jackson.Serialization
-import org.apache.spark.internal.LogKey._
-import org.apache.spark.internal.MDC
+import org.apache.spark.internal.{LogKeys, MDC}
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.errors.QueryExecutionErrors
import org.apache.spark.util.Utils
@@ -105,8 +104,9 @@ abstract class CompactibleFileStreamLog[T <: AnyRef : ClassTag](
defaultCompactInterval, compactibleBatchIds(0).toInt)
}
assert(interval > 0, s"intervalValue = $interval not positive value.")
- logInfo(log"Set the compact interval to ${MDC(COMPACT_INTERVAL, interval)} " +
- log"[defaultCompactInterval: ${MDC(DEFAULT_COMPACT_INTERVAL, defaultCompactInterval)}]")
+ logInfo(log"Set the compact interval to ${MDC(LogKeys.COMPACT_INTERVAL, interval)} " +
+ log"[defaultCompactInterval: " +
+ log"${MDC(LogKeys.DEFAULT_COMPACT_INTERVAL, defaultCompactInterval)}]")
interval
}
@@ -242,8 +242,8 @@ abstract class CompactibleFileStreamLog[T <: AnyRef : ClassTag](
}
if (elapsedMs >= COMPACT_LATENCY_WARN_THRESHOLD_MS) {
- logWarning(log"Compacting took ${MDC(ELAPSED_TIME, elapsedMs)} ms for compact batch " +
- log"${MDC(BATCH_ID, batchId)}")
+ logWarning(log"Compacting took ${MDC(LogKeys.ELAPSED_TIME, elapsedMs)} ms for " +
+ log"compact batch ${MDC(LogKeys.BATCH_ID, batchId)}")
} else {
logDebug(s"Compacting took $elapsedMs ms for compact batch $batchId")
}
@@ -309,9 +309,9 @@ abstract class CompactibleFileStreamLog[T <: AnyRef : ClassTag](
assert(isCompactionBatch(minCompactionBatchId, compactInterval),
s"$minCompactionBatchId is not a compaction batch")
- logInfo(log"Current compact batch id = ${MDC(CURRENT_BATCH_ID, currentBatchId)} " +
+ logInfo(log"Current compact batch id = ${MDC(LogKeys.CURRENT_BATCH_ID, currentBatchId)} " +
log"min compaction batch id to delete = " +
- log"${MDC(MIN_COMPACTION_BATCH_ID, minCompactionBatchId)}")
+ log"${MDC(LogKeys.MIN_COMPACTION_BATCH_ID, minCompactionBatchId)}")
val expiredTime = System.currentTimeMillis() - fileCleanupDelayMs
fileManager.list(metadataPath, (path: Path) => {
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSink.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSink.scala
index 806caebeaf4b..638da08d0fd9 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSink.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSink.scala
@@ -24,7 +24,7 @@ import org.apache.hadoop.fs.{FileSystem, Path}
import org.apache.spark.SparkException
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey._
+import org.apache.spark.internal.LogKeys.{BATCH_ID, ERROR, PATH}
import org.apache.spark.internal.io.FileCommitProtocol
import org.apache.spark.sql.{DataFrame, SparkSession}
import org.apache.spark.sql.catalyst.expressions._
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSinkLog.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSinkLog.scala
index 5059bdecca66..556438811c44 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSinkLog.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSinkLog.scala
@@ -19,7 +19,7 @@ package org.apache.spark.sql.execution.streaming
import org.apache.hadoop.fs.FileStatus
-import org.apache.spark.internal.LogKey._
+import org.apache.spark.internal.LogKeys._
import org.apache.spark.internal.MDC
import org.apache.spark.paths.SparkPath
import org.apache.spark.sql.SparkSession
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSource.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSource.scala
index b8151a07398a..2f5bb2f010e9 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSource.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSource.scala
@@ -27,8 +27,7 @@ import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.fs.{FileStatus, FileSystem, GlobFilter, Path}
import org.apache.spark.deploy.SparkHadoopUtil
-import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey._
+import org.apache.spark.internal.{Logging, LogKeys, MDC}
import org.apache.spark.paths.SparkPath
import org.apache.spark.sql.{DataFrame, Dataset, SparkSession}
import org.apache.spark.sql.catalyst.util.CaseInsensitiveMap
@@ -126,9 +125,9 @@ class FileStreamSource(
}
seenFiles.purge()
- logInfo(log"maxFilesPerBatch = ${MDC(NUM_FILES, maxFilesPerBatch)}, " +
- log"maxBytesPerBatch = ${MDC(NUM_BYTES, maxBytesPerBatch)}, " +
- log"maxFileAgeMs = ${MDC(TIME_UNITS, maxFileAgeMs)}")
+ logInfo(log"maxFilesPerBatch = ${MDC(LogKeys.NUM_FILES, maxFilesPerBatch)}, " +
+ log"maxBytesPerBatch = ${MDC(LogKeys.NUM_BYTES, maxBytesPerBatch)}, " +
+ log"maxFileAgeMs = ${MDC(LogKeys.TIME_UNITS, maxFileAgeMs)}")
private var unreadFiles: Seq[NewFileEntry] = _
@@ -252,8 +251,8 @@ class FileStreamSource(
FileEntry(path = p.urlEncoded, timestamp = timestamp, batchId = metadataLogCurrentOffset)
}.toArray
if (metadataLog.add(metadataLogCurrentOffset, fileEntries)) {
- logInfo(log"Log offset set to ${MDC(LOG_OFFSET, metadataLogCurrentOffset)} " +
- log"with ${MDC(NUM_FILES, batchFiles.size)} new files")
+ logInfo(log"Log offset set to ${MDC(LogKeys.LOG_OFFSET, metadataLogCurrentOffset)} " +
+ log"with ${MDC(LogKeys.NUM_FILES, batchFiles.size)} new files")
} else {
throw new IllegalStateException("Concurrent update to the log. Multiple streaming jobs " +
s"detected for $metadataLogCurrentOffset")
@@ -293,8 +292,9 @@ class FileStreamSource(
assert(startOffset <= endOffset)
val files = metadataLog.get(Some(startOffset + 1), Some(endOffset)).flatMap(_._2)
- logInfo(log"Processing ${MDC(NUM_FILES, files.length)} " +
- log"files from ${MDC(FILE_START_OFFSET, startOffset + 1)}:${MDC(FILE_END_OFFSET, endOffset)}")
+ logInfo(log"Processing ${MDC(LogKeys.NUM_FILES, files.length)} files from " +
+ log"${MDC(LogKeys.FILE_START_OFFSET, startOffset + 1)}:" +
+ log"${MDC(LogKeys.FILE_END_OFFSET, endOffset)}")
logTrace(s"Files are:\n\t" + files.mkString("\n\t"))
val newDataSource =
DataSource(
@@ -384,8 +384,8 @@ class FileStreamSource(
val listingTimeMs = NANOSECONDS.toMillis(endTime - startTime)
if (listingTimeMs > 2000) {
// Output a warning when listing files uses more than 2 seconds.
- logWarning(log"Listed ${MDC(NUM_FILES, files.size)} file(s) in " +
- log"${MDC(ELAPSED_TIME, listingTimeMs)} ms")
+ logWarning(log"Listed ${MDC(LogKeys.NUM_FILES, files.size)} file(s) in " +
+ log"${MDC(LogKeys.ELAPSED_TIME, listingTimeMs)} ms")
} else {
logTrace(s"Listed ${files.size} file(s) in $listingTimeMs ms")
}
@@ -633,13 +633,13 @@ object FileStreamSource {
logDebug(s"Archiving completed file $curPath to $newPath")
if (!fileSystem.rename(curPath, newPath)) {
- logWarning(log"Fail to move ${MDC(CURRENT_PATH, curPath)} to " +
- log"${MDC(NEW_PATH, newPath)} / skip moving file.")
+ logWarning(log"Fail to move ${MDC(LogKeys.CURRENT_PATH, curPath)} to " +
+ log"${MDC(LogKeys.NEW_PATH, newPath)} / skip moving file.")
}
} catch {
case NonFatal(e) =>
- logWarning(log"Fail to move ${MDC(CURRENT_PATH, curPath)} to " +
- log"${MDC(NEW_PATH, newPath)} / skip moving file.", e)
+ logWarning(log"Fail to move ${MDC(LogKeys.CURRENT_PATH, curPath)} to " +
+ log"${MDC(LogKeys.NEW_PATH, newPath)} / skip moving file.", e)
}
}
}
@@ -653,12 +653,14 @@ object FileStreamSource {
logDebug(s"Removing completed file $curPath")
if (!fileSystem.delete(curPath, false)) {
- logWarning(log"Failed to remove ${MDC(CURRENT_PATH, curPath)} / skip removing file.")
+ logWarning(
+ log"Failed to remove ${MDC(LogKeys.CURRENT_PATH, curPath)} / skip removing file.")
}
} catch {
case NonFatal(e) =>
// Log to error but swallow exception to avoid process being stopped
- logWarning(log"Fail to remove ${MDC(CURRENT_PATH, curPath)} / skip removing file.", e)
+ logWarning(
+ log"Fail to remove ${MDC(LogKeys.CURRENT_PATH, curPath)} / skip removing file.", e)
}
}
}
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/HDFSMetadataLog.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/HDFSMetadataLog.scala
index febe4dc8efc0..251cc16acdf4 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/HDFSMetadataLog.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/HDFSMetadataLog.scala
@@ -30,7 +30,7 @@ import org.json4s.{Formats, NoTypeHints}
import org.json4s.jackson.Serialization
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey._
+import org.apache.spark.internal.LogKeys._
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.errors.QueryExecutionErrors
import org.apache.spark.sql.internal.SQLConf
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/IncrementalExecution.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/IncrementalExecution.scala
index c3cb7949bbd6..6b2e899eaf0c 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/IncrementalExecution.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/IncrementalExecution.scala
@@ -23,7 +23,7 @@ import java.util.concurrent.atomic.AtomicInteger
import org.apache.hadoop.fs.Path
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey._
+import org.apache.spark.internal.LogKeys.{BATCH_TIMESTAMP, ERROR}
import org.apache.spark.sql.{SparkSession, Strategy}
import org.apache.spark.sql.catalyst.QueryPlanningTracker
import org.apache.spark.sql.catalyst.expressions.{CurrentBatchTimestamp, ExpressionWithRandomSeed}
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/ManifestFileCommitProtocol.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/ManifestFileCommitProtocol.scala
index 230b8f5dc4ef..b382642eb6bf 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/ManifestFileCommitProtocol.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/ManifestFileCommitProtocol.scala
@@ -26,7 +26,7 @@ import org.apache.hadoop.fs.Path
import org.apache.hadoop.mapreduce.{JobContext, TaskAttemptContext}
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey._
+import org.apache.spark.internal.LogKeys.{BATCH_ID, PATH}
import org.apache.spark.internal.io.FileCommitProtocol
import org.apache.spark.internal.io.FileCommitProtocol.TaskCommitMessage
import org.apache.spark.sql.errors.QueryExecutionErrors
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MetadataLogFileIndex.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MetadataLogFileIndex.scala
index bd1aa91e43ff..45bb69a9c056 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MetadataLogFileIndex.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MetadataLogFileIndex.scala
@@ -21,7 +21,7 @@ import scala.collection.mutable
import org.apache.hadoop.fs.{FileStatus, Path}
-import org.apache.spark.internal.LogKey._
+import org.apache.spark.internal.LogKeys._
import org.apache.spark.internal.MDC
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.execution.datasources._
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MicroBatchExecution.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MicroBatchExecution.scala
index ec7a259609d3..ef49de5e0857 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MicroBatchExecution.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MicroBatchExecution.scala
@@ -20,8 +20,7 @@ package org.apache.spark.sql.execution.streaming
import scala.collection.mutable.{Map => MutableMap}
import scala.collection.mutable
-import org.apache.spark.internal.LogKey._
-import org.apache.spark.internal.MDC
+import org.apache.spark.internal.{LogKeys, MDC}
import org.apache.spark.sql.{Dataset, SparkSession}
import org.apache.spark.sql.catalyst.encoders.ExpressionEncoder
import org.apache.spark.sql.catalyst.expressions.{Alias, Attribute, CurrentBatchTimestamp, CurrentDate, CurrentTimestamp, FileSourceMetadataAttribute, LocalTimestamp}
@@ -104,13 +103,13 @@ class MicroBatchExecution(
if (sparkSession.sessionState.conf.getConf(
SQLConf.STREAMING_TRIGGER_AVAILABLE_NOW_WRAPPER_ENABLED)) {
logInfo(log"Configured to use the wrapper of Trigger.AvailableNow for query " +
- log"${MDC(PRETTY_ID_STRING, prettyIdString)}.")
+ log"${MDC(LogKeys.PRETTY_ID_STRING, prettyIdString)}.")
MultiBatchExecutor()
} else {
val supportsTriggerAvailableNow = sources.distinct.forall { src =>
val supports = src.isInstanceOf[SupportsTriggerAvailableNow]
if (!supports) {
- logWarning(log"source [${MDC(SPARK_DATA_STREAM, src)}] does not support " +
+ logWarning(log"source [${MDC(LogKeys.SPARK_DATA_STREAM, src)}] does not support " +
log"Trigger.AvailableNow. Falling back to single batch execution. Note that this " +
log"may not guarantee processing new data if there is an uncommitted batch. " +
log"Please consult with data source developer to support Trigger.AvailableNow.")
@@ -158,9 +157,9 @@ class MicroBatchExecution(
val metadataPath = s"$resolvedCheckpointRoot/sources/$nextSourceId"
val source = dataSourceV1.createSource(metadataPath)
nextSourceId += 1
- logInfo(log"Using Source [${MDC(STREAMING_SOURCE, source)}] " +
- log"from DataSourceV1 named '${MDC(STREAMING_DATA_SOURCE_NAME, sourceName)}' " +
- log"[${MDC(STREAMING_DATA_SOURCE_DESCRIPTION, dataSourceV1)}]")
+ logInfo(log"Using Source [${MDC(LogKeys.STREAMING_SOURCE, source)}] " +
+ log"from DataSourceV1 named '${MDC(LogKeys.STREAMING_DATA_SOURCE_NAME, sourceName)}' " +
+ log"[${MDC(LogKeys.STREAMING_DATA_SOURCE_DESCRIPTION, dataSourceV1)}]")
StreamingExecutionRelation(source, output, dataSourceV1.catalogTable)(sparkSession)
})
@@ -173,9 +172,9 @@ class MicroBatchExecution(
// Materialize source to avoid creating it in every batch
val metadataPath = s"$resolvedCheckpointRoot/sources/$nextSourceId"
nextSourceId += 1
- logInfo(log"Reading table [${MDC(STREAMING_TABLE, table)}] " +
- log"from DataSourceV2 named '${MDC(STREAMING_DATA_SOURCE_NAME, srcName)}' " +
- log"${MDC(STREAMING_DATA_SOURCE_DESCRIPTION, dsStr)}")
+ logInfo(log"Reading table [${MDC(LogKeys.STREAMING_TABLE, table)}] " +
+ log"from DataSourceV2 named '${MDC(LogKeys.STREAMING_DATA_SOURCE_NAME, srcName)}' " +
+ log"${MDC(LogKeys.STREAMING_DATA_SOURCE_DESCRIPTION, dsStr)}")
// TODO: operator pushdown.
val scan = table.newScanBuilder(options).build()
val stream = scan.toMicroBatchStream(metadataPath)
@@ -193,9 +192,9 @@ class MicroBatchExecution(
val source =
v1.get.asInstanceOf[StreamingRelation].dataSource.createSource(metadataPath)
nextSourceId += 1
- logInfo(log"Using Source [${MDC(STREAMING_SOURCE, source)}] from DataSourceV2 " +
- log"named '${MDC(STREAMING_DATA_SOURCE_NAME, srcName)}' " +
- log"${MDC(STREAMING_DATA_SOURCE_DESCRIPTION, dsStr)}")
+ logInfo(log"Using Source [${MDC(LogKeys.STREAMING_SOURCE, source)}] from " +
+ log"DataSourceV2 named '${MDC(LogKeys.STREAMING_DATA_SOURCE_NAME, srcName)}' " +
+ log"${MDC(LogKeys.STREAMING_DATA_SOURCE_DESCRIPTION, dsStr)}")
// We don't have a catalog table but may have a table identifier. Given this is about
// v1 fallback path, we just give up and set the catalog table as None.
StreamingExecutionRelation(source, output, None)(sparkSession)
@@ -219,8 +218,8 @@ class MicroBatchExecution(
case s: SupportsAdmissionControl =>
val limit = s.getDefaultReadLimit
if (limit != ReadLimit.allAvailable()) {
- logWarning(log"The read limit ${MDC(READ_LIMIT, limit)} for " +
- log"${MDC(SPARK_DATA_STREAM, s)} is ignored when Trigger.Once is used.")
+ logWarning(log"The read limit ${MDC(LogKeys.READ_LIMIT, limit)} for " +
+ log"${MDC(LogKeys.SPARK_DATA_STREAM, s)} is ignored when Trigger.Once is used.")
}
s -> ReadLimit.allAvailable()
case s =>
@@ -286,7 +285,7 @@ class MicroBatchExecution(
// microBatchThread may spawn new jobs, so we need to cancel again to prevent a leak
sparkSession.sparkContext.cancelJobGroup(runId.toString)
}
- logInfo(log"Query ${MDC(PRETTY_ID_STRING, prettyIdString)} was stopped")
+ logInfo(log"Query ${MDC(LogKeys.PRETTY_ID_STRING, prettyIdString)} was stopped")
}
private val watermarkPropagator = WatermarkPropagator(sparkSession.sessionState.conf)
@@ -297,7 +296,7 @@ class MicroBatchExecution(
// shutdown and cleanup required for async log purge mechanism
asyncLogPurgeShutdown()
logInfo(log"Async log purge executor pool for query " +
- log"${MDC(PRETTY_ID_STRING, prettyIdString)} has been shutdown")
+ log"${MDC(LogKeys.PRETTY_ID_STRING, prettyIdString)} has been shutdown")
}
private def initializeExecution(
@@ -313,7 +312,7 @@ class MicroBatchExecution(
setLatestExecutionContext(execCtx)
populateStartOffsets(execCtx, sparkSessionForStream)
- logInfo(log"Stream started from ${MDC(STREAMING_OFFSETS_START, execCtx.startOffsets)}")
+ logInfo(log"Stream started from ${MDC(LogKeys.STREAMING_OFFSETS_START, execCtx.startOffsets)}")
execCtx
}
/**
@@ -520,16 +519,16 @@ class MicroBatchExecution(
}
} else if (latestCommittedBatchId < latestBatchId - 1) {
logWarning(log"Batch completion log latest batch id is " +
- log"${MDC(LATEST_COMMITTED_BATCH_ID, latestCommittedBatchId)}, which is not " +
- log"trailing batchid ${MDC(LATEST_BATCH_ID, latestBatchId)} by one")
+ log"${MDC(LogKeys.LATEST_COMMITTED_BATCH_ID, latestCommittedBatchId)}, which is " +
+ log"not trailing batchid ${MDC(LogKeys.LATEST_BATCH_ID, latestBatchId)} by one")
}
case None => logInfo("no commit log present")
}
// initialize committed offsets to start offsets of the most recent committed batch
committedOffsets = execCtx.startOffsets
- logInfo(log"Resuming at batch ${MDC(BATCH_ID, execCtx.batchId)} with committed offsets " +
- log"${MDC(STREAMING_OFFSETS_START, execCtx.startOffsets)} and available offsets " +
- log"${MDC(STREAMING_OFFSETS_END, execCtx.endOffsets)}")
+ logInfo(log"Resuming at batch ${MDC(LogKeys.BATCH_ID, execCtx.batchId)} with committed " +
+ log"offsets ${MDC(LogKeys.STREAMING_OFFSETS_START, execCtx.startOffsets)} and " +
+ log"available offsets ${MDC(LogKeys.STREAMING_OFFSETS_END, execCtx.endOffsets)}")
case None => // We are starting this stream for the first time.
logInfo(s"Starting new streaming query.")
execCtx.batchId = 0
@@ -758,8 +757,8 @@ class MicroBatchExecution(
}
} else if (catalogTable.exists(_ ne newRelation.catalogTable.get)) {
// Output a warning if `catalogTable` is provided by the source rather than engine
- logWarning(log"Source ${MDC(SPARK_DATA_STREAM, source)} should not produce the " +
- log"information of catalog table by its own.")
+ logWarning(log"Source ${MDC(LogKeys.SPARK_DATA_STREAM, source)} should not " +
+ log"produce the information of catalog table by its own.")
}
newRelation
}
@@ -883,8 +882,8 @@ class MicroBatchExecution(
throw QueryExecutionErrors.concurrentStreamLogUpdate(execCtx.batchId)
}
- logInfo(log"Committed offsets for batch ${MDC(BATCH_ID, execCtx.batchId)}. " +
- log"Metadata ${MDC(OFFSET_SEQUENCE_METADATA, execCtx.offsetSeqMetadata.toString)}")
+ logInfo(log"Committed offsets for batch ${MDC(LogKeys.BATCH_ID, execCtx.batchId)}. " +
+ log"Metadata ${MDC(LogKeys.OFFSET_SEQUENCE_METADATA, execCtx.offsetSeqMetadata.toString)}")
}
/**
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/OffsetSeq.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/OffsetSeq.scala
index 9193b7bd57f5..f0be33ad9a9d 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/OffsetSeq.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/OffsetSeq.scala
@@ -21,12 +21,12 @@ import org.json4s.{Formats, NoTypeHints}
import org.json4s.jackson.Serialization
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{CONFIG, DEFAULT_VALUE, NEW_VALUE, OLD_VALUE, TIP}
+import org.apache.spark.internal.LogKeys.{CONFIG, DEFAULT_VALUE, NEW_VALUE, OLD_VALUE, TIP}
import org.apache.spark.io.CompressionCodec
import org.apache.spark.sql.RuntimeConfig
import org.apache.spark.sql.connector.read.streaming.{Offset => OffsetV2, SparkDataStream}
import org.apache.spark.sql.execution.streaming.state.{FlatMapGroupsWithStateExecHelper, StreamingAggregationStateManager, SymmetricHashJoinStateManager}
-import org.apache.spark.sql.internal.SQLConf.{FLATMAPGROUPSWITHSTATE_STATE_FORMAT_VERSION, _}
+import org.apache.spark.sql.internal.SQLConf._
/**
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/ProgressReporter.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/ProgressReporter.scala
index 2e633009dd67..6ef6f0eb7118 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/ProgressReporter.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/ProgressReporter.scala
@@ -25,8 +25,7 @@ import java.util.{Optional, UUID}
import scala.collection.mutable
import scala.jdk.CollectionConverters._
-import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey._
+import org.apache.spark.internal.{Logging, LogKeys, MDC}
import org.apache.spark.sql.{Row, SparkSession}
import org.apache.spark.sql.catalyst.optimizer.InlineCTE
import org.apache.spark.sql.catalyst.plans.logical.{EventTimeWatermark, LogicalPlan, WithCTE}
@@ -82,7 +81,8 @@ class ProgressReporter(
addNewProgress(newProgress)
postEvent(new QueryProgressEvent(newProgress))
- logInfo(log"Streaming query made progress: ${MDC(STREAMING_QUERY_PROGRESS, newProgress)}")
+ logInfo(
+ log"Streaming query made progress: ${MDC(LogKeys.STREAMING_QUERY_PROGRESS, newProgress)}")
}
private def addNewProgress(newProgress: StreamingQueryProgress): Unit = {
@@ -105,7 +105,7 @@ class ProgressReporter(
if (lastNoExecutionProgressEventTime > Long.MinValue) {
postEvent(new QueryIdleEvent(id, runId, formatTimestamp(currentTriggerStartTimestamp)))
logInfo(log"Streaming query has been idle and waiting for new data more than " +
- log"${MDC(TIME_UNITS, noDataProgressEventInterval)} ms.")
+ log"${MDC(LogKeys.TIME_UNITS, noDataProgressEventInterval)} ms.")
}
lastNoExecutionProgressEventTime = now
@@ -384,9 +384,9 @@ abstract class ProgressContext(
val thresholdForLoggingMillis = 60 * 1000
if (finishTriggerDurationMillis > math.max(thresholdForLoggingMillis, processingTimeMills)) {
logWarning(log"Query progress update takes longer than batch processing time. Progress " +
- log"update takes ${MDC(FINISH_TRIGGER_DURATION, finishTriggerDurationMillis)} " +
- log"milliseconds. Batch processing takes ${MDC(PROCESSING_TIME, processingTimeMills)} " +
- log"milliseconds")
+ log"update takes ${MDC(LogKeys.FINISH_TRIGGER_DURATION, finishTriggerDurationMillis)} " +
+ log"milliseconds. Batch processing takes " +
+ log"${MDC(LogKeys.PROCESSING_TIME, processingTimeMills)} milliseconds")
}
}
@@ -489,8 +489,8 @@ abstract class ProgressContext(
logWarning(log"Could not report metrics as number leaves in trigger logical plan did " +
log"not match that of the execution plan:\nlogical plan leaves: " +
- log"${MDC(LOGICAL_PLAN_LEAVES, toString(allLogicalPlanLeaves))}\nexecution plan " +
- log"leaves: ${MDC(EXECUTION_PLAN_LEAVES, toString(allExecPlanLeaves))}\n")
+ log"${MDC(LogKeys.LOGICAL_PLAN_LEAVES, toString(allLogicalPlanLeaves))}\nexecution " +
+ log"plan leaves: ${MDC(LogKeys.EXECUTION_PLAN_LEAVES, toString(allExecPlanLeaves))}\n")
metricWarningLogged = true
}
Map.empty
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/ResolveWriteToStream.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/ResolveWriteToStream.scala
index 60052687b2d7..0d4ab9d147b8 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/ResolveWriteToStream.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/ResolveWriteToStream.scala
@@ -23,7 +23,7 @@ import scala.util.control.NonFatal
import org.apache.hadoop.fs.Path
-import org.apache.spark.internal.LogKey._
+import org.apache.spark.internal.LogKeys.{CHECKPOINT_LOCATION, CHECKPOINT_ROOT, CONFIG, PATH}
import org.apache.spark.internal.MDC
import org.apache.spark.sql.catalyst.SQLConfHelper
import org.apache.spark.sql.catalyst.analysis.UnsupportedOperationChecker
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala
index b4316bad9a80..9c2391f1b0ee 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala
@@ -33,7 +33,7 @@ import org.apache.hadoop.fs.Path
import org.apache.spark.{JobArtifactSet, SparkContext, SparkException, SparkThrowable}
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey._
+import org.apache.spark.internal.LogKeys.{CHECKPOINT_PATH, CHECKPOINT_ROOT, PATH, PRETTY_ID_STRING, SPARK_DATA_STREAM}
import org.apache.spark.sql._
import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
import org.apache.spark.sql.catalyst.streaming.InternalOutputModes._
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/TimerStateImpl.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/TimerStateImpl.scala
index dc7224d37407..a9c5a70e995d 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/TimerStateImpl.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/TimerStateImpl.scala
@@ -17,7 +17,7 @@
package org.apache.spark.sql.execution.streaming
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{EXPIRY_TIMESTAMP, KEY}
+import org.apache.spark.internal.LogKeys.{EXPIRY_TIMESTAMP, KEY}
import org.apache.spark.sql.catalyst.InternalRow
import org.apache.spark.sql.catalyst.encoders.ExpressionEncoder
import org.apache.spark.sql.catalyst.expressions._
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/TriggerExecutor.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/TriggerExecutor.scala
index 27e7fc015385..bfa838e43e28 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/TriggerExecutor.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/TriggerExecutor.scala
@@ -18,7 +18,7 @@
package org.apache.spark.sql.execution.streaming
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{ELAPSED_TIME, TRIGGER_INTERVAL}
+import org.apache.spark.internal.LogKeys.{ELAPSED_TIME, TRIGGER_INTERVAL}
import org.apache.spark.util.{Clock, SystemClock}
trait TriggerExecutor {
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/WatermarkTracker.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/WatermarkTracker.scala
index 83da9af739da..54c47ec4e6ed 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/WatermarkTracker.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/WatermarkTracker.scala
@@ -22,7 +22,7 @@ import java.util.Locale
import scala.collection.mutable
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey._
+import org.apache.spark.internal.LogKeys._
import org.apache.spark.sql.RuntimeConfig
import org.apache.spark.sql.execution.SparkPlan
import org.apache.spark.sql.internal.SQLConf
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousExecution.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousExecution.scala
index 0756449e1e53..fa49da5feeed 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousExecution.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousExecution.scala
@@ -25,7 +25,7 @@ import java.util.function.UnaryOperator
import scala.collection.mutable.{Map => MutableMap}
import org.apache.spark.SparkEnv
-import org.apache.spark.internal.LogKey._
+import org.apache.spark.internal.LogKeys._
import org.apache.spark.internal.MDC
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.catalyst.expressions.{CurrentDate, CurrentTimestampLike, LocalTimestamp}
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousQueuedDataReader.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousQueuedDataReader.scala
index 2a508f27a10e..398df496d15f 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousQueuedDataReader.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousQueuedDataReader.scala
@@ -24,7 +24,7 @@ import scala.util.control.NonFatal
import org.apache.spark.{SparkEnv, TaskContext}
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey._
+import org.apache.spark.internal.LogKeys._
import org.apache.spark.sql.catalyst.InternalRow
import org.apache.spark.sql.catalyst.expressions.UnsafeProjection
import org.apache.spark.sql.connector.read.PartitionReader
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousTextSocketSource.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousTextSocketSource.scala
index 45a2428d2784..420c3e3be16d 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousTextSocketSource.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousTextSocketSource.scala
@@ -30,7 +30,7 @@ import org.json4s.jackson.Serialization
import org.apache.spark.SparkEnv
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{HOST, PORT}
+import org.apache.spark.internal.LogKeys.{HOST, PORT}
import org.apache.spark.rpc.RpcEndpointRef
import org.apache.spark.sql.catalyst.InternalRow
import org.apache.spark.sql.catalyst.encoders.ExpressionEncoder
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousWriteRDD.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousWriteRDD.scala
index 3af7b2a76f93..5d54b5754915 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousWriteRDD.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousWriteRDD.scala
@@ -18,7 +18,7 @@
package org.apache.spark.sql.execution.streaming.continuous
import org.apache.spark.{Partition, SparkEnv, TaskContext}
-import org.apache.spark.internal.LogKey._
+import org.apache.spark.internal.LogKeys._
import org.apache.spark.internal.MDC
import org.apache.spark.rdd.RDD
import org.apache.spark.sql.catalyst.InternalRow
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/WriteToContinuousDataSourceExec.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/WriteToContinuousDataSourceExec.scala
index d2b6ef0eb796..6b9a13ed3eb0 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/WriteToContinuousDataSourceExec.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/WriteToContinuousDataSourceExec.scala
@@ -18,7 +18,7 @@
package org.apache.spark.sql.execution.streaming.continuous
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey._
+import org.apache.spark.internal.LogKeys._
import org.apache.spark.rdd.RDD
import org.apache.spark.sql.catalyst.InternalRow
import org.apache.spark.sql.catalyst.expressions.Attribute
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/sources/RateStreamMicroBatchStream.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/sources/RateStreamMicroBatchStream.scala
index 520e7fc23f4d..6705201c6731 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/sources/RateStreamMicroBatchStream.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/sources/RateStreamMicroBatchStream.scala
@@ -24,7 +24,7 @@ import java.util.concurrent.TimeUnit
import org.apache.commons.io.IOUtils
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey._
+import org.apache.spark.internal.LogKeys._
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.catalyst.InternalRow
import org.apache.spark.sql.catalyst.util.DateTimeUtils
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/sources/TextSocketMicroBatchStream.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/sources/TextSocketMicroBatchStream.scala
index ada39ca4e776..597b981ebe55 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/sources/TextSocketMicroBatchStream.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/sources/TextSocketMicroBatchStream.scala
@@ -26,7 +26,7 @@ import javax.annotation.concurrent.GuardedBy
import scala.collection.mutable.ListBuffer
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{HOST, PORT}
+import org.apache.spark.internal.LogKeys.{HOST, PORT}
import org.apache.spark.sql.catalyst.InternalRow
import org.apache.spark.sql.catalyst.util.DateTimeUtils
import org.apache.spark.sql.connector.read.{InputPartition, PartitionReader, PartitionReaderFactory}
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala
index 96fb2db6a419..853a79d2b6d2 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala
@@ -32,8 +32,7 @@ import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.fs._
import org.apache.spark.{SparkConf, SparkEnv}
-import org.apache.spark.internal.{Logging, MDC, MessageWithContext}
-import org.apache.spark.internal.LogKey._
+import org.apache.spark.internal.{Logging, LogKeys, MDC, MessageWithContext}
import org.apache.spark.io.CompressionCodec
import org.apache.spark.sql.catalyst.expressions.UnsafeRow
import org.apache.spark.sql.errors.QueryExecutionErrors
@@ -170,8 +169,9 @@ private[sql] class HDFSBackedStateStoreProvider extends StateStoreProvider with
verify(state == UPDATING, "Cannot commit after already committed or aborted")
commitUpdates(newVersion, mapToUpdate, compressedStream)
state = COMMITTED
- logInfo(log"Committed version ${MDC(COMMITTED_VERSION, newVersion)} " +
- log"for ${MDC(STATE_STORE_PROVIDER, this)} to file ${MDC(FILE_NAME, finalDeltaFile)}")
+ logInfo(log"Committed version ${MDC(LogKeys.COMMITTED_VERSION, newVersion)} " +
+ log"for ${MDC(LogKeys.STATE_STORE_PROVIDER, this)} to file " +
+ log"${MDC(LogKeys.FILE_NAME, finalDeltaFile)}")
newVersion
} catch {
case e: Throwable =>
@@ -189,8 +189,8 @@ private[sql] class HDFSBackedStateStoreProvider extends StateStoreProvider with
} else {
state = ABORTED
}
- logInfo(log"Aborted version ${MDC(STATE_STORE_VERSION, newVersion)} " +
- log"for ${MDC(STATE_STORE_PROVIDER, this)}")
+ logInfo(log"Aborted version ${MDC(LogKeys.STATE_STORE_VERSION, newVersion)} " +
+ log"for ${MDC(LogKeys.STATE_STORE_PROVIDER, this)}")
}
/**
@@ -256,16 +256,16 @@ private[sql] class HDFSBackedStateStoreProvider extends StateStoreProvider with
/** Get the state store for making updates to create a new `version` of the store. */
override def getStore(version: Long): StateStore = {
val newMap = getLoadedMapForStore(version)
- logInfo(log"Retrieved version ${MDC(STATE_STORE_VERSION, version)} " +
- log"of ${MDC(STATE_STORE_PROVIDER, HDFSBackedStateStoreProvider.this)} for update")
+ logInfo(log"Retrieved version ${MDC(LogKeys.STATE_STORE_VERSION, version)} " +
+ log"of ${MDC(LogKeys.STATE_STORE_PROVIDER, HDFSBackedStateStoreProvider.this)} for update")
new HDFSBackedStateStore(version, newMap)
}
/** Get the state store for reading to specific `version` of the store. */
override def getReadStore(version: Long): ReadStateStore = {
val newMap = getLoadedMapForStore(version)
- logInfo(log"Retrieved version ${MDC(STATE_STORE_VERSION, version)} of " +
- log"${MDC(STATE_STORE_PROVIDER, HDFSBackedStateStoreProvider.this)} for readonly")
+ logInfo(log"Retrieved version ${MDC(LogKeys.STATE_STORE_VERSION, version)} of " +
+ log"${MDC(LogKeys.STATE_STORE_PROVIDER, HDFSBackedStateStoreProvider.this)} for readonly")
new HDFSBackedReadStateStore(version, newMap)
}
@@ -360,8 +360,9 @@ private[sql] class HDFSBackedStateStoreProvider extends StateStoreProvider with
}
private def toMessageWithContext: MessageWithContext = {
- log"HDFSStateStoreProvider[id = (op=${MDC(OP_ID, stateStoreId.operatorId)}," +
- log"part=${MDC(PARTITION_ID, stateStoreId.partitionId)}),dir = ${MDC(PATH, baseDir)}]"
+ log"HDFSStateStoreProvider[id = (op=${MDC(LogKeys.OP_ID, stateStoreId.operatorId)}," +
+ log"part=${MDC(LogKeys.PARTITION_ID, stateStoreId.partitionId)})," +
+ log"dir = ${MDC(LogKeys.PATH, baseDir)}]"
}
override def toString(): String = {
@@ -489,7 +490,7 @@ private[sql] class HDFSBackedStateStoreProvider extends StateStoreProvider with
return loadedCurrentVersionMap.get
}
- logWarning(log"The state for version ${MDC(FILE_VERSION, version)} doesn't exist in " +
+ logWarning(log"The state for version ${MDC(LogKeys.FILE_VERSION, version)} doesn't exist in " +
log"loadedMaps. Reading snapshot file and delta files if needed..." +
log"Note that this is normal for the first batch of starting query.")
@@ -611,8 +612,8 @@ private[sql] class HDFSBackedStateStoreProvider extends StateStoreProvider with
} finally {
if (input != null) input.close()
}
- logInfo(log"Read delta file for version ${MDC(FILE_VERSION, version)} " +
- log"of ${MDC(STATE_STORE_PROVIDER, this)} from ${MDC(FILE_NAME, fileToRead)}")
+ logInfo(log"Read delta file for version ${MDC(LogKeys.FILE_VERSION, version)} " +
+ log"of ${MDC(LogKeys.STATE_STORE_PROVIDER, this)} from ${MDC(LogKeys.FILE_NAME, fileToRead)}")
}
private def writeSnapshotFile(
@@ -642,9 +643,9 @@ private[sql] class HDFSBackedStateStoreProvider extends StateStoreProvider with
cancelDeltaFile(compressedStream = output, rawStream = rawOutput)
throw e
}
- logInfo(log"Written snapshot file for version ${MDC(FILE_VERSION, version)} of " +
- log"${MDC(STATE_STORE_PROVIDER, this)} at ${MDC(FILE_NAME, targetFile)} for " +
- log"${MDC(OP_TYPE, opType)}")
+ logInfo(log"Written snapshot file for version ${MDC(LogKeys.FILE_VERSION, version)} of " +
+ log"${MDC(LogKeys.STATE_STORE_PROVIDER, this)} at ${MDC(LogKeys.FILE_NAME, targetFile)} " +
+ log"for ${MDC(LogKeys.OP_TYPE, opType)}")
}
/**
@@ -670,8 +671,8 @@ private[sql] class HDFSBackedStateStoreProvider extends StateStoreProvider with
// raw stream or close compressed stream.
case NonFatal(ex) =>
logInfo(log"Failed to cancel delta file for " +
- log"provider=${MDC(STATE_STORE_ID, stateStoreId)} " +
- log"with exception=${MDC(ERROR, ex)}")
+ log"provider=${MDC(LogKeys.STATE_STORE_ID, stateStoreId)} " +
+ log"with exception=${MDC(LogKeys.ERROR, ex)}")
}
}
@@ -720,8 +721,8 @@ private[sql] class HDFSBackedStateStoreProvider extends StateStoreProvider with
}
}
}
- logInfo(log"Read snapshot file for version ${MDC(SNAPSHOT_VERSION, version)} " +
- log"of ${MDC(STATE_STORE_PROVIDER, this)} from ${MDC(FILE_NAME, fileToRead)}")
+ logInfo(log"Read snapshot file for version ${MDC(LogKeys.SNAPSHOT_VERSION, version)} of " +
+ log"${MDC(LogKeys.STATE_STORE_PROVIDER, this)} from ${MDC(LogKeys.FILE_NAME, fileToRead)}")
Some(map)
} catch {
case _: FileNotFoundException =>
@@ -780,9 +781,9 @@ private[sql] class HDFSBackedStateStoreProvider extends StateStoreProvider with
}
logDebug(s"deleting files took $e2 ms.")
logInfo(log"Deleted files older than " +
- log"${MDC(FILE_VERSION, earliestFileToRetain.version)} for " +
- log"${MDC(STATE_STORE_PROVIDER, this)}: " +
- log"${MDC(FILE_NAME, filesToDelete.mkString(", "))}")
+ log"${MDC(LogKeys.FILE_VERSION, earliestFileToRetain.version)} for " +
+ log"${MDC(LogKeys.STATE_STORE_PROVIDER, this)}: " +
+ log"${MDC(LogKeys.FILE_NAME, filesToDelete.mkString(", "))}")
}
}
} catch {
@@ -840,8 +841,8 @@ private[sql] class HDFSBackedStateStoreProvider extends StateStoreProvider with
}
case "snapshot" =>
versionToFiles.put(version, StoreFile(version, path, isSnapshot = true))
- case _ =>
- logWarning(log"Could not identify file ${MDC(PATH, path)} for " + toMessageWithContext)
+ case _ => logWarning(
+ log"Could not identify file ${MDC(LogKeys.PATH, path)} for " + toMessageWithContext)
}
}
}
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala
index eb74bd40a285..caecf817c12f 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala
@@ -36,8 +36,7 @@ import org.rocksdb.CompressionType._
import org.rocksdb.TickerType._
import org.apache.spark.TaskContext
-import org.apache.spark.internal.{LogEntry, Logging, MDC}
-import org.apache.spark.internal.LogKey._
+import org.apache.spark.internal.{LogEntry, Logging, LogKeys, MDC}
import org.apache.spark.sql.catalyst.util.CaseInsensitiveMap
import org.apache.spark.sql.errors.QueryExecutionErrors
import org.apache.spark.util.{NextIterator, Utils}
@@ -178,7 +177,7 @@ class RocksDB(
assert(version >= 0)
acquire(LoadStore)
recordedMetrics = None
- logInfo(log"Loading ${MDC(VERSION_NUMBER, version)}")
+ logInfo(log"Loading ${MDC(LogKeys.VERSION_NUMBER, version)}")
try {
if (loadedVersion != version) {
closeDB()
@@ -213,7 +212,7 @@ class RocksDB(
if (conf.resetStatsOnLoad) {
nativeStats.reset
}
- logInfo(log"Loaded ${MDC(VERSION_NUMBER, version)}")
+ logInfo(log"Loaded ${MDC(LogKeys.VERSION_NUMBER, version)}")
} catch {
case t: Throwable =>
loadedVersion = -1 // invalidate loaded data
@@ -232,8 +231,9 @@ class RocksDB(
*/
private def replayChangelog(endVersion: Long): Unit = {
for (v <- loadedVersion + 1 to endVersion) {
- logInfo(log"replaying changelog from version ${MDC(LOADED_VERSION, loadedVersion)} -> " +
- log"${MDC(END_VERSION, endVersion)}")
+ logInfo(log"replaying changelog from version " +
+ log"${MDC(LogKeys.LOADED_VERSION, loadedVersion)} -> " +
+ log"${MDC(LogKeys.END_VERSION, endVersion)}")
var changelogReader: StateStoreChangelogReader = null
try {
changelogReader = fileManager.getChangelogReader(v, useColumnFamilies)
@@ -463,7 +463,7 @@ class RocksDB(
verifyColFamilyOperations("iterator", colFamilyName)
val iter = db.newIterator(colFamilyNameToHandleMap(colFamilyName))
- logInfo(log"Getting iterator from version ${MDC(LOADED_VERSION, loadedVersion)}")
+ logInfo(log"Getting iterator from version ${MDC(LogKeys.LOADED_VERSION, loadedVersion)}")
iter.seekToFirst()
// Attempt to close this iterator if there is a task failure, or a task interruption.
@@ -494,7 +494,7 @@ class RocksDB(
try {
logInfo(log"Counting keys - getting iterator from version " +
- log"${MDC(LOADED_VERSION, loadedVersion)}")
+ log"${MDC(LogKeys.LOADED_VERSION, loadedVersion)}")
iter.seekToFirst()
@@ -548,7 +548,7 @@ class RocksDB(
val newVersion = loadedVersion + 1
try {
- logInfo(log"Flushing updates for ${MDC(VERSION_NUMBER, newVersion)}")
+ logInfo(log"Flushing updates for ${MDC(LogKeys.VERSION_NUMBER, newVersion)}")
var compactTimeMs = 0L
var flushTimeMs = 0L
@@ -556,7 +556,7 @@ class RocksDB(
if (shouldCreateSnapshot()) {
// Need to flush the change to disk before creating a checkpoint
// because rocksdb wal is disabled.
- logInfo(log"Flushing updates for ${MDC(VERSION_NUMBER, newVersion)}")
+ logInfo(log"Flushing updates for ${MDC(LogKeys.VERSION_NUMBER, newVersion)}")
flushTimeMs = timeTakenMs {
// Flush updates to all available column families
assert(!colFamilyNameToHandleMap.isEmpty)
@@ -574,8 +574,8 @@ class RocksDB(
checkpointTimeMs = timeTakenMs {
val checkpointDir = createTempDir("checkpoint")
- logInfo(log"Creating checkpoint for ${MDC(VERSION_NUMBER, newVersion)} " +
- log"in ${MDC(PATH, checkpointDir)}")
+ logInfo(log"Creating checkpoint for ${MDC(LogKeys.VERSION_NUMBER, newVersion)} " +
+ log"in ${MDC(LogKeys.PATH, checkpointDir)}")
// Make sure the directory does not exist. Native RocksDB fails if the directory to
// checkpoint exists.
Utils.deleteRecursively(checkpointDir)
@@ -595,7 +595,7 @@ class RocksDB(
}
}
- logInfo(log"Syncing checkpoint for ${MDC(VERSION_NUMBER, newVersion)} to DFS")
+ logInfo(log"Syncing checkpoint for ${MDC(LogKeys.VERSION_NUMBER, newVersion)} to DFS")
val fileSyncTimeMs = timeTakenMs {
if (enableChangelogCheckpointing) {
try {
@@ -619,8 +619,8 @@ class RocksDB(
"fileSync" -> fileSyncTimeMs
)
recordedMetrics = Some(metrics)
- logInfo(log"Committed ${MDC(VERSION_NUMBER, newVersion)}, " +
- log"stats = ${MDC(METRICS_JSON, recordedMetrics.get.json)}")
+ logInfo(log"Committed ${MDC(LogKeys.VERSION_NUMBER, newVersion)}, " +
+ log"stats = ${MDC(LogKeys.METRICS_JSON, recordedMetrics.get.json)}")
loadedVersion
} catch {
case t: Throwable =>
@@ -655,9 +655,9 @@ class RocksDB(
fileManager.saveCheckpointToDfs(localDir, version, numKeys)
fileManagerMetrics = fileManager.latestSaveCheckpointMetrics
}
- logInfo(log"${MDC(LOG_ID, loggingId)}: Upload snapshot of version " +
- log"${MDC(VERSION_NUMBER, version)}," +
- log" time taken: ${MDC(TIME_UNITS, uploadTime)} ms")
+ logInfo(log"${MDC(LogKeys.LOG_ID, loggingId)}: Upload snapshot of version " +
+ log"${MDC(LogKeys.VERSION_NUMBER, version)}," +
+ log" time taken: ${MDC(LogKeys.TIME_UNITS, uploadTime)} ms")
} finally {
localCheckpoint.foreach(_.close())
}
@@ -676,7 +676,7 @@ class RocksDB(
// Make sure changelogWriter gets recreated next time.
changelogWriter = None
release(RollbackStore)
- logInfo(log"Rolled back to ${MDC(VERSION_NUMBER, loadedVersion)}")
+ logInfo(log"Rolled back to ${MDC(LogKeys.VERSION_NUMBER, loadedVersion)}")
}
def doMaintenance(): Unit = {
@@ -699,7 +699,7 @@ class RocksDB(
val cleanupTime = timeTakenMs {
fileManager.deleteOldVersions(conf.minVersionsToRetain)
}
- logInfo(log"Cleaned old data, time taken: ${MDC(TIME_UNITS, cleanupTime)} ms")
+ logInfo(log"Cleaned old data, time taken: ${MDC(LogKeys.TIME_UNITS, cleanupTime)} ms")
}
/** Release all resources */
@@ -806,7 +806,7 @@ class RocksDB(
rocksDBMetricsOpt = recordedMetrics
} catch {
case ex: Exception =>
- logInfo(log"Failed to acquire metrics with exception=${MDC(ERROR, ex)}")
+ logInfo(log"Failed to acquire metrics with exception=${MDC(LogKeys.ERROR, ex)}")
} finally {
release(ReportStoreMetrics)
}
@@ -844,8 +844,8 @@ class RocksDB(
Option(TaskContext.get()).foreach(_.addTaskCompletionListener[Unit] {
_ => this.release(StoreTaskCompletionListener)
})
- logInfo(log"RocksDB instance was acquired by ${MDC(THREAD, acquiredThreadInfo)} " +
- log"for opType=${MDC(OP_TYPE, opType.toString)}")
+ logInfo(log"RocksDB instance was acquired by ${MDC(LogKeys.THREAD, acquiredThreadInfo)} " +
+ log"for opType=${MDC(LogKeys.OP_TYPE, opType.toString)}")
}
}
@@ -856,8 +856,8 @@ class RocksDB(
* @param opType - operation type releasing the lock
*/
private def release(opType: RocksDBOpType): Unit = acquireLock.synchronized {
- logInfo(log"RocksDB instance was released by ${MDC(THREAD, acquiredThreadInfo)} " +
- log"for opType=${MDC(OP_TYPE, opType.toString)}")
+ logInfo(log"RocksDB instance was released by ${MDC(LogKeys.THREAD, acquiredThreadInfo)} " +
+ log"for opType=${MDC(LogKeys.OP_TYPE, opType.toString)}")
acquiredThreadInfo = null
acquireLock.notifyAll()
}
@@ -895,7 +895,7 @@ class RocksDB(
colFamilyHandles.asScala.toList.foreach { handle =>
colFamilyNameToHandleMap(handle.getName.map(_.toChar).mkString) = handle
}
- logInfo(log"Opened DB with conf ${MDC(CONFIG, conf)}")
+ logInfo(log"Opened DB with conf ${MDC(LogKeys.CONFIG, conf)}")
}
private def closeDB(): Unit = {
@@ -925,8 +925,8 @@ class RocksDB(
case InfoLogLevel.DEBUG_LEVEL => logDebug(_)
case _ => logTrace(_)
}
- loggingFunc(log"[NativeRocksDB-${MDC(ROCKS_DB_LOG_LEVEL, infoLogLevel.getValue)}] " +
- log"${MDC(ROCKS_DB_LOG_MESSAGE, logMsg)}")
+ loggingFunc(log"[NativeRocksDB-${MDC(LogKeys.ROCKS_DB_LOG_LEVEL, infoLogLevel.getValue)}]" +
+ log" ${MDC(LogKeys.ROCKS_DB_LOG_MESSAGE, logMsg)}")
}
}
@@ -939,7 +939,7 @@ class RocksDB(
// customized logger. We still set it as it might show up in RocksDB config file or logging.
dbOptions.setInfoLogLevel(dbLogLevel)
dbOptions.setLogger(dbLogger)
- logInfo(log"Set RocksDB native logging level to ${MDC(ROCKS_DB_LOG_LEVEL, dbLogLevel)}")
+ logInfo(log"Set RocksDB native logging level to ${MDC(LogKeys.ROCKS_DB_LOG_LEVEL, dbLogLevel)}")
dbLogger
}
@@ -954,8 +954,8 @@ class RocksDB(
Utils.deleteRecursively(file)
} catch {
case e: Exception =>
- logWarning(
- log"Error recursively deleting local dir ${MDC(PATH, file)} while ${MDC(ERROR, msg)}", e)
+ logWarning(log"Error recursively deleting local dir ${MDC(LogKeys.PATH, file)} " +
+ log"while ${MDC(LogKeys.ERROR, msg)}", e)
}
}
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBFileManager.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBFileManager.scala
index f1510e126795..d8b099a56312 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBFileManager.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBFileManager.scala
@@ -38,8 +38,7 @@ import org.json4s.{Formats, NoTypeHints}
import org.json4s.jackson.Serialization
import org.apache.spark.{SparkConf, SparkEnv}
-import org.apache.spark.internal.{Logging, MDC, MessageWithContext}
-import org.apache.spark.internal.LogKey._
+import org.apache.spark.internal.{Logging, LogKeys, MDC, MessageWithContext}
import org.apache.spark.io.CompressionCodec
import org.apache.spark.sql.errors.QueryExecutionErrors
import org.apache.spark.sql.execution.streaming.CheckpointFileManager
@@ -207,14 +206,14 @@ class RocksDBFileManager(
/** Save all the files in given local checkpoint directory as a committed version in DFS */
def saveCheckpointToDfs(checkpointDir: File, version: Long, numKeys: Long): Unit = {
logFilesInDir(checkpointDir, log"Saving checkpoint files " +
- log"for version ${MDC(VERSION_NUMBER, version)}")
+ log"for version ${MDC(LogKeys.VERSION_NUMBER, version)}")
val (localImmutableFiles, localOtherFiles) = listRocksDBFiles(checkpointDir)
val rocksDBFiles = saveImmutableFilesToDfs(version, localImmutableFiles)
val metadata = RocksDBCheckpointMetadata(rocksDBFiles, numKeys)
val metadataFile = localMetadataFile(checkpointDir)
metadata.writeToFile(metadataFile)
- logInfo(log"Written metadata for version ${MDC(VERSION_NUMBER, version)}:\n" +
- log"${MDC(METADATA_JSON, metadata.prettyJson)}")
+ logInfo(log"Written metadata for version ${MDC(LogKeys.VERSION_NUMBER, version)}:\n" +
+ log"${MDC(LogKeys.METADATA_JSON, metadata.prettyJson)}")
if (version <= 1 && numKeys <= 0) {
// If we're writing the initial version and there's no data, we have to explicitly initialize
@@ -230,7 +229,7 @@ class RocksDBFileManager(
}
}
zipToDfsFile(localOtherFiles :+ metadataFile, dfsBatchZipFile(version))
- logInfo(log"Saved checkpoint file for version ${MDC(VERSION_NUMBER, version)}")
+ logInfo(log"Saved checkpoint file for version ${MDC(LogKeys.VERSION_NUMBER, version)}")
}
/**
@@ -240,7 +239,7 @@ class RocksDBFileManager(
* local directory.
*/
def loadCheckpointFromDfs(version: Long, localDir: File): RocksDBCheckpointMetadata = {
- logInfo(log"Loading checkpoint files for version ${MDC(VERSION_NUMBER, version)}")
+ logInfo(log"Loading checkpoint files for version ${MDC(LogKeys.VERSION_NUMBER, version)}")
// The unique ids of SST files are checked when opening a rocksdb instance. The SST files
// in larger versions can't be reused even if they have the same size and name because
// they belong to another rocksdb instance.
@@ -258,15 +257,15 @@ class RocksDBFileManager(
// Copy the necessary immutable files
val metadataFile = localMetadataFile(localDir)
val metadata = RocksDBCheckpointMetadata.readFromFile(metadataFile)
- logInfo(log"Read metadata for version ${MDC(VERSION_NUMBER, version)}:\n" +
- log"${MDC(METADATA_JSON, metadata.prettyJson)}")
+ logInfo(log"Read metadata for version ${MDC(LogKeys.VERSION_NUMBER, version)}:\n" +
+ log"${MDC(LogKeys.METADATA_JSON, metadata.prettyJson)}")
loadImmutableFilesFromDfs(metadata.immutableFiles, localDir)
versionToRocksDBFiles.put(version, metadata.immutableFiles)
metadataFile.delete()
metadata
}
logFilesInDir(localDir, log"Loaded checkpoint files " +
- log"for version ${MDC(VERSION_NUMBER, version)}")
+ log"for version ${MDC(LogKeys.VERSION_NUMBER, version)}")
metadata
}
@@ -332,8 +331,8 @@ class RocksDBFileManager(
val orphanFiles = fileModificationTimes
.filter(_._2 < oldestTrackedFileModificationTime).keys.toSeq
if (orphanFiles.nonEmpty) {
- logInfo(log"Found ${MDC(NUM_FILES, orphanFiles.size)} " +
- log"orphan files: ${MDC(FILE_MODIFICATION_TIME, orphanFiles.take(20).mkString(", "))}" +
+ logInfo(log"Found ${MDC(LogKeys.NUM_FILES, orphanFiles.size)} orphan files: " +
+ log"${MDC(LogKeys.FILE_MODIFICATION_TIME, orphanFiles.take(20).mkString(", "))}" +
log"... (display at most 20 filenames) that should be deleted.")
}
orphanFiles
@@ -346,11 +345,11 @@ class RocksDBFileManager(
versionsToDelete.foreach { version =>
try {
fm.delete(dfsChangelogFile(version))
- logInfo(log"Deleted changelog file ${MDC(VERSION_NUMBER, version)}")
+ logInfo(log"Deleted changelog file ${MDC(LogKeys.VERSION_NUMBER, version)}")
} catch {
case e: Exception =>
logWarning(
- log"Error deleting changelog file for version ${MDC(FILE_VERSION, version)}", e)
+ log"Error deleting changelog file for version ${MDC(LogKeys.FILE_VERSION, version)}", e)
}
}
}
@@ -438,8 +437,8 @@ class RocksDBFileManager(
val allLogFiles = if (fm.exists(logDir)) fm.list(logDir).toImmutableArraySeq else Seq.empty
filesToDelete ++= findOrphanFiles(fileToMaxUsedVersion.keys.toSeq, allSstFiles ++ allLogFiles)
.map(_ -> -1L)
- logInfo(log"Deleting ${MDC(NUM_FILES, filesToDelete.size)} " +
- log"files not used in versions >= ${MDC(VERSION_NUMBER, minVersionToRetain)}")
+ logInfo(log"Deleting ${MDC(LogKeys.NUM_FILES, filesToDelete.size)} " +
+ log"files not used in versions >= ${MDC(LogKeys.VERSION_NUMBER, minVersionToRetain)}")
var failedToDelete = 0
filesToDelete.foreach { case (dfsFileName, maxUsedVersion) =>
try {
@@ -454,10 +453,10 @@ class RocksDBFileManager(
case e: Exception =>
failedToDelete += 1
if (maxUsedVersion == -1) {
- logWarning(log"Error deleting orphan file ${MDC(PATH, dfsFileName)}", e)
+ logWarning(log"Error deleting orphan file ${MDC(LogKeys.PATH, dfsFileName)}", e)
} else {
- logWarning(log"Error deleting file ${MDC(PATH, dfsFileName)}, " +
- log"last used in version ${MDC(MAX_FILE_VERSION, maxUsedVersion)}", e)
+ logWarning(log"Error deleting file ${MDC(LogKeys.PATH, dfsFileName)}, " +
+ log"last used in version ${MDC(LogKeys.MAX_FILE_VERSION, maxUsedVersion)}", e)
}
}
}
@@ -471,14 +470,14 @@ class RocksDBFileManager(
logDebug(s"Deleted version $version")
} catch {
case e: Exception =>
- logWarning(log"Error deleting version file ${MDC(PATH, versionFile)} for " +
- log"version ${MDC(FILE_VERSION, version)}", e)
+ logWarning(log"Error deleting version file ${MDC(LogKeys.PATH, versionFile)} for " +
+ log"version ${MDC(LogKeys.FILE_VERSION, version)}", e)
}
}
- logInfo(log"Deleted ${MDC(NUM_FILES, filesToDelete.size - failedToDelete)} files " +
+ logInfo(log"Deleted ${MDC(LogKeys.NUM_FILES, filesToDelete.size - failedToDelete)} files " +
log"(failed to delete" +
- log"${MDC(NUM_FILES_FAILED_TO_DELETE, failedToDelete)} files) " +
- log"not used in versions >= ${MDC(MIN_VERSION_NUMBER, minVersionToRetain)}")
+ log"${MDC(LogKeys.NUM_FILES_FAILED_TO_DELETE, failedToDelete)} files) " +
+ log"not used in versions >= ${MDC(LogKeys.MIN_VERSION_NUMBER, minVersionToRetain)}")
val changelogVersionsToDelete = changelogFiles
.map(_.getName.stripSuffix(".changelog")).map(_.toLong)
.filter(_ < minVersionToRetain)
@@ -491,7 +490,7 @@ class RocksDBFileManager(
localFiles: Seq[File]): Seq[RocksDBImmutableFile] = {
// Get the immutable files used in previous versions, as some of those uploaded files can be
// reused for this version
- logInfo(log"Saving RocksDB files to DFS for ${MDC(VERSION_NUMBER, version)}")
+ logInfo(log"Saving RocksDB files to DFS for ${MDC(LogKeys.VERSION_NUMBER, version)}")
var bytesCopied = 0L
var filesCopied = 0L
@@ -502,7 +501,8 @@ class RocksDBFileManager(
if (existingDfsFile.isDefined && existingDfsFile.get.sizeBytes == localFile.length()) {
val dfsFile = existingDfsFile.get
filesReused += 1
- logInfo(log"reusing file ${MDC(DFS_FILE, dfsFile)} for ${MDC(FILE_NAME, localFile)}")
+ logInfo(log"reusing file ${MDC(LogKeys.DFS_FILE, dfsFile)} for " +
+ log"${MDC(LogKeys.FILE_NAME, localFile)}")
RocksDBImmutableFile(localFile.getName, dfsFile.dfsFileName, dfsFile.sizeBytes)
} else {
val localFileName = localFile.getName
@@ -515,8 +515,8 @@ class RocksDBFileManager(
fs.copyFromLocalFile(
new Path(localFile.getAbsoluteFile.toURI), dfsFile)
val localFileSize = localFile.length()
- logInfo(log"Copied ${MDC(FILE_NAME, localFile)} to ${MDC(DFS_FILE, dfsFile)} - " +
- log"${MDC(NUM_BYTES, localFileSize)} bytes")
+ logInfo(log"Copied ${MDC(LogKeys.FILE_NAME, localFile)} to " +
+ log"${MDC(LogKeys.DFS_FILE, dfsFile)} - ${MDC(LogKeys.NUM_BYTES, localFileSize)} bytes")
filesCopied += 1
bytesCopied += localFileSize
@@ -526,10 +526,10 @@ class RocksDBFileManager(
immutableDfsFile
}
}
- logInfo(log"Copied ${MDC(NUM_FILES_COPIED, filesCopied)} files " +
- log"(${MDC(NUM_BYTES, bytesCopied)} bytes) from local to" +
- log" DFS for version ${MDC(VERSION_NUMBER, version)}. " +
- log"${MDC(NUM_FILES_REUSED, filesReused)} files reused without copying.")
+ logInfo(log"Copied ${MDC(LogKeys.NUM_FILES_COPIED, filesCopied)} files " +
+ log"(${MDC(LogKeys.NUM_BYTES, bytesCopied)} bytes) from local to" +
+ log" DFS for version ${MDC(LogKeys.VERSION_NUMBER, version)}. " +
+ log"${MDC(LogKeys.NUM_FILES_REUSED, filesReused)} files reused without copying.")
versionToRocksDBFiles.put(version, immutableFiles)
// Cleanup locally deleted files from the localFilesToDfsFiles map
@@ -581,13 +581,13 @@ class RocksDBFileManager(
if (!isSameFile) {
existingFile.delete()
localFilesToDfsFiles.remove(existingFile.getName)
- logInfo(log"Deleted local file ${MDC(FILE_NAME, existingFile)} " +
- log"with size ${MDC(NUM_BYTES, existingFileSize)} mapped" +
- log" to previous dfsFile ${MDC(DFS_FILE, prevDfsFile.getOrElse("null"))}")
+ logInfo(log"Deleted local file ${MDC(LogKeys.FILE_NAME, existingFile)} " +
+ log"with size ${MDC(LogKeys.NUM_BYTES, existingFileSize)} mapped" +
+ log" to previous dfsFile ${MDC(LogKeys.DFS_FILE, prevDfsFile.getOrElse("null"))}")
} else {
- logInfo(log"reusing ${MDC(DFS_FILE, prevDfsFile)} present at " +
- log"${MDC(EXISTING_FILE, existingFile)} " +
- log"for ${MDC(FILE_NAME, requiredFile)}")
+ logInfo(log"reusing ${MDC(LogKeys.DFS_FILE, prevDfsFile)} present at " +
+ log"${MDC(LogKeys.EXISTING_FILE, existingFile)} " +
+ log"for ${MDC(LogKeys.FILE_NAME, requiredFile)}")
}
}
@@ -613,15 +613,16 @@ class RocksDBFileManager(
filesCopied += 1
bytesCopied += localFileSize
localFilesToDfsFiles.put(localFileName, file)
- logInfo(log"Copied ${MDC(DFS_FILE, dfsFile)} to ${MDC(FILE_NAME, localFile)} - " +
- log"${MDC(NUM_BYTES, localFileSize)} bytes")
+ logInfo(log"Copied ${MDC(LogKeys.DFS_FILE, dfsFile)} to " +
+ log"${MDC(LogKeys.FILE_NAME, localFile)} - " +
+ log"${MDC(LogKeys.NUM_BYTES, localFileSize)} bytes")
} else {
filesReused += 1
}
}
- logInfo(log"Copied ${MDC(NUM_FILES_COPIED, filesCopied)} files " +
- log"(${MDC(NUM_BYTES, bytesCopied)} bytes) from DFS to local with " +
- log"${MDC(NUM_FILES_REUSED, filesReused)} files reused.")
+ logInfo(log"Copied ${MDC(LogKeys.NUM_FILES_COPIED, filesCopied)} files " +
+ log"(${MDC(LogKeys.NUM_BYTES, bytesCopied)} bytes) from DFS to local with " +
+ log"${MDC(LogKeys.NUM_FILES_REUSED, filesReused)} files reused.")
loadCheckpointMetrics = RocksDBFileManagerMetrics(
bytesCopied = bytesCopied,
@@ -637,7 +638,7 @@ class RocksDBFileManager(
.filterNot(currentLocalFiles.contains)
mappingsToClean.foreach { f =>
- logInfo(log"cleaning ${MDC(FILE_NAME, f)} from the localFilesToDfsFiles map")
+ logInfo(log"cleaning ${MDC(LogKeys.FILE_NAME, f)} from the localFilesToDfsFiles map")
localFilesToDfsFiles.remove(f)
}
}
@@ -672,8 +673,8 @@ class RocksDBFileManager(
totalBytes += bytes
}
zout.close() // so that any error in closing also cancels the output stream
- logInfo(log"Zipped ${MDC(NUM_BYTES, totalBytes)} bytes (before compression) to " +
- log"${MDC(FILE_NAME, filesStr)}")
+ logInfo(log"Zipped ${MDC(LogKeys.NUM_BYTES, totalBytes)} bytes (before compression) to " +
+ log"${MDC(LogKeys.FILE_NAME, filesStr)}")
// The other fields saveCheckpointMetrics should have been filled
saveCheckpointMetrics =
saveCheckpointMetrics.copy(zipFileBytesUncompressed = Some(totalBytes))
@@ -695,8 +696,8 @@ class RocksDBFileManager(
lazy val files = Option(Utils.recursiveList(dir)).getOrElse(Array.empty).map { f =>
s"${f.getAbsolutePath} - ${f.length()} bytes"
}
- logInfo(msg + log" - ${MDC(NUM_FILES, files.length)} files\n\t" +
- log"${MDC(FILE_NAME, files.mkString("\n\t"))}")
+ logInfo(msg + log" - ${MDC(LogKeys.NUM_FILES, files.length)} files\n\t" +
+ log"${MDC(LogKeys.FILE_NAME, files.mkString("\n\t"))}")
}
private def newDFSFileName(localFileName: String): String = {
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBMemoryManager.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBMemoryManager.scala
index 66451234abc4..273cbbc5e87d 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBMemoryManager.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBMemoryManager.scala
@@ -20,7 +20,7 @@ package org.apache.spark.sql.execution.streaming.state
import org.rocksdb._
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey._
+import org.apache.spark.internal.LogKeys._
/**
* Singleton responsible for managing cache and write buffer manager associated with all RocksDB
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateStoreProvider.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateStoreProvider.scala
index 154f4ea5fc3e..9f568d075cd9 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateStoreProvider.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateStoreProvider.scala
@@ -25,7 +25,7 @@ import org.apache.hadoop.conf.Configuration
import org.apache.spark.{SparkConf, SparkEnv}
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey._
+import org.apache.spark.internal.LogKeys._
import org.apache.spark.sql.catalyst.expressions.UnsafeRow
import org.apache.spark.sql.errors.QueryExecutionErrors
import org.apache.spark.sql.types.StructType
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStore.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStore.scala
index 9ada54e20543..2a1a0f7b01d9 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStore.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStore.scala
@@ -30,8 +30,7 @@ import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.fs.Path
import org.apache.spark.{SparkContext, SparkEnv, SparkUnsupportedOperationException}
-import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey._
+import org.apache.spark.internal.{Logging, LogKeys, MDC}
import org.apache.spark.sql.catalyst.expressions.UnsafeRow
import org.apache.spark.sql.catalyst.util.UnsafeRowUtils
import org.apache.spark.sql.errors.QueryExecutionErrors
@@ -703,9 +702,10 @@ object StateStore extends Logging {
}
if (loadTimeMs > 2000L) {
- logWarning(log"Loaded state store provider in loadTimeMs=${MDC(LOAD_TIME, loadTimeMs)} " +
- log"for storeId=${MDC(STORE_ID, storeProviderId.storeId.toString)} and " +
- log"queryRunId=${MDC(QUERY_RUN_ID, storeProviderId.queryRunId)}")
+ logWarning(log"Loaded state store provider in loadTimeMs=" +
+ log"${MDC(LogKeys.LOAD_TIME, loadTimeMs)} " +
+ log"for storeId=${MDC(LogKeys.STORE_ID, storeProviderId.storeId.toString)} and " +
+ log"queryRunId=${MDC(LogKeys.QUERY_RUN_ID, storeProviderId.queryRunId)}")
}
val otherProviderIds = loadedProviders.keys.filter(_ != storeProviderId).toSeq
@@ -821,18 +821,18 @@ object StateStore extends Logging {
provider.doMaintenance()
if (!verifyIfStoreInstanceActive(id)) {
unload(id)
- logInfo(log"Unloaded ${MDC(STATE_STORE_PROVIDER, provider)}")
+ logInfo(log"Unloaded ${MDC(LogKeys.STATE_STORE_PROVIDER, provider)}")
}
} catch {
case NonFatal(e) =>
- logWarning(log"Error managing ${MDC(STATE_STORE_PROVIDER, provider)}, " +
+ logWarning(log"Error managing ${MDC(LogKeys.STATE_STORE_PROVIDER, provider)}, " +
log"stopping management thread", e)
threadPoolException.set(e)
} finally {
val duration = System.currentTimeMillis() - startTime
val logMsg =
- log"Finished maintenance task for provider=${MDC(STATE_STORE_PROVIDER, id)}" +
- log" in elapsed_time=${MDC(TIME_UNITS, duration)}\n"
+ log"Finished maintenance task for provider=${MDC(LogKeys.STATE_STORE_PROVIDER, id)}" +
+ log" in elapsed_time=${MDC(LogKeys.TIME_UNITS, duration)}\n"
if (duration > 5000) {
logInfo(logMsg)
} else {
@@ -844,7 +844,7 @@ object StateStore extends Logging {
}
})
} else {
- logInfo(log"Not processing partition ${MDC(PARTITION_ID, id)} " +
+ logInfo(log"Not processing partition ${MDC(LogKeys.PARTITION_ID, id)} " +
log"for maintenance because it is currently " +
log"being processed")
}
@@ -861,9 +861,9 @@ object StateStore extends Logging {
.map(_.reportActiveInstance(storeProviderId, host, executorId, otherProviderIds))
.getOrElse(Seq.empty[StateStoreProviderId])
logInfo(log"Reported that the loaded instance " +
- log"${MDC(STATE_STORE_PROVIDER, storeProviderId)} is active")
+ log"${MDC(LogKeys.STATE_STORE_PROVIDER, storeProviderId)} is active")
logDebug(log"The loaded instances are going to unload: " +
- log"${MDC(STATE_STORE_PROVIDER, providerIdsToUnload.mkString(", "))}")
+ log"${MDC(LogKeys.STATE_STORE_PROVIDER, providerIdsToUnload.mkString(", "))}")
providerIdsToUnload
} else {
Seq.empty[StateStoreProviderId]
@@ -895,7 +895,7 @@ object StateStore extends Logging {
_coordRef = StateStoreCoordinatorRef.forExecutor(env)
}
logInfo(log"Retrieved reference to StateStoreCoordinator: " +
- log"${MDC(STATE_STORE_PROVIDER, _coordRef)}")
+ log"${MDC(LogKeys.STATE_STORE_PROVIDER, _coordRef)}")
Some(_coordRef)
} else {
_coordRef = null
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStoreChangelog.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStoreChangelog.scala
index a52d2bb2c610..23f867d3e6c0 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStoreChangelog.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStoreChangelog.scala
@@ -26,7 +26,7 @@ import org.apache.commons.io.IOUtils
import org.apache.hadoop.fs.{FSError, Path}
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey._
+import org.apache.spark.internal.LogKeys._
import org.apache.spark.io.CompressionCodec
import org.apache.spark.sql.errors.QueryExecutionErrors
import org.apache.spark.sql.execution.streaming.CheckpointFileManager
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StreamingSessionWindowStateManager.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StreamingSessionWindowStateManager.scala
index cbb7c1e56907..71df9dc65b41 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StreamingSessionWindowStateManager.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StreamingSessionWindowStateManager.scala
@@ -18,7 +18,7 @@
package org.apache.spark.sql.execution.streaming.state
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey._
+import org.apache.spark.internal.LogKeys._
import org.apache.spark.sql.catalyst.InternalRow
import org.apache.spark.sql.catalyst.expressions.{Attribute, Literal, UnsafeProjection, UnsafeRow}
import org.apache.spark.sql.catalyst.expressions.codegen.GenerateUnsafeProjection
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/SymmetricHashJoinStateManager.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/SymmetricHashJoinStateManager.scala
index b541f59a2165..43ffdf9829b4 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/SymmetricHashJoinStateManager.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/SymmetricHashJoinStateManager.scala
@@ -25,7 +25,7 @@ import org.apache.hadoop.conf.Configuration
import org.apache.spark.TaskContext
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey._
+import org.apache.spark.internal.LogKeys.{END_INDEX, START_INDEX, STATE_STORE_ID}
import org.apache.spark.sql.catalyst.InternalRow
import org.apache.spark.sql.catalyst.expressions.{Attribute, AttributeReference, Expression, JoinedRow, Literal, SafeProjection, SpecificInternalRow, UnsafeProjection, UnsafeRow}
import org.apache.spark.sql.catalyst.types.DataTypeUtils.toAttributes
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/ui/SQLAppStatusListener.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/ui/SQLAppStatusListener.scala
index 5e2aae45c680..bf33ba2c96f1 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/ui/SQLAppStatusListener.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/ui/SQLAppStatusListener.scala
@@ -26,7 +26,7 @@ import scala.util.control.NonFatal
import org.apache.spark.{JobExecutionStatus, SparkConf}
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.CLASS_NAME
+import org.apache.spark.internal.LogKeys.CLASS_NAME
import org.apache.spark.internal.config.Status._
import org.apache.spark.scheduler._
import org.apache.spark.sql.connector.metric.CustomMetric
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/internal/SharedState.scala b/sql/core/src/main/scala/org/apache/spark/sql/internal/SharedState.scala
index af1304dccee0..cb93ff25da0a 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/internal/SharedState.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/internal/SharedState.scala
@@ -30,7 +30,7 @@ import org.apache.hadoop.fs.{FsUrlStreamHandlerFactory, Path}
import org.apache.spark.{SparkConf, SparkContext}
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{CONFIG, CONFIG2}
+import org.apache.spark.internal.LogKeys.{CONFIG, CONFIG2}
import org.apache.spark.sql.catalyst.catalog._
import org.apache.spark.sql.errors.QueryExecutionErrors
import org.apache.spark.sql.execution.CacheManager
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/jdbc/PostgresDialect.scala b/sql/core/src/main/scala/org/apache/spark/sql/jdbc/PostgresDialect.scala
index e4cd79e3f53c..f3fb115c7057 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/jdbc/PostgresDialect.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/jdbc/PostgresDialect.scala
@@ -24,7 +24,7 @@ import java.util.Locale
import scala.util.Using
-import org.apache.spark.internal.LogKey.COLUMN_NAME
+import org.apache.spark.internal.LogKeys.COLUMN_NAME
import org.apache.spark.internal.MDC
import org.apache.spark.sql.AnalysisException
import org.apache.spark.sql.catalyst.SQLConfHelper
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/streaming/StreamingQueryManager.scala b/sql/core/src/main/scala/org/apache/spark/sql/streaming/StreamingQueryManager.scala
index 424c4f8a7d26..12397468f8ed 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/streaming/StreamingQueryManager.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/streaming/StreamingQueryManager.scala
@@ -26,7 +26,7 @@ import scala.jdk.CollectionConverters._
import org.apache.spark.annotation.Evolving
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{QUERY_ID, RUN_ID}
+import org.apache.spark.internal.LogKeys.{QUERY_ID, RUN_ID}
import org.apache.spark.sql.{DataFrame, SparkSession}
import org.apache.spark.sql.catalyst.catalog.CatalogTable
import org.apache.spark.sql.catalyst.streaming.{WriteToStream, WriteToStreamStatement}
diff --git a/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkExecuteStatementOperation.scala b/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkExecuteStatementOperation.scala
index c79746889ae1..295ff1e4a14f 100644
--- a/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkExecuteStatementOperation.scala
+++ b/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkExecuteStatementOperation.scala
@@ -31,7 +31,7 @@ import org.apache.hive.service.cli.session.HiveSession
import org.apache.hive.service.rpc.thrift.{TCLIServiceConstants, TColumnDesc, TPrimitiveTypeEntry, TRowSet, TTableSchema, TTypeDesc, TTypeEntry, TTypeId, TTypeQualifiers, TTypeQualifierValue}
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey._
+import org.apache.spark.internal.LogKeys._
import org.apache.spark.sql.{DataFrame, Row, SQLContext}
import org.apache.spark.sql.catalyst.util.CharVarcharUtils
import org.apache.spark.sql.catalyst.util.DateTimeConstants.MILLIS_PER_SECOND
diff --git a/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkGetCatalogsOperation.scala b/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkGetCatalogsOperation.scala
index 103a90ff9e74..fd99a5b246d9 100644
--- a/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkGetCatalogsOperation.scala
+++ b/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkGetCatalogsOperation.scala
@@ -23,7 +23,7 @@ import org.apache.hive.service.cli.operation.GetCatalogsOperation
import org.apache.hive.service.cli.session.HiveSession
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey._
+import org.apache.spark.internal.LogKeys._
import org.apache.spark.sql.SQLContext
/**
diff --git a/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkGetColumnsOperation.scala b/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkGetColumnsOperation.scala
index 96381b194d23..6d250a6b302f 100644
--- a/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkGetColumnsOperation.scala
+++ b/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkGetColumnsOperation.scala
@@ -28,7 +28,7 @@ import org.apache.hive.service.cli.operation.GetColumnsOperation
import org.apache.hive.service.cli.session.HiveSession
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey._
+import org.apache.spark.internal.LogKeys._
import org.apache.spark.sql.SQLContext
import org.apache.spark.sql.catalyst.TableIdentifier
import org.apache.spark.sql.catalyst.catalog.SessionCatalog
diff --git a/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkGetSchemasOperation.scala b/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkGetSchemasOperation.scala
index 3ec7ec3d502f..2381ee78314d 100644
--- a/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkGetSchemasOperation.scala
+++ b/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkGetSchemasOperation.scala
@@ -26,7 +26,7 @@ import org.apache.hive.service.cli.operation.MetadataOperation.DEFAULT_HIVE_CATA
import org.apache.hive.service.cli.session.HiveSession
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey._
+import org.apache.spark.internal.LogKeys._
import org.apache.spark.sql.SQLContext
/**
diff --git a/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkGetTableTypesOperation.scala b/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkGetTableTypesOperation.scala
index 06e16026bceb..a0c6cd1dcd92 100644
--- a/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkGetTableTypesOperation.scala
+++ b/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkGetTableTypesOperation.scala
@@ -25,7 +25,7 @@ import org.apache.hive.service.cli.operation.GetTableTypesOperation
import org.apache.hive.service.cli.session.HiveSession
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey._
+import org.apache.spark.internal.LogKeys._
import org.apache.spark.sql.SQLContext
import org.apache.spark.sql.catalyst.catalog.CatalogTableType
diff --git a/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkGetTablesOperation.scala b/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkGetTablesOperation.scala
index b5467ddfb2b5..ffd519087a9a 100644
--- a/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkGetTablesOperation.scala
+++ b/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkGetTablesOperation.scala
@@ -28,7 +28,7 @@ import org.apache.hive.service.cli.operation.GetTablesOperation
import org.apache.hive.service.cli.session.HiveSession
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey._
+import org.apache.spark.internal.LogKeys._
import org.apache.spark.sql.SQLContext
import org.apache.spark.sql.catalyst.catalog.CatalogTableType._
diff --git a/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkGetTypeInfoOperation.scala b/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkGetTypeInfoOperation.scala
index 7661e2888904..9ae62ed2fed7 100644
--- a/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkGetTypeInfoOperation.scala
+++ b/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkGetTypeInfoOperation.scala
@@ -27,7 +27,7 @@ import org.apache.hive.service.cli.operation.GetTypeInfoOperation
import org.apache.hive.service.cli.session.HiveSession
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey._
+import org.apache.spark.internal.LogKeys._
import org.apache.spark.sql.SQLContext
/**
diff --git a/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkOperation.scala b/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkOperation.scala
index 9b2ede1a4c3d..11e4817fe2a4 100644
--- a/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkOperation.scala
+++ b/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkOperation.scala
@@ -22,7 +22,7 @@ import org.apache.hive.service.cli.operation.Operation
import org.apache.spark.SparkContext
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{HIVE_OPERATION_TYPE, STATEMENT_ID}
+import org.apache.spark.internal.LogKeys.{HIVE_OPERATION_TYPE, STATEMENT_ID}
import org.apache.spark.sql.{SparkSession, SQLContext}
import org.apache.spark.sql.catalyst.CurrentUserContext.CURRENT_USER
import org.apache.spark.sql.catalyst.catalog.CatalogTableType
diff --git a/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkSQLCLIDriver.scala b/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkSQLCLIDriver.scala
index 888c086e9042..e64e1c283e27 100644
--- a/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkSQLCLIDriver.scala
+++ b/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkSQLCLIDriver.scala
@@ -41,7 +41,7 @@ import sun.misc.{Signal, SignalHandler}
import org.apache.spark.{ErrorMessageFormat, SparkConf, SparkThrowable, SparkThrowableHelper}
import org.apache.spark.deploy.SparkHadoopUtil
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey._
+import org.apache.spark.internal.LogKeys._
import org.apache.spark.sql.AnalysisException
import org.apache.spark.sql.catalyst.analysis.FunctionRegistry
import org.apache.spark.sql.catalyst.util.SQLKeywordUtils
diff --git a/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkSQLDriver.scala b/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkSQLDriver.scala
index 29e468aaa9fe..a9c5d3e25079 100644
--- a/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkSQLDriver.scala
+++ b/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkSQLDriver.scala
@@ -28,7 +28,7 @@ import org.apache.hadoop.hive.ql.processors.CommandProcessorResponse
import org.apache.spark.SparkThrowable
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.COMMAND
+import org.apache.spark.internal.LogKeys.COMMAND
import org.apache.spark.sql.SQLContext
import org.apache.spark.sql.catalyst.plans.logical.CommandResult
import org.apache.spark.sql.execution.{QueryExecution, SQLExecution}
diff --git a/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/ui/HiveThriftServer2Listener.scala b/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/ui/HiveThriftServer2Listener.scala
index 8b7e9b00cb52..8d03d5f848b7 100644
--- a/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/ui/HiveThriftServer2Listener.scala
+++ b/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/ui/HiveThriftServer2Listener.scala
@@ -26,7 +26,7 @@ import org.apache.hive.service.server.HiveServer2
import org.apache.spark.{SparkConf, SparkContext}
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey._
+import org.apache.spark.internal.LogKeys._
import org.apache.spark.internal.config.Status.LIVE_ENTITY_UPDATE_PERIOD
import org.apache.spark.scheduler._
import org.apache.spark.sql.hive.thriftserver.HiveThriftServer2.ExecutionState
diff --git a/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala b/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala
index 1808986ff2e6..77ed81482396 100644
--- a/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala
+++ b/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala
@@ -34,7 +34,7 @@ import org.apache.thrift.TException
import org.apache.spark.{SparkConf, SparkException}
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{DATABASE_NAME, INCOMPATIBLE_TYPES, PROVIDER, SCHEMA, SCHEMA2, TABLE_NAME}
+import org.apache.spark.internal.LogKeys.{DATABASE_NAME, INCOMPATIBLE_TYPES, PROVIDER, SCHEMA, SCHEMA2, TABLE_NAME}
import org.apache.spark.sql.AnalysisException
import org.apache.spark.sql.catalyst.TableIdentifier
import org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException
diff --git a/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala b/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala
index df52cf712bdd..60858089875a 100644
--- a/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala
+++ b/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala
@@ -26,7 +26,7 @@ import org.apache.hadoop.fs.Path
import org.apache.spark.SparkException
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey._
+import org.apache.spark.internal.LogKeys._
import org.apache.spark.sql.{AnalysisException, SparkSession}
import org.apache.spark.sql.catalyst.{QualifiedTableName, TableIdentifier}
import org.apache.spark.sql.catalyst.catalog._
diff --git a/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala b/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
index 5f59e9ca95f9..101d31d60985 100644
--- a/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
+++ b/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
@@ -37,7 +37,7 @@ import org.apache.hive.common.util.HiveVersionInfo
import org.apache.spark.SparkConf
import org.apache.spark.deploy.SparkHadoopUtil
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey
+import org.apache.spark.internal.LogKeys
import org.apache.spark.sql._
import org.apache.spark.sql.catalyst.catalog.CatalogTable
import org.apache.spark.sql.execution.command.DDLUtils
@@ -298,7 +298,7 @@ private[spark] object HiveUtils extends Logging {
conf: SparkConf,
hadoopConf: Configuration): HiveClientImpl = {
logInfo(log"Initializing execution hive, version " +
- log"${MDC(LogKey.HIVE_METASTORE_VERSION, builtinHiveVersion)}")
+ log"${MDC(LogKeys.HIVE_METASTORE_VERSION, builtinHiveVersion)}")
val loader = new IsolatedClientLoader(
version = IsolatedClientLoader.hiveVersion(builtinHiveVersion),
sparkConf = conf,
@@ -332,7 +332,7 @@ private[spark] object HiveUtils extends Logging {
if (file.getName == "*") {
val files = file.getParentFile.listFiles()
if (files == null) {
- logWarning(log"Hive jar path '${MDC(LogKey.PATH, file.getPath)}' does not exist.")
+ logWarning(log"Hive jar path '${MDC(LogKeys.PATH, file.getPath)}' does not exist.")
Nil
} else {
files.filter(_.getName.toLowerCase(Locale.ROOT).endsWith(".jar")).map(_.toURI.toURL)
@@ -345,8 +345,8 @@ private[spark] object HiveUtils extends Logging {
def logInitWithPath(jars: Seq[URL]): Unit = {
logInfo(log"Initializing HiveMetastoreConnection version " +
- log"${MDC(LogKey.HIVE_METASTORE_VERSION, hiveMetastoreVersion)} using paths: " +
- log"${MDC(LogKey.PATH, jars.mkString(", "))}")
+ log"${MDC(LogKeys.HIVE_METASTORE_VERSION, hiveMetastoreVersion)} using paths: " +
+ log"${MDC(LogKeys.PATH, jars.mkString(", "))}")
}
val isolatedLoader = if (hiveMetastoreJars == "builtin") {
@@ -360,7 +360,7 @@ private[spark] object HiveUtils extends Logging {
logInfo(
log"Initializing HiveMetastoreConnection version " +
- log"${MDC(LogKey.HIVE_METASTORE_VERSION, hiveMetastoreVersion)} using Spark classes.")
+ log"${MDC(LogKeys.HIVE_METASTORE_VERSION, hiveMetastoreVersion)} using Spark classes.")
new IsolatedClientLoader(
version = metaVersion,
sparkConf = conf,
@@ -374,7 +374,7 @@ private[spark] object HiveUtils extends Logging {
// TODO: Support for loading the jars from an already downloaded location.
logInfo(
log"Initializing HiveMetastoreConnection version " +
- log"${MDC(LogKey.HIVE_METASTORE_VERSION, hiveMetastoreVersion)} using maven.")
+ log"${MDC(LogKeys.HIVE_METASTORE_VERSION, hiveMetastoreVersion)} using maven.")
IsolatedClientLoader.forVersion(
hiveMetastoreVersion = hiveMetastoreVersion,
hadoopVersion = VersionInfo.getVersion,
diff --git a/sql/hive/src/main/scala/org/apache/spark/sql/hive/TableReader.scala b/sql/hive/src/main/scala/org/apache/spark/sql/hive/TableReader.scala
index 60970eecc2df..a93c6bd6b4e9 100644
--- a/sql/hive/src/main/scala/org/apache/spark/sql/hive/TableReader.scala
+++ b/sql/hive/src/main/scala/org/apache/spark/sql/hive/TableReader.scala
@@ -37,7 +37,7 @@ import org.apache.hadoop.mapreduce.{InputFormat => newInputClass}
import org.apache.spark.deploy.SparkHadoopUtil
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey._
+import org.apache.spark.internal.LogKeys._
import org.apache.spark.rdd.{EmptyRDD, HadoopRDD, NewHadoopRDD, RDD, UnionRDD}
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.catalyst.{InternalRow, SQLConfHelper}
diff --git a/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala b/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
index 43993f4974db..1cd60c0d3fff 100644
--- a/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
+++ b/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
@@ -47,7 +47,7 @@ import org.apache.hadoop.security.UserGroupInformation
import org.apache.spark.{SparkConf, SparkException}
import org.apache.spark.deploy.SparkHadoopUtil.SOURCE_SPARK
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey._
+import org.apache.spark.internal.LogKeys._
import org.apache.spark.metrics.source.HiveCatalogMetrics
import org.apache.spark.sql.catalyst.TableIdentifier
import org.apache.spark.sql.catalyst.analysis.{DatabaseAlreadyExistsException, NoSuchDatabaseException, NoSuchPartitionException, NoSuchPartitionsException, NoSuchTableException, PartitionsAlreadyExistException}
diff --git a/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala b/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala
index 07daa2938628..c03fed4cc318 100644
--- a/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala
+++ b/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala
@@ -38,7 +38,7 @@ import org.apache.hadoop.hive.ql.session.SessionState
import org.apache.hadoop.hive.serde.serdeConstants
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{CONFIG, CONFIG2, CONFIG3}
+import org.apache.spark.internal.LogKeys.{CONFIG, CONFIG2, CONFIG3}
import org.apache.spark.metrics.source.HiveCatalogMetrics
import org.apache.spark.sql.catalyst.{FunctionIdentifier, InternalRow}
import org.apache.spark.sql.catalyst.analysis.NoSuchPermanentFunctionException
diff --git a/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala b/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala
index 7e7c246abe97..b0570f5d3035 100644
--- a/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala
+++ b/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala
@@ -31,7 +31,7 @@ import org.apache.hadoop.hive.shims.ShimLoader
import org.apache.spark.SparkConf
import org.apache.spark.deploy.SparkSubmit
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{FALLBACK_VERSION, HADOOP_VERSION, PATH}
+import org.apache.spark.internal.LogKeys.{FALLBACK_VERSION, HADOOP_VERSION, PATH}
import org.apache.spark.sql.catalyst.util.quietly
import org.apache.spark.sql.errors.QueryExecutionErrors
import org.apache.spark.sql.hive.HiveUtils
diff --git a/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/HiveFileFormat.scala b/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/HiveFileFormat.scala
index 1b76478a5cf3..cabdddd4c475 100644
--- a/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/HiveFileFormat.scala
+++ b/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/HiveFileFormat.scala
@@ -32,7 +32,7 @@ import org.apache.hadoop.mapred.{JobConf, Reporter}
import org.apache.hadoop.mapreduce.{Job, TaskAttemptContext}
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.CLASS_NAME
+import org.apache.spark.internal.LogKeys.CLASS_NAME
import org.apache.spark.internal.config.SPECULATION_ENABLED
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.catalyst.InternalRow
diff --git a/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/HiveTempPath.scala b/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/HiveTempPath.scala
index a16191b72a8d..16edfea67e38 100644
--- a/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/HiveTempPath.scala
+++ b/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/HiveTempPath.scala
@@ -32,7 +32,7 @@ import org.apache.hadoop.hive.ql.exec.TaskRunner
import org.apache.spark.SparkException
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.PATH
+import org.apache.spark.internal.LogKeys.PATH
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.errors.QueryExecutionErrors
import org.apache.spark.sql.hive.HiveExternalCatalog
diff --git a/sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcFileOperator.scala b/sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcFileOperator.scala
index d51894542f4f..d588e9f5bd5c 100644
--- a/sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcFileOperator.scala
+++ b/sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcFileOperator.scala
@@ -26,7 +26,7 @@ import org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector
import org.apache.spark.deploy.SparkHadoopUtil
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.PATH
+import org.apache.spark.internal.LogKeys.PATH
import org.apache.spark.sql.catalyst.parser.CatalystSqlParser
import org.apache.spark.sql.errors.QueryExecutionErrors
import org.apache.spark.sql.types.StructType
diff --git a/sql/hive/src/main/scala/org/apache/spark/sql/hive/security/HiveDelegationTokenProvider.scala b/sql/hive/src/main/scala/org/apache/spark/sql/hive/security/HiveDelegationTokenProvider.scala
index 13ff721736b2..0e357d5e39b2 100644
--- a/sql/hive/src/main/scala/org/apache/spark/sql/hive/security/HiveDelegationTokenProvider.scala
+++ b/sql/hive/src/main/scala/org/apache/spark/sql/hive/security/HiveDelegationTokenProvider.scala
@@ -33,7 +33,7 @@ import org.apache.hadoop.security.token.Token
import org.apache.spark.SparkConf
import org.apache.spark.deploy.SparkHadoopUtil
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.CLASS_NAME
+import org.apache.spark.internal.LogKeys.CLASS_NAME
import org.apache.spark.internal.config.KEYTAB
import org.apache.spark.security.HadoopDelegationTokenProvider
import org.apache.spark.sql.hive.client.HiveClientImpl
diff --git a/streaming/src/main/scala/org/apache/spark/streaming/Checkpoint.scala b/streaming/src/main/scala/org/apache/spark/streaming/Checkpoint.scala
index c8dadcd8186b..a5d22f0bb5bd 100644
--- a/streaming/src/main/scala/org/apache/spark/streaming/Checkpoint.scala
+++ b/streaming/src/main/scala/org/apache/spark/streaming/Checkpoint.scala
@@ -29,7 +29,7 @@ import org.apache.hadoop.fs.{FileSystem, Path}
import org.apache.spark.{SparkConf, SparkException}
import org.apache.spark.deploy.SparkHadoopUtil
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{BACKUP_FILE, CHECKPOINT_FILE, CHECKPOINT_TIME, PATH, RETRY_COUNT, TEMP_FILE}
+import org.apache.spark.internal.LogKeys.{BACKUP_FILE, CHECKPOINT_FILE, CHECKPOINT_TIME, PATH, RETRY_COUNT, TEMP_FILE}
import org.apache.spark.internal.config.UI._
import org.apache.spark.io.CompressionCodec
import org.apache.spark.streaming.scheduler.JobGenerator
diff --git a/streaming/src/main/scala/org/apache/spark/streaming/dstream/DStream.scala b/streaming/src/main/scala/org/apache/spark/streaming/dstream/DStream.scala
index 5fe70bb0a24d..38f55f80657b 100644
--- a/streaming/src/main/scala/org/apache/spark/streaming/dstream/DStream.scala
+++ b/streaming/src/main/scala/org/apache/spark/streaming/dstream/DStream.scala
@@ -27,7 +27,7 @@ import scala.util.matching.Regex
import org.apache.spark.{SparkContext, SparkException}
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{FROM_TIME, SLIDE_DURATION, TO_TIME}
+import org.apache.spark.internal.LogKeys.{FROM_TIME, SLIDE_DURATION, TO_TIME}
import org.apache.spark.internal.io.SparkHadoopWriterUtils
import org.apache.spark.rdd.{BlockRDD, RDD, RDDOperationScope}
import org.apache.spark.storage.StorageLevel
diff --git a/streaming/src/main/scala/org/apache/spark/streaming/dstream/DStreamCheckpointData.scala b/streaming/src/main/scala/org/apache/spark/streaming/dstream/DStreamCheckpointData.scala
index aadcc6705ffa..8894b3cdc239 100644
--- a/streaming/src/main/scala/org/apache/spark/streaming/dstream/DStreamCheckpointData.scala
+++ b/streaming/src/main/scala/org/apache/spark/streaming/dstream/DStreamCheckpointData.scala
@@ -25,7 +25,7 @@ import scala.reflect.ClassTag
import org.apache.hadoop.fs.{FileSystem, Path}
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{PATH, TIME}
+import org.apache.spark.internal.LogKeys.{PATH, TIME}
import org.apache.spark.streaming.Time
import org.apache.spark.util.Utils
diff --git a/streaming/src/main/scala/org/apache/spark/streaming/dstream/FileInputDStream.scala b/streaming/src/main/scala/org/apache/spark/streaming/dstream/FileInputDStream.scala
index a6ce2fdbb0d6..d133454b832f 100644
--- a/streaming/src/main/scala/org/apache/spark/streaming/dstream/FileInputDStream.scala
+++ b/streaming/src/main/scala/org/apache/spark/streaming/dstream/FileInputDStream.scala
@@ -26,7 +26,7 @@ import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.fs.{FileStatus, FileSystem, Path}
import org.apache.hadoop.mapreduce.{InputFormat => NewInputFormat}
-import org.apache.spark.internal.LogKey.{ELAPSED_TIME, PATH}
+import org.apache.spark.internal.LogKeys.{ELAPSED_TIME, PATH}
import org.apache.spark.internal.MDC
import org.apache.spark.rdd.{RDD, UnionRDD}
import org.apache.spark.streaming._
diff --git a/streaming/src/main/scala/org/apache/spark/streaming/dstream/InputDStream.scala b/streaming/src/main/scala/org/apache/spark/streaming/dstream/InputDStream.scala
index ee4acc8d8b04..2deb388eb4b8 100644
--- a/streaming/src/main/scala/org/apache/spark/streaming/dstream/InputDStream.scala
+++ b/streaming/src/main/scala/org/apache/spark/streaming/dstream/InputDStream.scala
@@ -22,7 +22,7 @@ import java.util.Locale
import scala.reflect.ClassTag
import org.apache.spark.SparkContext
-import org.apache.spark.internal.LogKey.{LAST_VALID_TIME, TIME}
+import org.apache.spark.internal.LogKeys.{LAST_VALID_TIME, TIME}
import org.apache.spark.internal.MDC
import org.apache.spark.rdd.RDDOperationScope
import org.apache.spark.streaming.{Duration, StreamingContext, Time}
diff --git a/streaming/src/main/scala/org/apache/spark/streaming/receiver/BlockGenerator.scala b/streaming/src/main/scala/org/apache/spark/streaming/receiver/BlockGenerator.scala
index 01d11bb5a9a5..33995e6ad078 100644
--- a/streaming/src/main/scala/org/apache/spark/streaming/receiver/BlockGenerator.scala
+++ b/streaming/src/main/scala/org/apache/spark/streaming/receiver/BlockGenerator.scala
@@ -23,7 +23,7 @@ import scala.collection.mutable.ArrayBuffer
import org.apache.spark.{SparkConf, SparkException}
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.STATUS
+import org.apache.spark.internal.LogKeys.STATUS
import org.apache.spark.storage.StreamBlockId
import org.apache.spark.streaming.StreamingConf.BLOCK_INTERVAL
import org.apache.spark.streaming.util.RecurringTimer
diff --git a/streaming/src/main/scala/org/apache/spark/streaming/receiver/ReceivedBlockHandler.scala b/streaming/src/main/scala/org/apache/spark/streaming/receiver/ReceivedBlockHandler.scala
index f59c0da3db99..e513a75b6990 100644
--- a/streaming/src/main/scala/org/apache/spark/streaming/receiver/ReceivedBlockHandler.scala
+++ b/streaming/src/main/scala/org/apache/spark/streaming/receiver/ReceivedBlockHandler.scala
@@ -25,7 +25,7 @@ import org.apache.hadoop.fs.Path
import org.apache.spark.{SparkConf, SparkException}
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{EFFECTIVE_STORAGE_LEVEL, STORAGE_LEVEL, STORAGE_LEVEL_DESERIALIZED, STORAGE_LEVEL_REPLICATION}
+import org.apache.spark.internal.LogKeys.{EFFECTIVE_STORAGE_LEVEL, STORAGE_LEVEL, STORAGE_LEVEL_DESERIALIZED, STORAGE_LEVEL_REPLICATION}
import org.apache.spark.serializer.SerializerManager
import org.apache.spark.storage._
import org.apache.spark.streaming.receiver.WriteAheadLogBasedBlockHandler._
diff --git a/streaming/src/main/scala/org/apache/spark/streaming/receiver/ReceiverSupervisor.scala b/streaming/src/main/scala/org/apache/spark/streaming/receiver/ReceiverSupervisor.scala
index 5d656f03b7ac..79bfd8674b44 100644
--- a/streaming/src/main/scala/org/apache/spark/streaming/receiver/ReceiverSupervisor.scala
+++ b/streaming/src/main/scala/org/apache/spark/streaming/receiver/ReceiverSupervisor.scala
@@ -26,7 +26,7 @@ import scala.util.control.NonFatal
import org.apache.spark.SparkConf
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{DELAY, ERROR, MESSAGE, STREAM_ID}
+import org.apache.spark.internal.LogKeys.{DELAY, ERROR, MESSAGE, STREAM_ID}
import org.apache.spark.storage.StreamBlockId
import org.apache.spark.util.{ThreadUtils, Utils}
diff --git a/streaming/src/main/scala/org/apache/spark/streaming/receiver/ReceiverSupervisorImpl.scala b/streaming/src/main/scala/org/apache/spark/streaming/receiver/ReceiverSupervisorImpl.scala
index d38ecc10ad54..47beb4521950 100644
--- a/streaming/src/main/scala/org/apache/spark/streaming/receiver/ReceiverSupervisorImpl.scala
+++ b/streaming/src/main/scala/org/apache/spark/streaming/receiver/ReceiverSupervisorImpl.scala
@@ -29,7 +29,7 @@ import org.apache.hadoop.conf.Configuration
import org.apache.spark.{SparkEnv, SparkException}
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{ERROR, MESSAGE}
+import org.apache.spark.internal.LogKeys.{ERROR, MESSAGE}
import org.apache.spark.rpc.{RpcEnv, ThreadSafeRpcEndpoint}
import org.apache.spark.storage.StreamBlockId
import org.apache.spark.streaming.Time
diff --git a/streaming/src/main/scala/org/apache/spark/streaming/scheduler/JobGenerator.scala b/streaming/src/main/scala/org/apache/spark/streaming/scheduler/JobGenerator.scala
index 8bd07eeaf3d8..c0636af69038 100644
--- a/streaming/src/main/scala/org/apache/spark/streaming/scheduler/JobGenerator.scala
+++ b/streaming/src/main/scala/org/apache/spark/streaming/scheduler/JobGenerator.scala
@@ -22,7 +22,7 @@ import java.util.concurrent.TimeUnit
import scala.util.{Failure, Success, Try}
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.TIMEOUT
+import org.apache.spark.internal.LogKeys.TIMEOUT
import org.apache.spark.rdd.RDD
import org.apache.spark.streaming.{Checkpoint, CheckpointWriter, StreamingConf, Time}
import org.apache.spark.streaming.api.python.PythonDStream
diff --git a/streaming/src/main/scala/org/apache/spark/streaming/scheduler/ReceivedBlockTracker.scala b/streaming/src/main/scala/org/apache/spark/streaming/scheduler/ReceivedBlockTracker.scala
index 2d18fe89daef..f2700737384b 100644
--- a/streaming/src/main/scala/org/apache/spark/streaming/scheduler/ReceivedBlockTracker.scala
+++ b/streaming/src/main/scala/org/apache/spark/streaming/scheduler/ReceivedBlockTracker.scala
@@ -28,7 +28,7 @@ import org.apache.hadoop.fs.Path
import org.apache.spark.SparkConf
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{RECEIVED_BLOCK_INFO, RECEIVED_BLOCK_TRACKER_LOG_EVENT}
+import org.apache.spark.internal.LogKeys.{RECEIVED_BLOCK_INFO, RECEIVED_BLOCK_TRACKER_LOG_EVENT}
import org.apache.spark.network.util.JavaUtils
import org.apache.spark.streaming.Time
import org.apache.spark.streaming.util.{WriteAheadLog, WriteAheadLogUtils}
diff --git a/streaming/src/main/scala/org/apache/spark/streaming/scheduler/ReceiverTracker.scala b/streaming/src/main/scala/org/apache/spark/streaming/scheduler/ReceiverTracker.scala
index 90f97aba9b63..48273b3b593c 100644
--- a/streaming/src/main/scala/org/apache/spark/streaming/scheduler/ReceiverTracker.scala
+++ b/streaming/src/main/scala/org/apache/spark/streaming/scheduler/ReceiverTracker.scala
@@ -25,7 +25,7 @@ import scala.util.{Failure, Success}
import org.apache.spark._
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{ERROR, MESSAGE, RECEIVER_ID, RECEIVER_IDS, STREAM_ID}
+import org.apache.spark.internal.LogKeys.{ERROR, MESSAGE, RECEIVER_ID, RECEIVER_IDS, STREAM_ID}
import org.apache.spark.rdd.RDD
import org.apache.spark.rpc._
import org.apache.spark.scheduler.{ExecutorCacheTaskLocation, TaskLocation}
diff --git a/streaming/src/main/scala/org/apache/spark/streaming/util/BatchedWriteAheadLog.scala b/streaming/src/main/scala/org/apache/spark/streaming/util/BatchedWriteAheadLog.scala
index d1393da9e04f..5dcdd573c744 100644
--- a/streaming/src/main/scala/org/apache/spark/streaming/util/BatchedWriteAheadLog.scala
+++ b/streaming/src/main/scala/org/apache/spark/streaming/util/BatchedWriteAheadLog.scala
@@ -30,7 +30,7 @@ import scala.util.control.NonFatal
import org.apache.spark.SparkConf
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.RECORDS
+import org.apache.spark.internal.LogKeys.RECORDS
import org.apache.spark.network.util.JavaUtils
import org.apache.spark.util.{ThreadUtils, Utils}
diff --git a/streaming/src/main/scala/org/apache/spark/streaming/util/FileBasedWriteAheadLog.scala b/streaming/src/main/scala/org/apache/spark/streaming/util/FileBasedWriteAheadLog.scala
index 46400012209c..6162f4819016 100644
--- a/streaming/src/main/scala/org/apache/spark/streaming/util/FileBasedWriteAheadLog.scala
+++ b/streaming/src/main/scala/org/apache/spark/streaming/util/FileBasedWriteAheadLog.scala
@@ -32,7 +32,7 @@ import org.apache.hadoop.fs.Path
import org.apache.spark.SparkConf
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{RETRY_COUNT, WRITE_AHEAD_LOG_INFO}
+import org.apache.spark.internal.LogKeys.{RETRY_COUNT, WRITE_AHEAD_LOG_INFO}
import org.apache.spark.util.{CompletionIterator, ThreadUtils}
import org.apache.spark.util.ArrayImplicits._
diff --git a/streaming/src/test/scala/org/apache/spark/streaming/MasterFailureTest.scala b/streaming/src/test/scala/org/apache/spark/streaming/MasterFailureTest.scala
index afe17936043a..771e65ed40b5 100644
--- a/streaming/src/test/scala/org/apache/spark/streaming/MasterFailureTest.scala
+++ b/streaming/src/test/scala/org/apache/spark/streaming/MasterFailureTest.scala
@@ -33,7 +33,7 @@ import org.apache.hadoop.fs.Path
import org.scalatest.Assertions._
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.PATH
+import org.apache.spark.internal.LogKeys.PATH
import org.apache.spark.streaming.dstream.DStream
import org.apache.spark.util.Utils
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org