You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "panbingkun (via GitHub)" <gi...@apache.org> on 2024/04/24 00:04:45 UTC
[PR] [SPARK-47963][CORE][TESTS] Add an external LogKey usage case in UT [spark]
panbingkun opened a new pull request, #46193:
URL: https://github.com/apache/spark/pull/46193
### What changes were proposed in this pull request?
The pr aims to add `an external LogKey` usage case in UT.
### Why are the changes needed?
Give an example to developers who use the LogKey mechanism in external system.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
- Manually test.
- Pass GA.
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47963][CORE] Make the external Spark ecosystem can use structured logging mechanisms [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #46193:
URL: https://github.com/apache/spark/pull/46193#discussion_r1578922427
##########
common/utils/src/test/scala/org/apache/spark/util/StructuredLoggingSuite.scala:
##########
@@ -144,6 +146,22 @@ trait LoggingSuiteBase
}
}
+ // An enumeration value that is not defined in `LogKey`, you can define it anywhere
+ private val externalLogKey = LogKey.VALUE
+ private val externalLog = log"${MDC(externalLogKey, "External log message.")}"
+ test("Logging with external LogKey") {
Review Comment:
> If necessary in the future, I think it would be easier to implement based on the `current changes`, such as
>
> * `ILogKey`
> * `InternalLogKey` extends `ILogKey`
> * `LogKey` extends `InternalLogKey`
> * `ExternalLogKey` extends `ILogKey`
> * `ThirdPartyLogKey` extends `ExternalLogKey`
Yes, we can further expand after this PR is completed.
Personally, I think we can add a String field to the MDC to identify the namespace. For example, the namespace of Spark's internal MDC can always be `SPARK`, and the third-party Spark ecosystem can customize their own namespace. This can make the key of `MessageWithContext` become similar to `SPARK.mdcKey`, `CUSTOM_NAMESPACE.mdcKey`.
Perhaps this is not a perfect method, we can further discuss it after this PR is completed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47963][CORE] Make the external Spark ecosystem can use structured logging mechanisms [spark]
Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on code in PR #46193:
URL: https://github.com/apache/spark/pull/46193#discussion_r1578772766
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/OffsetSeq.scala:
##########
@@ -21,12 +21,12 @@ import org.json4s.{Formats, NoTypeHints}
import org.json4s.jackson.Serialization
import org.apache.spark.internal.{Logging, MDC}
-import org.apache.spark.internal.LogKey.{CONFIG, DEFAULT_VALUE, NEW_VALUE, OLD_VALUE, TIP}
+import org.apache.spark.internal.LogKeys.{CONFIG, DEFAULT_VALUE, NEW_VALUE, OLD_VALUE, TIP}
import org.apache.spark.io.CompressionCodec
import org.apache.spark.sql.RuntimeConfig
import org.apache.spark.sql.connector.read.streaming.{Offset => OffsetV2, SparkDataStream}
import org.apache.spark.sql.execution.streaming.state.{FlatMapGroupsWithStateExecHelper, StreamingAggregationStateManager, SymmetricHashJoinStateManager}
-import org.apache.spark.sql.internal.SQLConf.{FLATMAPGROUPSWITHSTATE_STATE_FORMAT_VERSION, _}
+import org.apache.spark.sql.internal.SQLConf._
Review Comment:
This is not related to the renaming of `LogKey -> LogKeys`, but it is almost more reasonable to change it to this way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47963][CORE][TESTS] Add an external LogKey usage case in UT [spark]
Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on PR #46193:
URL: https://github.com/apache/spark/pull/46193#issuecomment-2073933759
Wait a moment, I'm checking
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47963][CORE] Make the external Spark ecosystem can use structured logging mechanisms [spark]
Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang commented on code in PR #46193:
URL: https://github.com/apache/spark/pull/46193#discussion_r1580054051
##########
common/utils/src/test/scala/org/apache/spark/util/LogKeySuite.scala:
##########
@@ -57,20 +57,20 @@ class LogKeySuite
private val logKeyFilePath = getWorkspaceFilePath("common", "utils", "src", "main", "scala",
"org", "apache", "spark", "internal", "LogKey.scala")
- // regenerate the file `LogKey.scala` with its enumeration fields sorted alphabetically
+ // regenerate the file `LogKey.scala` with its members sorted alphabetically
private def regenerateLogKeyFile(
- originalKeys: Seq[LogKey], sortedKeys: Seq[LogKey]): Unit = {
+ originalKeys: Seq[String], sortedKeys: Seq[String]): Unit = {
if (originalKeys != sortedKeys) {
val logKeyFile = logKeyFilePath.toFile
- logInfo(s"Regenerating LogKey file $logKeyFile")
+ logInfo(s"Regenerating the file $logKeyFile")
val originalContents = FileUtils.readLines(logKeyFile, StandardCharsets.UTF_8)
val sortedContents = new JList[String]()
var firstMatch = false
originalContents.asScala.foreach { line =>
- if (line.trim.startsWith("val ") && line.trim.endsWith(" = Value")) {
+ if (line.trim.startsWith("case object ") && line.trim.endsWith(" extends LogKey")) {
Review Comment:
Not directly related to this PR, but we need to think about what if there are code comments around the LogKey.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47963][CORE] Make the external Spark ecosystem can use structured logging mechanisms [spark]
Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on code in PR #46193:
URL: https://github.com/apache/spark/pull/46193#discussion_r1578638859
##########
common/utils/src/main/scala/org/apache/spark/internal/LogKey.scala:
##########
@@ -16,393 +16,393 @@
*/
package org.apache.spark.internal
+trait ILogKey
+
/**
* Various keys used for mapped diagnostic contexts(MDC) in logging.
* All structured logging keys should be defined here for standardization.
*/
-object LogKey extends Enumeration {
- val ACCUMULATOR_ID = Value
- val ACTUAL_NUM_FILES = Value
- val ACTUAL_PARTITION_COLUMN = Value
- val ALPHA = Value
- val ANALYSIS_ERROR = Value
- val APP_ATTEMPT_ID = Value
- val APP_DESC = Value
- val APP_ID = Value
- val APP_NAME = Value
- val APP_STATE = Value
- val ARGS = Value
- val BACKUP_FILE = Value
- val BATCH_ID = Value
- val BLOCK_ID = Value
- val BLOCK_MANAGER_ID = Value
- val BROADCAST_ID = Value
- val BUCKET = Value
- val BYTECODE_SIZE = Value
- val CACHED_TABLE_PARTITION_METADATA_SIZE = Value
- val CACHE_AUTO_REMOVED_SIZE = Value
- val CACHE_UNTIL_HIGHEST_CONSUMED_SIZE = Value
- val CACHE_UNTIL_LAST_PRODUCED_SIZE = Value
- val CALL_SITE_LONG_FORM = Value
- val CATALOG_NAME = Value
- val CATEGORICAL_FEATURES = Value
- val CHECKPOINT_FILE = Value
- val CHECKPOINT_TIME = Value
- val CHECKSUM_FILE_NUM = Value
- val CLASS_LOADER = Value
- val CLASS_NAME = Value
- val CLUSTER_CENTROIDS = Value
- val CLUSTER_ID = Value
- val CLUSTER_LABEL = Value
- val CLUSTER_LEVEL = Value
- val CLUSTER_WEIGHT = Value
- val CODEC_LEVEL = Value
- val CODEC_NAME = Value
- val CODEGEN_STAGE_ID = Value
- val COLUMN_DATA_TYPE_SOURCE = Value
- val COLUMN_DATA_TYPE_TARGET = Value
- val COLUMN_DEFAULT_VALUE = Value
- val COLUMN_NAME = Value
- val COMMAND = Value
- val COMMAND_OUTPUT = Value
- val COMPONENT = Value
- val CONFIG = Value
- val CONFIG2 = Value
- val CONFIG3 = Value
- val CONFIG4 = Value
- val CONFIG5 = Value
- val CONSUMER = Value
- val CONTAINER = Value
- val CONTAINER_ID = Value
- val CONTAINER_STATE = Value
- val COST = Value
- val COUNT = Value
- val CROSS_VALIDATION_METRIC = Value
- val CROSS_VALIDATION_METRICS = Value
- val CSV_HEADER_COLUMN_NAME = Value
- val CSV_HEADER_COLUMN_NAMES = Value
- val CSV_HEADER_LENGTH = Value
- val CSV_SCHEMA_FIELD_NAME = Value
- val CSV_SCHEMA_FIELD_NAMES = Value
- val CSV_SOURCE = Value
- val CURRENT_PATH = Value
- val DATA = Value
- val DATABASE_NAME = Value
- val DATAFRAME_CACHE_ENTRY = Value
- val DATAFRAME_ID = Value
- val DATA_FILE_NUM = Value
- val DATA_SOURCE = Value
- val DATA_SOURCES = Value
- val DATA_SOURCE_PROVIDER = Value
- val DEFAULT_ISOLATION_LEVEL = Value
- val DEFAULT_VALUE = Value
- val DELAY = Value
- val DELEGATE = Value
- val DELTA = Value
- val DESCRIPTION = Value
- val DESIRED_PARTITIONS_SIZE = Value
- val DIFF_DELTA = Value
- val DIVISIBLE_CLUSTER_INDICES_SIZE = Value
- val DRIVER_ID = Value
- val DROPPED_PARTITIONS = Value
- val DURATION = Value
- val EFFECTIVE_STORAGE_LEVEL = Value
- val ELAPSED_TIME = Value
- val ENCODING = Value
- val END_INDEX = Value
- val END_POINT = Value
- val ENGINE = Value
- val ERROR = Value
- val ESTIMATOR_PARAMETER_MAP = Value
- val EVENT_LOOP = Value
- val EVENT_QUEUE = Value
- val EXECUTE_INFO = Value
- val EXECUTE_KEY = Value
- val EXECUTION_PLAN_LEAVES = Value
- val EXECUTOR_DESIRED_COUNT = Value
- val EXECUTOR_ENVS = Value
- val EXECUTOR_ENV_REGEX = Value
- val EXECUTOR_ID = Value
- val EXECUTOR_IDS = Value
- val EXECUTOR_LAUNCH_COMMANDS = Value
- val EXECUTOR_LAUNCH_COUNT = Value
- val EXECUTOR_RESOURCES = Value
- val EXECUTOR_STATE = Value
- val EXECUTOR_TARGET_COUNT = Value
- val EXIT_CODE = Value
- val EXPECTED_NUM_FILES = Value
- val EXPECTED_PARTITION_COLUMN = Value
- val EXPIRY_TIMESTAMP = Value
- val EXPR = Value
- val EXPR_TERMS = Value
- val EXTENDED_EXPLAIN_GENERATOR = Value
- val FAILURES = Value
- val FALLBACK_VERSION = Value
- val FEATURE_COLUMN = Value
- val FEATURE_DIMENSION = Value
- val FIELD_NAME = Value
- val FILE_ABSOLUTE_PATH = Value
- val FILE_FORMAT = Value
- val FILE_FORMAT2 = Value
- val FILE_NAME = Value
- val FILE_VERSION = Value
- val FINISH_TRIGGER_DURATION = Value
- val FROM_OFFSET = Value
- val FROM_TIME = Value
- val FUNCTION_NAME = Value
- val FUNCTION_PARAMETER = Value
- val GROUP_ID = Value
- val HADOOP_VERSION = Value
- val HASH_JOIN_KEYS = Value
- val HEARTBEAT_INTERVAL = Value
- val HISTORY_DIR = Value
- val HIVE_CLIENT_VERSION = Value
- val HIVE_METASTORE_VERSION = Value
- val HIVE_OPERATION_STATE = Value
- val HIVE_OPERATION_TYPE = Value
- val HOST = Value
- val HOST_PORT = Value
- val INCOMPATIBLE_TYPES = Value
- val INDEX = Value
- val INDEX_FILE_NUM = Value
- val INDEX_NAME = Value
- val INFERENCE_MODE = Value
- val INITIAL_CAPACITY = Value
- val INITIAL_HEARTBEAT_INTERVAL = Value
- val INIT_MODE = Value
- val INTERVAL = Value
- val ISOLATION_LEVEL = Value
- val JOB_ID = Value
- val JOIN_CONDITION = Value
- val JOIN_CONDITION_SUB_EXPR = Value
- val K8S_CONTEXT = Value
- val KAFKA_PULLS_COUNT = Value
- val KAFKA_RECORDS_PULLED_COUNT = Value
- val KEY = Value
- val KEYTAB = Value
- val LABEL_COLUMN = Value
- val LARGEST_CLUSTER_INDEX = Value
- val LAST_ACCESS_TIME = Value
- val LAST_VALID_TIME = Value
- val LATEST_BATCH_ID = Value
- val LATEST_COMMITTED_BATCH_ID = Value
- val LEARNING_RATE = Value
- val LEFT_EXPR = Value
- val LINE = Value
- val LINE_NUM = Value
- val LISTENER = Value
- val LOAD_FACTOR = Value
- val LOAD_TIME = Value
- val LOGICAL_PLAN_COLUMNS = Value
- val LOGICAL_PLAN_LEAVES = Value
- val LOG_TYPE = Value
- val LOWER_BOUND = Value
- val MALFORMATTED_STIRNG = Value
- val MASTER_URL = Value
- val MAX_ATTEMPTS = Value
- val MAX_CACHE_UNTIL_HIGHEST_CONSUMED_SIZE = Value
- val MAX_CACHE_UNTIL_LAST_PRODUCED_SIZE = Value
- val MAX_CAPACITY = Value
- val MAX_CATEGORIES = Value
- val MAX_EXECUTOR_FAILURES = Value
- val MAX_FILE_VERSION = Value
- val MAX_MEMORY_SIZE = Value
- val MAX_PARTITIONS_SIZE = Value
- val MAX_SIZE = Value
- val MAX_TABLE_PARTITION_METADATA_SIZE = Value
- val MEMORY_SIZE = Value
- val MERGE_DIR_NAME = Value
- val MESSAGE = Value
- val METHOD_NAME = Value
- val MIN_FREQUENT_PATTERN_COUNT = Value
- val MIN_POINT_PER_CLUSTER = Value
- val MIN_SIZE = Value
- val MODEL_WEIGHTS = Value
- val NAMESPACE = Value
- val NEW_FEATURE_COLUMN_NAME = Value
- val NEW_LABEL_COLUMN_NAME = Value
- val NEW_PATH = Value
- val NEW_VALUE = Value
- val NODES = Value
- val NODE_LOCATION = Value
- val NORM = Value
- val NUM_BIN = Value
- val NUM_CLASSES = Value
- val NUM_COLUMNS = Value
- val NUM_EXAMPLES = Value
- val NUM_FEATURES = Value
- val NUM_FILES = Value
- val NUM_FREQUENT_ITEMS = Value
- val NUM_ITERATIONS = Value
- val NUM_LOCAL_FREQUENT_PATTERN = Value
- val NUM_PARTITION = Value
- val NUM_POINT = Value
- val NUM_PREFIXES = Value
- val NUM_SEQUENCES = Value
- val OBJECT_ID = Value
- val OFFSET = Value
- val OFFSETS = Value
- val OLD_BLOCK_MANAGER_ID = Value
- val OLD_VALUE = Value
- val OPTIMIZED_PLAN_COLUMNS = Value
- val OPTIMIZER_CLASS_NAME = Value
- val OPTIONS = Value
- val OP_ID = Value
- val OP_TYPE = Value
- val OVERHEAD_MEMORY_SIZE = Value
- val PARSE_MODE = Value
- val PARTITIONED_FILE_READER = Value
- val PARTITIONS_SIZE = Value
- val PARTITION_ID = Value
- val PARTITION_SPECIFICATION = Value
- val PARTITION_SPECS = Value
- val PATH = Value
- val PATHS = Value
- val PIPELINE_STAGE_UID = Value
- val POD_COUNT = Value
- val POD_ID = Value
- val POD_NAME = Value
- val POD_NAMESPACE = Value
- val POD_PHASE = Value
- val POD_SHARED_SLOT_COUNT = Value
- val POD_STATE = Value
- val POD_TARGET_COUNT = Value
- val POLICY = Value
- val PORT = Value
- val PRINCIPAL = Value
- val PROCESSING_TIME = Value
- val PRODUCER_ID = Value
- val PROVIDER = Value
- val PVC_METADATA_NAME = Value
- val QUERY_CACHE_VALUE = Value
- val QUERY_HINT = Value
- val QUERY_ID = Value
- val QUERY_PLAN = Value
- val QUERY_PLAN_COMPARISON = Value
- val QUERY_PLAN_LENGTH_ACTUAL = Value
- val QUERY_PLAN_LENGTH_MAX = Value
- val QUERY_RUN_ID = Value
- val RANGE = Value
- val RDD_ID = Value
- val READ_LIMIT = Value
- val REASON = Value
- val REATTACHABLE = Value
- val RECEIVED_BLOCK_INFO = Value
- val RECEIVED_BLOCK_TRACKER_LOG_EVENT = Value
- val RECEIVER_ID = Value
- val RECEIVER_IDS = Value
- val RECORDS = Value
- val REDACTED_STATEMENT = Value
- val REDUCE_ID = Value
- val RELATION_NAME = Value
- val RELATIVE_TOLERANCE = Value
- val REMAINING_PARTITIONS = Value
- val REPORT_DETAILS = Value
- val RESOURCE = Value
- val RESOURCE_NAME = Value
- val RESOURCE_PROFILE_ID = Value
- val RESOURCE_PROFILE_IDS = Value
- val RETRY_COUNT = Value
- val RETRY_INTERVAL = Value
- val RIGHT_EXPR = Value
- val RMSE = Value
- val RPC_ENDPOINT_REF = Value
- val RULE_BATCH_NAME = Value
- val RULE_NAME = Value
- val RULE_NUMBER_OF_RUNS = Value
- val RUN_ID = Value
- val SCHEMA = Value
- val SCHEMA2 = Value
- val SERVICE_NAME = Value
- val SESSION_HOLD_INFO = Value
- val SESSION_ID = Value
- val SESSION_KEY = Value
- val SHARD_ID = Value
- val SHUFFLE_BLOCK_INFO = Value
- val SHUFFLE_ID = Value
- val SHUFFLE_MERGE_ID = Value
- val SHUFFLE_SERVICE_NAME = Value
- val SIZE = Value
- val SLEEP_TIME = Value
- val SLIDE_DURATION = Value
- val SMALLEST_CLUSTER_INDEX = Value
- val SPARK_DATA_STREAM = Value
- val SPARK_PLAN_ID = Value
- val SQL_TEXT = Value
- val SRC_PATH = Value
- val STAGE_ID = Value
- val START_INDEX = Value
- val STATEMENT_ID = Value
- val STATE_STORE_PROVIDER = Value
- val STATUS = Value
- val STORAGE_LEVEL = Value
- val STORAGE_LEVEL_DESERIALIZED = Value
- val STORAGE_LEVEL_REPLICATION = Value
- val STORE_ID = Value
- val STREAM_ID = Value
- val STREAM_NAME = Value
- val SUBMISSION_ID = Value
- val SUBSAMPLING_RATE = Value
- val SUB_QUERY = Value
- val TABLE_NAME = Value
- val TABLE_TYPES = Value
- val TARGET_PATH = Value
- val TASK_ATTEMPT_ID = Value
- val TASK_ID = Value
- val TASK_NAME = Value
- val TASK_SET_NAME = Value
- val TASK_STATE = Value
- val TEMP_FILE = Value
- val TEMP_PATH = Value
- val TEST_SIZE = Value
- val THREAD = Value
- val THREAD_NAME = Value
- val TID = Value
- val TIME = Value
- val TIMEOUT = Value
- val TIMER = Value
- val TIMER_LABEL = Value
- val TIME_UNITS = Value
- val TIP = Value
- val TOKEN_REGEX = Value
- val TOPIC = Value
- val TOPIC_PARTITION = Value
- val TOPIC_PARTITIONS = Value
- val TOPIC_PARTITION_OFFSET = Value
- val TOPIC_PARTITION_OFFSET_RANGE = Value
- val TOTAL = Value
- val TOTAL_EFFECTIVE_TIME = Value
- val TOTAL_RECORDS_READ = Value
- val TOTAL_TIME = Value
- val TOTAL_TIME_READ = Value
- val TO_TIME = Value
- val TRAINING_SIZE = Value
- val TRAIN_VALIDATION_SPLIT_METRIC = Value
- val TRAIN_VALIDATION_SPLIT_METRICS = Value
- val TRAIN_WORD_COUNT = Value
- val TREE_NODE = Value
- val TRIGGER_INTERVAL = Value
- val UI_FILTER = Value
- val UI_FILTER_PARAMS = Value
- val UI_PROXY_BASE = Value
- val UNSUPPORTED_EXPR = Value
- val UNSUPPORTED_HINT_REASON = Value
- val UNTIL_OFFSET = Value
- val UPPER_BOUND = Value
- val URI = Value
- val USER_ID = Value
- val USER_NAME = Value
- val VALUE = Value
- val VIRTUAL_CORES = Value
- val VOCAB_SIZE = Value
- val WAIT_RESULT_TIME = Value
- val WAIT_SEND_TIME = Value
- val WAIT_TIME = Value
- val WATERMARK_CONSTRAINT = Value
- val WEIGHTED_NUM = Value
- val WORKER_URL = Value
- val WRITE_AHEAD_LOG_INFO = Value
- val XSD_PATH = Value
-
- type LogKey = Value
+object LogKey extends ILogKey {
Review Comment:
@gengliangwang
There is `a small` question, I actually think this name is `not good`, but I made a few compromises. If we change this name, it will bring import changes to all classes that use it. This may involve many files, can we accept it? eg:
```
import org.apache.spark.internal.LogKey.{CODEC_LEVEL, CODEC_NAME, CONFIG, PATH}
=>
import org.apache.spark.internal.LogKeys.{CODEC_LEVEL, CODEC_NAME, CONFIG, PATH}
```
<img width="599" alt="image" src="https://github.com/apache/spark/assets/15246973/dcde51ff-9b0d-4bfe-9e6e-4893aa05f257">
About `200+`, it sounds like it's okay too, isn't it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47963][CORE] Make the external Spark ecosystem can use structured logging mechanisms [spark]
Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on code in PR #46193:
URL: https://github.com/apache/spark/pull/46193#discussion_r1578780993
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala:
##########
@@ -20,15 +20,15 @@ package org.apache.spark.sql.catalyst.optimizer
import scala.collection.mutable
import org.apache.spark.SparkException
-import org.apache.spark.internal.LogKey._
+import org.apache.spark.internal.LogKeys._
import org.apache.spark.internal.MDC
import org.apache.spark.sql.catalyst.SQLConfHelper
import org.apache.spark.sql.catalyst.analysis._
import org.apache.spark.sql.catalyst.catalog.{InMemoryCatalog, SessionCatalog}
import org.apache.spark.sql.catalyst.expressions._
import org.apache.spark.sql.catalyst.expressions.aggregate._
import org.apache.spark.sql.catalyst.plans._
-import org.apache.spark.sql.catalyst.plans.logical.{RepartitionOperation, _}
+import org.apache.spark.sql.catalyst.plans.logical._
Review Comment:
This is not related to the renaming of `LogKey -> LogKeys`, but it is almost more reasonable to change it to this way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47963][CORE][TESTS] Add an external LogKey usage case in UT [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #46193:
URL: https://github.com/apache/spark/pull/46193#discussion_r1577167593
##########
common/utils/src/test/scala/org/apache/spark/util/StructuredLoggingSuite.scala:
##########
@@ -144,6 +146,22 @@ trait LoggingSuiteBase
}
}
+ // An enumeration value that is not defined in `LogKey`, you can define it anywhere
+ private val externalLogKey = LogKey.VALUE
+ private val externalLog = log"${MDC(externalLogKey, "External log message.")}"
+ test("Logging with external LogKey") {
Review Comment:
In https://github.com/apache/spark/pull/46186/files, I try to relax the type of `MDC.key` to `Enumeration#Value`, which allows us to extend the use of other enumeration types as `MDC.key` to enhance a bit of extensibility.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47963][CORE] Make the external Spark ecosystem can use structured logging mechanisms [spark]
Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on code in PR #46193:
URL: https://github.com/apache/spark/pull/46193#discussion_r1578791101
##########
core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala:
##########
@@ -33,8 +33,8 @@ import org.apache.spark.InternalAccumulator.{input, shuffleRead}
import org.apache.spark.TaskState.TaskState
import org.apache.spark.errors.SparkCoreErrors
import org.apache.spark.executor.ExecutorMetrics
-import org.apache.spark.internal.{config, Logging, LogKey, MDC}
-import org.apache.spark.internal.LogKey.{REASON, TASK_SET_NAME, TASK_STATE, TID}
+import org.apache.spark.internal.{config, Logging, LogKeys, MDC}
+import org.apache.spark.internal.LogKeys.{REASON, TASK_SET_NAME, TASK_STATE, TID}
Review Comment:
merge to `import org.apache.spark.internal.LogKeys`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47963][CORE] Make the external Spark ecosystem can use structured logging mechanisms [spark]
Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang closed pull request #46193: [SPARK-47963][CORE] Make the external Spark ecosystem can use structured logging mechanisms
URL: https://github.com/apache/spark/pull/46193
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47963][CORE] Make the external Spark ecosystem can use structured logging mechanisms [spark]
Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on code in PR #46193:
URL: https://github.com/apache/spark/pull/46193#discussion_r1578640785
##########
common/utils/src/main/scala/org/apache/spark/internal/LogKey.scala:
##########
@@ -16,393 +16,393 @@
*/
package org.apache.spark.internal
+trait ILogKey
+
/**
* Various keys used for mapped diagnostic contexts(MDC) in logging.
* All structured logging keys should be defined here for standardization.
*/
-object LogKey extends Enumeration {
- val ACCUMULATOR_ID = Value
- val ACTUAL_NUM_FILES = Value
- val ACTUAL_PARTITION_COLUMN = Value
- val ALPHA = Value
- val ANALYSIS_ERROR = Value
- val APP_ATTEMPT_ID = Value
- val APP_DESC = Value
- val APP_ID = Value
- val APP_NAME = Value
- val APP_STATE = Value
- val ARGS = Value
- val BACKUP_FILE = Value
- val BATCH_ID = Value
- val BLOCK_ID = Value
- val BLOCK_MANAGER_ID = Value
- val BROADCAST_ID = Value
- val BUCKET = Value
- val BYTECODE_SIZE = Value
- val CACHED_TABLE_PARTITION_METADATA_SIZE = Value
- val CACHE_AUTO_REMOVED_SIZE = Value
- val CACHE_UNTIL_HIGHEST_CONSUMED_SIZE = Value
- val CACHE_UNTIL_LAST_PRODUCED_SIZE = Value
- val CALL_SITE_LONG_FORM = Value
- val CATALOG_NAME = Value
- val CATEGORICAL_FEATURES = Value
- val CHECKPOINT_FILE = Value
- val CHECKPOINT_TIME = Value
- val CHECKSUM_FILE_NUM = Value
- val CLASS_LOADER = Value
- val CLASS_NAME = Value
- val CLUSTER_CENTROIDS = Value
- val CLUSTER_ID = Value
- val CLUSTER_LABEL = Value
- val CLUSTER_LEVEL = Value
- val CLUSTER_WEIGHT = Value
- val CODEC_LEVEL = Value
- val CODEC_NAME = Value
- val CODEGEN_STAGE_ID = Value
- val COLUMN_DATA_TYPE_SOURCE = Value
- val COLUMN_DATA_TYPE_TARGET = Value
- val COLUMN_DEFAULT_VALUE = Value
- val COLUMN_NAME = Value
- val COMMAND = Value
- val COMMAND_OUTPUT = Value
- val COMPONENT = Value
- val CONFIG = Value
- val CONFIG2 = Value
- val CONFIG3 = Value
- val CONFIG4 = Value
- val CONFIG5 = Value
- val CONSUMER = Value
- val CONTAINER = Value
- val CONTAINER_ID = Value
- val CONTAINER_STATE = Value
- val COST = Value
- val COUNT = Value
- val CROSS_VALIDATION_METRIC = Value
- val CROSS_VALIDATION_METRICS = Value
- val CSV_HEADER_COLUMN_NAME = Value
- val CSV_HEADER_COLUMN_NAMES = Value
- val CSV_HEADER_LENGTH = Value
- val CSV_SCHEMA_FIELD_NAME = Value
- val CSV_SCHEMA_FIELD_NAMES = Value
- val CSV_SOURCE = Value
- val CURRENT_PATH = Value
- val DATA = Value
- val DATABASE_NAME = Value
- val DATAFRAME_CACHE_ENTRY = Value
- val DATAFRAME_ID = Value
- val DATA_FILE_NUM = Value
- val DATA_SOURCE = Value
- val DATA_SOURCES = Value
- val DATA_SOURCE_PROVIDER = Value
- val DEFAULT_ISOLATION_LEVEL = Value
- val DEFAULT_VALUE = Value
- val DELAY = Value
- val DELEGATE = Value
- val DELTA = Value
- val DESCRIPTION = Value
- val DESIRED_PARTITIONS_SIZE = Value
- val DIFF_DELTA = Value
- val DIVISIBLE_CLUSTER_INDICES_SIZE = Value
- val DRIVER_ID = Value
- val DROPPED_PARTITIONS = Value
- val DURATION = Value
- val EFFECTIVE_STORAGE_LEVEL = Value
- val ELAPSED_TIME = Value
- val ENCODING = Value
- val END_INDEX = Value
- val END_POINT = Value
- val ENGINE = Value
- val ERROR = Value
- val ESTIMATOR_PARAMETER_MAP = Value
- val EVENT_LOOP = Value
- val EVENT_QUEUE = Value
- val EXECUTE_INFO = Value
- val EXECUTE_KEY = Value
- val EXECUTION_PLAN_LEAVES = Value
- val EXECUTOR_DESIRED_COUNT = Value
- val EXECUTOR_ENVS = Value
- val EXECUTOR_ENV_REGEX = Value
- val EXECUTOR_ID = Value
- val EXECUTOR_IDS = Value
- val EXECUTOR_LAUNCH_COMMANDS = Value
- val EXECUTOR_LAUNCH_COUNT = Value
- val EXECUTOR_RESOURCES = Value
- val EXECUTOR_STATE = Value
- val EXECUTOR_TARGET_COUNT = Value
- val EXIT_CODE = Value
- val EXPECTED_NUM_FILES = Value
- val EXPECTED_PARTITION_COLUMN = Value
- val EXPIRY_TIMESTAMP = Value
- val EXPR = Value
- val EXPR_TERMS = Value
- val EXTENDED_EXPLAIN_GENERATOR = Value
- val FAILURES = Value
- val FALLBACK_VERSION = Value
- val FEATURE_COLUMN = Value
- val FEATURE_DIMENSION = Value
- val FIELD_NAME = Value
- val FILE_ABSOLUTE_PATH = Value
- val FILE_FORMAT = Value
- val FILE_FORMAT2 = Value
- val FILE_NAME = Value
- val FILE_VERSION = Value
- val FINISH_TRIGGER_DURATION = Value
- val FROM_OFFSET = Value
- val FROM_TIME = Value
- val FUNCTION_NAME = Value
- val FUNCTION_PARAMETER = Value
- val GROUP_ID = Value
- val HADOOP_VERSION = Value
- val HASH_JOIN_KEYS = Value
- val HEARTBEAT_INTERVAL = Value
- val HISTORY_DIR = Value
- val HIVE_CLIENT_VERSION = Value
- val HIVE_METASTORE_VERSION = Value
- val HIVE_OPERATION_STATE = Value
- val HIVE_OPERATION_TYPE = Value
- val HOST = Value
- val HOST_PORT = Value
- val INCOMPATIBLE_TYPES = Value
- val INDEX = Value
- val INDEX_FILE_NUM = Value
- val INDEX_NAME = Value
- val INFERENCE_MODE = Value
- val INITIAL_CAPACITY = Value
- val INITIAL_HEARTBEAT_INTERVAL = Value
- val INIT_MODE = Value
- val INTERVAL = Value
- val ISOLATION_LEVEL = Value
- val JOB_ID = Value
- val JOIN_CONDITION = Value
- val JOIN_CONDITION_SUB_EXPR = Value
- val K8S_CONTEXT = Value
- val KAFKA_PULLS_COUNT = Value
- val KAFKA_RECORDS_PULLED_COUNT = Value
- val KEY = Value
- val KEYTAB = Value
- val LABEL_COLUMN = Value
- val LARGEST_CLUSTER_INDEX = Value
- val LAST_ACCESS_TIME = Value
- val LAST_VALID_TIME = Value
- val LATEST_BATCH_ID = Value
- val LATEST_COMMITTED_BATCH_ID = Value
- val LEARNING_RATE = Value
- val LEFT_EXPR = Value
- val LINE = Value
- val LINE_NUM = Value
- val LISTENER = Value
- val LOAD_FACTOR = Value
- val LOAD_TIME = Value
- val LOGICAL_PLAN_COLUMNS = Value
- val LOGICAL_PLAN_LEAVES = Value
- val LOG_TYPE = Value
- val LOWER_BOUND = Value
- val MALFORMATTED_STIRNG = Value
- val MASTER_URL = Value
- val MAX_ATTEMPTS = Value
- val MAX_CACHE_UNTIL_HIGHEST_CONSUMED_SIZE = Value
- val MAX_CACHE_UNTIL_LAST_PRODUCED_SIZE = Value
- val MAX_CAPACITY = Value
- val MAX_CATEGORIES = Value
- val MAX_EXECUTOR_FAILURES = Value
- val MAX_FILE_VERSION = Value
- val MAX_MEMORY_SIZE = Value
- val MAX_PARTITIONS_SIZE = Value
- val MAX_SIZE = Value
- val MAX_TABLE_PARTITION_METADATA_SIZE = Value
- val MEMORY_SIZE = Value
- val MERGE_DIR_NAME = Value
- val MESSAGE = Value
- val METHOD_NAME = Value
- val MIN_FREQUENT_PATTERN_COUNT = Value
- val MIN_POINT_PER_CLUSTER = Value
- val MIN_SIZE = Value
- val MODEL_WEIGHTS = Value
- val NAMESPACE = Value
- val NEW_FEATURE_COLUMN_NAME = Value
- val NEW_LABEL_COLUMN_NAME = Value
- val NEW_PATH = Value
- val NEW_VALUE = Value
- val NODES = Value
- val NODE_LOCATION = Value
- val NORM = Value
- val NUM_BIN = Value
- val NUM_CLASSES = Value
- val NUM_COLUMNS = Value
- val NUM_EXAMPLES = Value
- val NUM_FEATURES = Value
- val NUM_FILES = Value
- val NUM_FREQUENT_ITEMS = Value
- val NUM_ITERATIONS = Value
- val NUM_LOCAL_FREQUENT_PATTERN = Value
- val NUM_PARTITION = Value
- val NUM_POINT = Value
- val NUM_PREFIXES = Value
- val NUM_SEQUENCES = Value
- val OBJECT_ID = Value
- val OFFSET = Value
- val OFFSETS = Value
- val OLD_BLOCK_MANAGER_ID = Value
- val OLD_VALUE = Value
- val OPTIMIZED_PLAN_COLUMNS = Value
- val OPTIMIZER_CLASS_NAME = Value
- val OPTIONS = Value
- val OP_ID = Value
- val OP_TYPE = Value
- val OVERHEAD_MEMORY_SIZE = Value
- val PARSE_MODE = Value
- val PARTITIONED_FILE_READER = Value
- val PARTITIONS_SIZE = Value
- val PARTITION_ID = Value
- val PARTITION_SPECIFICATION = Value
- val PARTITION_SPECS = Value
- val PATH = Value
- val PATHS = Value
- val PIPELINE_STAGE_UID = Value
- val POD_COUNT = Value
- val POD_ID = Value
- val POD_NAME = Value
- val POD_NAMESPACE = Value
- val POD_PHASE = Value
- val POD_SHARED_SLOT_COUNT = Value
- val POD_STATE = Value
- val POD_TARGET_COUNT = Value
- val POLICY = Value
- val PORT = Value
- val PRINCIPAL = Value
- val PROCESSING_TIME = Value
- val PRODUCER_ID = Value
- val PROVIDER = Value
- val PVC_METADATA_NAME = Value
- val QUERY_CACHE_VALUE = Value
- val QUERY_HINT = Value
- val QUERY_ID = Value
- val QUERY_PLAN = Value
- val QUERY_PLAN_COMPARISON = Value
- val QUERY_PLAN_LENGTH_ACTUAL = Value
- val QUERY_PLAN_LENGTH_MAX = Value
- val QUERY_RUN_ID = Value
- val RANGE = Value
- val RDD_ID = Value
- val READ_LIMIT = Value
- val REASON = Value
- val REATTACHABLE = Value
- val RECEIVED_BLOCK_INFO = Value
- val RECEIVED_BLOCK_TRACKER_LOG_EVENT = Value
- val RECEIVER_ID = Value
- val RECEIVER_IDS = Value
- val RECORDS = Value
- val REDACTED_STATEMENT = Value
- val REDUCE_ID = Value
- val RELATION_NAME = Value
- val RELATIVE_TOLERANCE = Value
- val REMAINING_PARTITIONS = Value
- val REPORT_DETAILS = Value
- val RESOURCE = Value
- val RESOURCE_NAME = Value
- val RESOURCE_PROFILE_ID = Value
- val RESOURCE_PROFILE_IDS = Value
- val RETRY_COUNT = Value
- val RETRY_INTERVAL = Value
- val RIGHT_EXPR = Value
- val RMSE = Value
- val RPC_ENDPOINT_REF = Value
- val RULE_BATCH_NAME = Value
- val RULE_NAME = Value
- val RULE_NUMBER_OF_RUNS = Value
- val RUN_ID = Value
- val SCHEMA = Value
- val SCHEMA2 = Value
- val SERVICE_NAME = Value
- val SESSION_HOLD_INFO = Value
- val SESSION_ID = Value
- val SESSION_KEY = Value
- val SHARD_ID = Value
- val SHUFFLE_BLOCK_INFO = Value
- val SHUFFLE_ID = Value
- val SHUFFLE_MERGE_ID = Value
- val SHUFFLE_SERVICE_NAME = Value
- val SIZE = Value
- val SLEEP_TIME = Value
- val SLIDE_DURATION = Value
- val SMALLEST_CLUSTER_INDEX = Value
- val SPARK_DATA_STREAM = Value
- val SPARK_PLAN_ID = Value
- val SQL_TEXT = Value
- val SRC_PATH = Value
- val STAGE_ID = Value
- val START_INDEX = Value
- val STATEMENT_ID = Value
- val STATE_STORE_PROVIDER = Value
- val STATUS = Value
- val STORAGE_LEVEL = Value
- val STORAGE_LEVEL_DESERIALIZED = Value
- val STORAGE_LEVEL_REPLICATION = Value
- val STORE_ID = Value
- val STREAM_ID = Value
- val STREAM_NAME = Value
- val SUBMISSION_ID = Value
- val SUBSAMPLING_RATE = Value
- val SUB_QUERY = Value
- val TABLE_NAME = Value
- val TABLE_TYPES = Value
- val TARGET_PATH = Value
- val TASK_ATTEMPT_ID = Value
- val TASK_ID = Value
- val TASK_NAME = Value
- val TASK_SET_NAME = Value
- val TASK_STATE = Value
- val TEMP_FILE = Value
- val TEMP_PATH = Value
- val TEST_SIZE = Value
- val THREAD = Value
- val THREAD_NAME = Value
- val TID = Value
- val TIME = Value
- val TIMEOUT = Value
- val TIMER = Value
- val TIMER_LABEL = Value
- val TIME_UNITS = Value
- val TIP = Value
- val TOKEN_REGEX = Value
- val TOPIC = Value
- val TOPIC_PARTITION = Value
- val TOPIC_PARTITIONS = Value
- val TOPIC_PARTITION_OFFSET = Value
- val TOPIC_PARTITION_OFFSET_RANGE = Value
- val TOTAL = Value
- val TOTAL_EFFECTIVE_TIME = Value
- val TOTAL_RECORDS_READ = Value
- val TOTAL_TIME = Value
- val TOTAL_TIME_READ = Value
- val TO_TIME = Value
- val TRAINING_SIZE = Value
- val TRAIN_VALIDATION_SPLIT_METRIC = Value
- val TRAIN_VALIDATION_SPLIT_METRICS = Value
- val TRAIN_WORD_COUNT = Value
- val TREE_NODE = Value
- val TRIGGER_INTERVAL = Value
- val UI_FILTER = Value
- val UI_FILTER_PARAMS = Value
- val UI_PROXY_BASE = Value
- val UNSUPPORTED_EXPR = Value
- val UNSUPPORTED_HINT_REASON = Value
- val UNTIL_OFFSET = Value
- val UPPER_BOUND = Value
- val URI = Value
- val USER_ID = Value
- val USER_NAME = Value
- val VALUE = Value
- val VIRTUAL_CORES = Value
- val VOCAB_SIZE = Value
- val WAIT_RESULT_TIME = Value
- val WAIT_SEND_TIME = Value
- val WAIT_TIME = Value
- val WATERMARK_CONSTRAINT = Value
- val WEIGHTED_NUM = Value
- val WORKER_URL = Value
- val WRITE_AHEAD_LOG_INFO = Value
- val XSD_PATH = Value
-
- type LogKey = Value
+object LogKey extends ILogKey {
Review Comment:
`Also, there seems no need to extend any trait.`
Yeah, You are right, this is an intermediate process that I forgot to remove from my changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47963][CORE][TESTS] Add an external LogKey usage case in UT [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #46193:
URL: https://github.com/apache/spark/pull/46193#discussion_r1577159936
##########
common/utils/src/test/scala/org/apache/spark/util/StructuredLoggingSuite.scala:
##########
@@ -144,6 +146,22 @@ trait LoggingSuiteBase
}
}
+ // An enumeration value that is not defined in `LogKey`, you can define it anywhere
+ private val externalLogKey = LogKey.VALUE
+ private val externalLog = log"${MDC(externalLogKey, "External log message.")}"
+ test("Logging with external LogKey") {
Review Comment:
For this case, the key for `externalLog.context` will be an empty string. Is this expected?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47963][CORE] Make the external Spark ecosystem can use structured logging mechanisms [spark]
Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on code in PR #46193:
URL: https://github.com/apache/spark/pull/46193#discussion_r1577397848
##########
common/utils/src/test/scala/org/apache/spark/util/StructuredLoggingSuite.scala:
##########
@@ -144,6 +146,22 @@ trait LoggingSuiteBase
}
}
+ // An enumeration value that is not defined in `LogKey`, you can define it anywhere
+ private val externalLogKey = LogKey.VALUE
+ private val externalLog = log"${MDC(externalLogKey, "External log message.")}"
+ test("Logging with external LogKey") {
Review Comment:
@LuciferYang @gengliangwang
I have updated a version that removes `enumeration`. When using structured logging mechanisms in external Spark related ecosystems, the following format can be used:
```
case object CUSTOME_LOG_KEY extends ILogKey
```
At present, this form is more in line with the traditional Spark style, and we can `constrain` the `class` of MDC parameter `key`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47963][CORE] Make the external Spark ecosystem can use structured logging mechanisms [spark]
Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on code in PR #46193:
URL: https://github.com/apache/spark/pull/46193#discussion_r1577686240
##########
common/utils/src/test/scala/org/apache/spark/util/StructuredLoggingSuite.scala:
##########
@@ -144,6 +146,22 @@ trait LoggingSuiteBase
}
}
+ // An enumeration value that is not defined in `LogKey`, you can define it anywhere
+ private val externalLogKey = LogKey.VALUE
+ private val externalLog = log"${MDC(externalLogKey, "External log message.")}"
+ test("Logging with external LogKey") {
Review Comment:
If necessary in the future, I think it would be easier to implement based on the `current changes`, such as
- `ILogKey`
- `InternalLogKey` extends `ILogKey`
- `LogKey` extends `InternalLogKey`
- `ExternalLogKey` extends `ILogKey`
- `ThirdPartyLogKey` extends `ExternalLogKey`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47963][CORE] Make the external Spark ecosystem can use structured logging mechanisms [spark]
Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on code in PR #46193:
URL: https://github.com/apache/spark/pull/46193#discussion_r1578797076
##########
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsAllocator.scala:
##########
@@ -33,8 +33,8 @@ import org.apache.spark.deploy.k8s.Config._
import org.apache.spark.deploy.k8s.Constants._
import org.apache.spark.deploy.k8s.KubernetesConf
import org.apache.spark.deploy.k8s.KubernetesUtils.addOwnerReference
-import org.apache.spark.internal.{Logging, LogKey, MDC}
-import org.apache.spark.internal.LogKey.{COUNT, EXECUTOR_IDS, TIMEOUT}
+import org.apache.spark.internal.{Logging, LogKeys, MDC}
+import org.apache.spark.internal.LogKeys.{COUNT, EXECUTOR_IDS, TIMEOUT}
Review Comment:
merge to `import org.apache.spark.internal.LogKeys`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47963][CORE] Make the external Spark ecosystem can use structured logging mechanisms [spark]
Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on code in PR #46193:
URL: https://github.com/apache/spark/pull/46193#discussion_r1578623558
##########
common/utils/src/test/scala/org/apache/spark/util/StructuredLoggingSuite.scala:
##########
@@ -144,6 +146,22 @@ trait LoggingSuiteBase
}
}
+ // An enumeration value that is not defined in `LogKey`, you can define it anywhere
+ private val externalLogKey = LogKey.VALUE
+ private val externalLog = log"${MDC(externalLogKey, "External log message.")}"
+ test("Logging with external LogKey") {
Review Comment:
@gengliangwang
What do you think of the current modification approach?
If we define it as `a string`, it feels like there is `no binding requirement` for `LogKey` at all
Perhaps we need more eyes to review the changes above?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47963][CORE] Make the external Spark ecosystem can use structured logging mechanisms [spark]
Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on code in PR #46193:
URL: https://github.com/apache/spark/pull/46193#discussion_r1578768196
##########
common/utils/src/main/scala/org/apache/spark/internal/README.md:
##########
@@ -2,7 +2,7 @@
## LogKey
-LogKeys serve as identifiers for mapped diagnostic contexts (MDC) within logs. Follow these guidelines when adding new LogKeys:
+LogKey serve as identifiers for mapped diagnostic contexts (MDC) within logs. Follow these guidelines when adding new LogKey:
Review Comment:
I'm not sure if it's more accurate to call it `LogKeys` or `LogKey`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47963][CORE] Make the external Spark ecosystem can use structured logging mechanisms [spark]
Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang commented on code in PR #46193:
URL: https://github.com/apache/spark/pull/46193#discussion_r1578903240
##########
common/utils/src/main/scala/org/apache/spark/internal/LogKey.scala:
##########
@@ -16,399 +16,399 @@
*/
package org.apache.spark.internal
+trait LogKey
+
/**
* Various keys used for mapped diagnostic contexts(MDC) in logging.
* All structured logging keys should be defined here for standardization.
*/
-object LogKey extends Enumeration {
- val ACCUMULATOR_ID = Value
- val ACTUAL_NUM_FILES = Value
- val ACTUAL_PARTITION_COLUMN = Value
- val ALPHA = Value
- val ANALYSIS_ERROR = Value
- val APP_ATTEMPT_ID = Value
- val APP_DESC = Value
- val APP_ID = Value
- val APP_NAME = Value
- val APP_STATE = Value
- val ARGS = Value
- val BACKUP_FILE = Value
- val BATCH_ID = Value
- val BATCH_WRITE = Value
- val BLOCK_ID = Value
- val BLOCK_MANAGER_ID = Value
- val BROADCAST_ID = Value
- val BUCKET = Value
- val BYTECODE_SIZE = Value
- val CACHED_TABLE_PARTITION_METADATA_SIZE = Value
- val CACHE_AUTO_REMOVED_SIZE = Value
- val CACHE_UNTIL_HIGHEST_CONSUMED_SIZE = Value
- val CACHE_UNTIL_LAST_PRODUCED_SIZE = Value
- val CALL_SITE_LONG_FORM = Value
- val CATALOG_NAME = Value
- val CATEGORICAL_FEATURES = Value
- val CHECKPOINT_FILE = Value
- val CHECKPOINT_TIME = Value
- val CHECKSUM_FILE_NUM = Value
- val CLASS_LOADER = Value
- val CLASS_NAME = Value
- val CLUSTER_CENTROIDS = Value
- val CLUSTER_ID = Value
- val CLUSTER_LABEL = Value
- val CLUSTER_LEVEL = Value
- val CLUSTER_WEIGHT = Value
- val CODEC_LEVEL = Value
- val CODEC_NAME = Value
- val CODEGEN_STAGE_ID = Value
- val COLUMN_DATA_TYPE_SOURCE = Value
- val COLUMN_DATA_TYPE_TARGET = Value
- val COLUMN_DEFAULT_VALUE = Value
- val COLUMN_NAME = Value
- val COMMAND = Value
- val COMMAND_OUTPUT = Value
- val COMPONENT = Value
- val CONFIG = Value
- val CONFIG2 = Value
- val CONFIG3 = Value
- val CONFIG4 = Value
- val CONFIG5 = Value
- val CONSUMER = Value
- val CONTAINER = Value
- val CONTAINER_ID = Value
- val CONTAINER_STATE = Value
- val COST = Value
- val COUNT = Value
- val CROSS_VALIDATION_METRIC = Value
- val CROSS_VALIDATION_METRICS = Value
- val CSV_HEADER_COLUMN_NAME = Value
- val CSV_HEADER_COLUMN_NAMES = Value
- val CSV_HEADER_LENGTH = Value
- val CSV_SCHEMA_FIELD_NAME = Value
- val CSV_SCHEMA_FIELD_NAMES = Value
- val CSV_SOURCE = Value
- val CURRENT_PATH = Value
- val DATA = Value
- val DATABASE_NAME = Value
- val DATAFRAME_CACHE_ENTRY = Value
- val DATAFRAME_ID = Value
- val DATA_FILE_NUM = Value
- val DATA_SOURCE = Value
- val DATA_SOURCES = Value
- val DATA_SOURCE_PROVIDER = Value
- val DEFAULT_ISOLATION_LEVEL = Value
- val DEFAULT_VALUE = Value
- val DELAY = Value
- val DELEGATE = Value
- val DELTA = Value
- val DESCRIPTION = Value
- val DESIRED_PARTITIONS_SIZE = Value
- val DIFF_DELTA = Value
- val DIVISIBLE_CLUSTER_INDICES_SIZE = Value
- val DRIVER_ID = Value
- val DROPPED_PARTITIONS = Value
- val DURATION = Value
- val EFFECTIVE_STORAGE_LEVEL = Value
- val ELAPSED_TIME = Value
- val ENCODING = Value
- val END_INDEX = Value
- val END_POINT = Value
- val ENGINE = Value
- val ERROR = Value
- val ESTIMATOR_PARAMETER_MAP = Value
- val EVENT_LOOP = Value
- val EVENT_QUEUE = Value
- val EXCEPTION = Value
- val EXECUTE_INFO = Value
- val EXECUTE_KEY = Value
- val EXECUTION_PLAN_LEAVES = Value
- val EXECUTOR_DESIRED_COUNT = Value
- val EXECUTOR_ENVS = Value
- val EXECUTOR_ENV_REGEX = Value
- val EXECUTOR_ID = Value
- val EXECUTOR_IDS = Value
- val EXECUTOR_LAUNCH_COMMANDS = Value
- val EXECUTOR_LAUNCH_COUNT = Value
- val EXECUTOR_RESOURCES = Value
- val EXECUTOR_STATE = Value
- val EXECUTOR_TARGET_COUNT = Value
- val EXIT_CODE = Value
- val EXPECTED_NUM_FILES = Value
- val EXPECTED_PARTITION_COLUMN = Value
- val EXPIRY_TIMESTAMP = Value
- val EXPR = Value
- val EXPR_TERMS = Value
- val EXTENDED_EXPLAIN_GENERATOR = Value
- val FAILURES = Value
- val FALLBACK_VERSION = Value
- val FEATURE_COLUMN = Value
- val FEATURE_DIMENSION = Value
- val FIELD_NAME = Value
- val FILE_ABSOLUTE_PATH = Value
- val FILE_FORMAT = Value
- val FILE_FORMAT2 = Value
- val FILE_NAME = Value
- val FILE_VERSION = Value
- val FINISH_TRIGGER_DURATION = Value
- val FROM_OFFSET = Value
- val FROM_TIME = Value
- val FUNCTION_NAME = Value
- val FUNCTION_PARAMETER = Value
- val GROUP_ID = Value
- val HADOOP_VERSION = Value
- val HASH_JOIN_KEYS = Value
- val HEARTBEAT_INTERVAL = Value
- val HISTORY_DIR = Value
- val HIVE_CLIENT_VERSION = Value
- val HIVE_METASTORE_VERSION = Value
- val HIVE_OPERATION_STATE = Value
- val HIVE_OPERATION_TYPE = Value
- val HOST = Value
- val HOST_PORT = Value
- val IDENTIFIER = Value
- val INCOMPATIBLE_TYPES = Value
- val INDEX = Value
- val INDEX_FILE_NUM = Value
- val INDEX_NAME = Value
- val INFERENCE_MODE = Value
- val INITIAL_CAPACITY = Value
- val INITIAL_HEARTBEAT_INTERVAL = Value
- val INIT_MODE = Value
- val INTERVAL = Value
- val ISOLATION_LEVEL = Value
- val JOB_ID = Value
- val JOIN_CONDITION = Value
- val JOIN_CONDITION_SUB_EXPR = Value
- val K8S_CONTEXT = Value
- val KAFKA_PULLS_COUNT = Value
- val KAFKA_RECORDS_PULLED_COUNT = Value
- val KEY = Value
- val KEYTAB = Value
- val LABEL_COLUMN = Value
- val LARGEST_CLUSTER_INDEX = Value
- val LAST_ACCESS_TIME = Value
- val LAST_VALID_TIME = Value
- val LATEST_BATCH_ID = Value
- val LATEST_COMMITTED_BATCH_ID = Value
- val LEARNING_RATE = Value
- val LEFT_EXPR = Value
- val LINE = Value
- val LINE_NUM = Value
- val LISTENER = Value
- val LOAD_FACTOR = Value
- val LOAD_TIME = Value
- val LOGICAL_PLAN_COLUMNS = Value
- val LOGICAL_PLAN_LEAVES = Value
- val LOG_TYPE = Value
- val LOWER_BOUND = Value
- val MALFORMATTED_STIRNG = Value
- val MASTER_URL = Value
- val MAX_ATTEMPTS = Value
- val MAX_CACHE_UNTIL_HIGHEST_CONSUMED_SIZE = Value
- val MAX_CACHE_UNTIL_LAST_PRODUCED_SIZE = Value
- val MAX_CAPACITY = Value
- val MAX_CATEGORIES = Value
- val MAX_EXECUTOR_FAILURES = Value
- val MAX_FILE_VERSION = Value
- val MAX_MEMORY_SIZE = Value
- val MAX_PARTITIONS_SIZE = Value
- val MAX_SIZE = Value
- val MAX_TABLE_PARTITION_METADATA_SIZE = Value
- val MEMORY_SIZE = Value
- val MERGE_DIR_NAME = Value
- val MESSAGE = Value
- val METHOD_NAME = Value
- val MIN_FREQUENT_PATTERN_COUNT = Value
- val MIN_POINT_PER_CLUSTER = Value
- val MIN_SIZE = Value
- val MODEL_WEIGHTS = Value
- val NAMESPACE = Value
- val NEW_FEATURE_COLUMN_NAME = Value
- val NEW_LABEL_COLUMN_NAME = Value
- val NEW_PATH = Value
- val NEW_VALUE = Value
- val NODES = Value
- val NODE_LOCATION = Value
- val NORM = Value
- val NUM_BIN = Value
- val NUM_CLASSES = Value
- val NUM_COLUMNS = Value
- val NUM_EXAMPLES = Value
- val NUM_FEATURES = Value
- val NUM_FILES = Value
- val NUM_FREQUENT_ITEMS = Value
- val NUM_ITERATIONS = Value
- val NUM_LOCAL_FREQUENT_PATTERN = Value
- val NUM_PARTITION = Value
- val NUM_POINT = Value
- val NUM_PREFIXES = Value
- val NUM_SEQUENCES = Value
- val OBJECT_ID = Value
- val OFFSET = Value
- val OFFSETS = Value
- val OLD_BLOCK_MANAGER_ID = Value
- val OLD_VALUE = Value
- val OPTIMIZED_PLAN_COLUMNS = Value
- val OPTIMIZER_CLASS_NAME = Value
- val OPTIONS = Value
- val OP_ID = Value
- val OP_TYPE = Value
- val OVERHEAD_MEMORY_SIZE = Value
- val PARSE_MODE = Value
- val PARTITIONED_FILE_READER = Value
- val PARTITIONS_SIZE = Value
- val PARTITION_ID = Value
- val PARTITION_SPECIFICATION = Value
- val PARTITION_SPECS = Value
- val PATH = Value
- val PATHS = Value
- val PIPELINE_STAGE_UID = Value
- val POD_COUNT = Value
- val POD_ID = Value
- val POD_NAME = Value
- val POD_NAMESPACE = Value
- val POD_PHASE = Value
- val POD_SHARED_SLOT_COUNT = Value
- val POD_STATE = Value
- val POD_TARGET_COUNT = Value
- val POLICY = Value
- val PORT = Value
- val PRINCIPAL = Value
- val PROCESSING_TIME = Value
- val PRODUCER_ID = Value
- val PROVIDER = Value
- val PVC_METADATA_NAME = Value
- val QUERY_CACHE_VALUE = Value
- val QUERY_HINT = Value
- val QUERY_ID = Value
- val QUERY_PLAN = Value
- val QUERY_PLAN_COMPARISON = Value
- val QUERY_PLAN_LENGTH_ACTUAL = Value
- val QUERY_PLAN_LENGTH_MAX = Value
- val QUERY_RUN_ID = Value
- val RANGE = Value
- val RDD_ID = Value
- val READ_LIMIT = Value
- val REASON = Value
- val REATTACHABLE = Value
- val RECEIVED_BLOCK_INFO = Value
- val RECEIVED_BLOCK_TRACKER_LOG_EVENT = Value
- val RECEIVER_ID = Value
- val RECEIVER_IDS = Value
- val RECORDS = Value
- val REDACTED_STATEMENT = Value
- val REDUCE_ID = Value
- val RELATION_NAME = Value
- val RELATIVE_TOLERANCE = Value
- val REMAINING_PARTITIONS = Value
- val REPORT_DETAILS = Value
- val RESOURCE = Value
- val RESOURCE_NAME = Value
- val RESOURCE_PROFILE_ID = Value
- val RESOURCE_PROFILE_IDS = Value
- val RETRY_COUNT = Value
- val RETRY_INTERVAL = Value
- val RIGHT_EXPR = Value
- val RMSE = Value
- val RPC_ENDPOINT_REF = Value
- val RULE_BATCH_NAME = Value
- val RULE_NAME = Value
- val RULE_NUMBER_OF_RUNS = Value
- val RUN_ID = Value
- val SCHEMA = Value
- val SCHEMA2 = Value
- val SERVICE_NAME = Value
- val SESSION_HOLD_INFO = Value
- val SESSION_ID = Value
- val SESSION_KEY = Value
- val SHARD_ID = Value
- val SHUFFLE_BLOCK_INFO = Value
- val SHUFFLE_ID = Value
- val SHUFFLE_MERGE_ID = Value
- val SHUFFLE_SERVICE_NAME = Value
- val SIZE = Value
- val SLEEP_TIME = Value
- val SLIDE_DURATION = Value
- val SMALLEST_CLUSTER_INDEX = Value
- val SPARK_DATA_STREAM = Value
- val SPARK_PLAN_ID = Value
- val SQL_TEXT = Value
- val SRC_PATH = Value
- val STAGE_ATTEMPT = Value
- val STAGE_ID = Value
- val START_INDEX = Value
- val STATEMENT_ID = Value
- val STATE_STORE_PROVIDER = Value
- val STATUS = Value
- val STDERR = Value
- val STORAGE_LEVEL = Value
- val STORAGE_LEVEL_DESERIALIZED = Value
- val STORAGE_LEVEL_REPLICATION = Value
- val STORE_ID = Value
- val STREAM_ID = Value
- val STREAM_NAME = Value
- val SUBMISSION_ID = Value
- val SUBSAMPLING_RATE = Value
- val SUB_QUERY = Value
- val TABLE_NAME = Value
- val TABLE_TYPES = Value
- val TARGET_PATH = Value
- val TASK_ATTEMPT_ID = Value
- val TASK_ID = Value
- val TASK_NAME = Value
- val TASK_SET_NAME = Value
- val TASK_STATE = Value
- val TEMP_FILE = Value
- val TEMP_PATH = Value
- val TEST_SIZE = Value
- val THREAD = Value
- val THREAD_NAME = Value
- val TID = Value
- val TIME = Value
- val TIMEOUT = Value
- val TIMER = Value
- val TIMER_LABEL = Value
- val TIME_UNITS = Value
- val TIP = Value
- val TOKEN_REGEX = Value
- val TOPIC = Value
- val TOPIC_PARTITION = Value
- val TOPIC_PARTITIONS = Value
- val TOPIC_PARTITION_OFFSET = Value
- val TOPIC_PARTITION_OFFSET_RANGE = Value
- val TOTAL = Value
- val TOTAL_EFFECTIVE_TIME = Value
- val TOTAL_RECORDS_READ = Value
- val TOTAL_TIME = Value
- val TOTAL_TIME_READ = Value
- val TO_TIME = Value
- val TRAINING_SIZE = Value
- val TRAIN_VALIDATION_SPLIT_METRIC = Value
- val TRAIN_VALIDATION_SPLIT_METRICS = Value
- val TRAIN_WORD_COUNT = Value
- val TREE_NODE = Value
- val TRIGGER_INTERVAL = Value
- val UI_FILTER = Value
- val UI_FILTER_PARAMS = Value
- val UI_PROXY_BASE = Value
- val UNSUPPORTED_EXPR = Value
- val UNSUPPORTED_HINT_REASON = Value
- val UNTIL_OFFSET = Value
- val UPPER_BOUND = Value
- val URI = Value
- val USER_ID = Value
- val USER_NAME = Value
- val VALUE = Value
- val VIRTUAL_CORES = Value
- val VOCAB_SIZE = Value
- val WAIT_RESULT_TIME = Value
- val WAIT_SEND_TIME = Value
- val WAIT_TIME = Value
- val WATERMARK_CONSTRAINT = Value
- val WEIGHTED_NUM = Value
- val WORKER_URL = Value
- val WRITE_AHEAD_LOG_INFO = Value
- val WRITE_JOB_UUID = Value
- val XSD_PATH = Value
-
- type LogKey = Value
+object LogKeys {
+ case object ACCUMULATOR_ID extends LogKey
+ case object ACTUAL_NUM_FILES extends LogKey
+ case object ACTUAL_PARTITION_COLUMN extends LogKey
+ case object ALPHA extends LogKey
+ case object ANALYSIS_ERROR extends LogKey
+ case object APP_ATTEMPT_ID extends LogKey
+ case object APP_DESC extends LogKey
+ case object APP_ID extends LogKey
+ case object APP_NAME extends LogKey
+ case object APP_STATE extends LogKey
+ case object ARGS extends LogKey
+ case object BACKUP_FILE extends LogKey
+ case object BATCH_ID extends LogKey
+ case object BATCH_WRITE extends LogKey
+ case object BLOCK_ID extends LogKey
+ case object BLOCK_MANAGER_ID extends LogKey
+ case object BROADCAST_ID extends LogKey
+ case object BUCKET extends LogKey
+ case object BYTECODE_SIZE extends LogKey
+ case object CACHED_TABLE_PARTITION_METADATA_SIZE extends LogKey
+ case object CACHE_AUTO_REMOVED_SIZE extends LogKey
+ case object CACHE_UNTIL_HIGHEST_CONSUMED_SIZE extends LogKey
+ case object CACHE_UNTIL_LAST_PRODUCED_SIZE extends LogKey
+ case object CALL_SITE_LONG_FORM extends LogKey
+ case object CATALOG_NAME extends LogKey
+ case object CATEGORICAL_FEATURES extends LogKey
+ case object CHECKPOINT_FILE extends LogKey
+ case object CHECKPOINT_TIME extends LogKey
+ case object CHECKSUM_FILE_NUM extends LogKey
+ case object CLASS_LOADER extends LogKey
+ case object CLASS_NAME extends LogKey
+ case object CLUSTER_CENTROIDS extends LogKey
+ case object CLUSTER_ID extends LogKey
+ case object CLUSTER_LABEL extends LogKey
+ case object CLUSTER_LEVEL extends LogKey
+ case object CLUSTER_WEIGHT extends LogKey
+ case object CODEC_LEVEL extends LogKey
+ case object CODEC_NAME extends LogKey
+ case object CODEGEN_STAGE_ID extends LogKey
+ case object COLUMN_DATA_TYPE_SOURCE extends LogKey
+ case object COLUMN_DATA_TYPE_TARGET extends LogKey
+ case object COLUMN_DEFAULT_VALUE extends LogKey
+ case object COLUMN_NAME extends LogKey
+ case object COMMAND extends LogKey
+ case object COMMAND_OUTPUT extends LogKey
+ case object COMPONENT extends LogKey
+ case object CONFIG extends LogKey
+ case object CONFIG2 extends LogKey
+ case object CONFIG3 extends LogKey
+ case object CONFIG4 extends LogKey
+ case object CONFIG5 extends LogKey
+ case object CONSUMER extends LogKey
+ case object CONTAINER extends LogKey
+ case object CONTAINER_ID extends LogKey
+ case object CONTAINER_STATE extends LogKey
+ case object COST extends LogKey
+ case object COUNT extends LogKey
+ case object CROSS_VALIDATION_METRIC extends LogKey
+ case object CROSS_VALIDATION_METRICS extends LogKey
+ case object CSV_HEADER_COLUMN_NAME extends LogKey
+ case object CSV_HEADER_COLUMN_NAMES extends LogKey
+ case object CSV_HEADER_LENGTH extends LogKey
+ case object CSV_SCHEMA_FIELD_NAME extends LogKey
+ case object CSV_SCHEMA_FIELD_NAMES extends LogKey
+ case object CSV_SOURCE extends LogKey
+ case object CURRENT_PATH extends LogKey
+ case object DATA extends LogKey
+ case object DATABASE_NAME extends LogKey
+ case object DATAFRAME_CACHE_ENTRY extends LogKey
+ case object DATAFRAME_ID extends LogKey
+ case object DATA_FILE_NUM extends LogKey
+ case object DATA_SOURCE extends LogKey
+ case object DATA_SOURCES extends LogKey
+ case object DATA_SOURCE_PROVIDER extends LogKey
+ case object DEFAULT_ISOLATION_LEVEL extends LogKey
+ case object DEFAULT_VALUE extends LogKey
+ case object DELAY extends LogKey
+ case object DELEGATE extends LogKey
+ case object DELTA extends LogKey
+ case object DESCRIPTION extends LogKey
+ case object DESIRED_PARTITIONS_SIZE extends LogKey
+ case object DIFF_DELTA extends LogKey
+ case object DIVISIBLE_CLUSTER_INDICES_SIZE extends LogKey
+ case object DRIVER_ID extends LogKey
+ case object DROPPED_PARTITIONS extends LogKey
+ case object DURATION extends LogKey
+ case object EFFECTIVE_STORAGE_LEVEL extends LogKey
+ case object ELAPSED_TIME extends LogKey
+ case object ENCODING extends LogKey
+ case object END_INDEX extends LogKey
+ case object END_POINT extends LogKey
+ case object ENGINE extends LogKey
+ case object ERROR extends LogKey
+ case object ESTIMATOR_PARAMETER_MAP extends LogKey
+ case object EVENT_LOOP extends LogKey
+ case object EVENT_QUEUE extends LogKey
+ case object EXCEPTION extends LogKey
+ case object EXECUTE_INFO extends LogKey
+ case object EXECUTE_KEY extends LogKey
+ case object EXECUTION_PLAN_LEAVES extends LogKey
+ case object EXECUTOR_DESIRED_COUNT extends LogKey
+ case object EXECUTOR_ENVS extends LogKey
+ case object EXECUTOR_ENV_REGEX extends LogKey
+ case object EXECUTOR_ID extends LogKey
+ case object EXECUTOR_IDS extends LogKey
+ case object EXECUTOR_LAUNCH_COMMANDS extends LogKey
+ case object EXECUTOR_LAUNCH_COUNT extends LogKey
+ case object EXECUTOR_RESOURCES extends LogKey
+ case object EXECUTOR_STATE extends LogKey
+ case object EXECUTOR_TARGET_COUNT extends LogKey
+ case object EXIT_CODE extends LogKey
+ case object EXPECTED_NUM_FILES extends LogKey
+ case object EXPECTED_PARTITION_COLUMN extends LogKey
+ case object EXPIRY_TIMESTAMP extends LogKey
+ case object EXPR extends LogKey
+ case object EXPR_TERMS extends LogKey
+ case object EXTENDED_EXPLAIN_GENERATOR extends LogKey
+ case object FAILURES extends LogKey
+ case object FALLBACK_VERSION extends LogKey
+ case object FEATURE_COLUMN extends LogKey
+ case object FEATURE_DIMENSION extends LogKey
+ case object FIELD_NAME extends LogKey
+ case object FILE_ABSOLUTE_PATH extends LogKey
+ case object FILE_FORMAT extends LogKey
+ case object FILE_FORMAT2 extends LogKey
+ case object FILE_NAME extends LogKey
+ case object FILE_VERSION extends LogKey
+ case object FINISH_TRIGGER_DURATION extends LogKey
+ case object FROM_OFFSET extends LogKey
+ case object FROM_TIME extends LogKey
+ case object FUNCTION_NAME extends LogKey
+ case object FUNCTION_PARAMETER extends LogKey
+ case object GROUP_ID extends LogKey
+ case object HADOOP_VERSION extends LogKey
+ case object HASH_JOIN_KEYS extends LogKey
+ case object HEARTBEAT_INTERVAL extends LogKey
+ case object HISTORY_DIR extends LogKey
+ case object HIVE_CLIENT_VERSION extends LogKey
+ case object HIVE_METASTORE_VERSION extends LogKey
+ case object HIVE_OPERATION_STATE extends LogKey
+ case object HIVE_OPERATION_TYPE extends LogKey
+ case object HOST extends LogKey
+ case object HOST_PORT extends LogKey
+ case object IDENTIFIER extends LogKey
+ case object INCOMPATIBLE_TYPES extends LogKey
+ case object INDEX extends LogKey
+ case object INDEX_FILE_NUM extends LogKey
+ case object INDEX_NAME extends LogKey
+ case object INFERENCE_MODE extends LogKey
+ case object INITIAL_CAPACITY extends LogKey
+ case object INITIAL_HEARTBEAT_INTERVAL extends LogKey
+ case object INIT_MODE extends LogKey
+ case object INTERVAL extends LogKey
+ case object ISOLATION_LEVEL extends LogKey
+ case object JOB_ID extends LogKey
+ case object JOIN_CONDITION extends LogKey
+ case object JOIN_CONDITION_SUB_EXPR extends LogKey
+ case object K8S_CONTEXT extends LogKey
+ case object KAFKA_PULLS_COUNT extends LogKey
+ case object KAFKA_RECORDS_PULLED_COUNT extends LogKey
+ case object KEY extends LogKey
+ case object KEYTAB extends LogKey
+ case object LABEL_COLUMN extends LogKey
+ case object LARGEST_CLUSTER_INDEX extends LogKey
+ case object LAST_ACCESS_TIME extends LogKey
+ case object LAST_VALID_TIME extends LogKey
+ case object LATEST_BATCH_ID extends LogKey
+ case object LATEST_COMMITTED_BATCH_ID extends LogKey
+ case object LEARNING_RATE extends LogKey
+ case object LEFT_EXPR extends LogKey
+ case object LINE extends LogKey
+ case object LINE_NUM extends LogKey
+ case object LISTENER extends LogKey
+ case object LOAD_FACTOR extends LogKey
+ case object LOAD_TIME extends LogKey
+ case object LOGICAL_PLAN_COLUMNS extends LogKey
+ case object LOGICAL_PLAN_LEAVES extends LogKey
+ case object LOG_TYPE extends LogKey
+ case object LOWER_BOUND extends LogKey
+ case object MALFORMATTED_STIRNG extends LogKey
+ case object MASTER_URL extends LogKey
+ case object MAX_ATTEMPTS extends LogKey
+ case object MAX_CACHE_UNTIL_HIGHEST_CONSUMED_SIZE extends LogKey
+ case object MAX_CACHE_UNTIL_LAST_PRODUCED_SIZE extends LogKey
+ case object MAX_CAPACITY extends LogKey
+ case object MAX_CATEGORIES extends LogKey
+ case object MAX_EXECUTOR_FAILURES extends LogKey
+ case object MAX_FILE_VERSION extends LogKey
+ case object MAX_MEMORY_SIZE extends LogKey
+ case object MAX_PARTITIONS_SIZE extends LogKey
+ case object MAX_SIZE extends LogKey
+ case object MAX_TABLE_PARTITION_METADATA_SIZE extends LogKey
+ case object MEMORY_SIZE extends LogKey
+ case object MERGE_DIR_NAME extends LogKey
+ case object MESSAGE extends LogKey
+ case object METHOD_NAME extends LogKey
+ case object MIN_FREQUENT_PATTERN_COUNT extends LogKey
+ case object MIN_POINT_PER_CLUSTER extends LogKey
+ case object MIN_SIZE extends LogKey
+ case object MODEL_WEIGHTS extends LogKey
+ case object NAMESPACE extends LogKey
+ case object NEW_FEATURE_COLUMN_NAME extends LogKey
+ case object NEW_LABEL_COLUMN_NAME extends LogKey
+ case object NEW_PATH extends LogKey
+ case object NEW_VALUE extends LogKey
+ case object NODES extends LogKey
+ case object NODE_LOCATION extends LogKey
+ case object NORM extends LogKey
+ case object NUM_BIN extends LogKey
+ case object NUM_CLASSES extends LogKey
+ case object NUM_COLUMNS extends LogKey
+ case object NUM_EXAMPLES extends LogKey
+ case object NUM_FEATURES extends LogKey
+ case object NUM_FILES extends LogKey
+ case object NUM_FREQUENT_ITEMS extends LogKey
+ case object NUM_ITERATIONS extends LogKey
+ case object NUM_LOCAL_FREQUENT_PATTERN extends LogKey
+ case object NUM_PARTITION extends LogKey
+ case object NUM_POINT extends LogKey
+ case object NUM_PREFIXES extends LogKey
+ case object NUM_SEQUENCES extends LogKey
+ case object OBJECT_ID extends LogKey
+ case object OFFSET extends LogKey
+ case object OFFSETS extends LogKey
+ case object OLD_BLOCK_MANAGER_ID extends LogKey
+ case object OLD_VALUE extends LogKey
+ case object OPTIMIZED_PLAN_COLUMNS extends LogKey
+ case object OPTIMIZER_CLASS_NAME extends LogKey
+ case object OPTIONS extends LogKey
+ case object OP_ID extends LogKey
+ case object OP_TYPE extends LogKey
+ case object OVERHEAD_MEMORY_SIZE extends LogKey
+ case object PARSE_MODE extends LogKey
+ case object PARTITIONED_FILE_READER extends LogKey
+ case object PARTITIONS_SIZE extends LogKey
+ case object PARTITION_ID extends LogKey
+ case object PARTITION_SPECIFICATION extends LogKey
+ case object PARTITION_SPECS extends LogKey
+ case object PATH extends LogKey
+ case object PATHS extends LogKey
+ case object PIPELINE_STAGE_UID extends LogKey
+ case object POD_COUNT extends LogKey
+ case object POD_ID extends LogKey
+ case object POD_NAME extends LogKey
+ case object POD_NAMESPACE extends LogKey
+ case object POD_PHASE extends LogKey
+ case object POD_SHARED_SLOT_COUNT extends LogKey
+ case object POD_STATE extends LogKey
+ case object POD_TARGET_COUNT extends LogKey
+ case object POLICY extends LogKey
+ case object PORT extends LogKey
+ case object PRINCIPAL extends LogKey
+ case object PROCESSING_TIME extends LogKey
+ case object PRODUCER_ID extends LogKey
+ case object PROVIDER extends LogKey
+ case object PVC_METADATA_NAME extends LogKey
+ case object QUERY_CACHE_VALUE extends LogKey
+ case object QUERY_HINT extends LogKey
+ case object QUERY_ID extends LogKey
+ case object QUERY_PLAN extends LogKey
+ case object QUERY_PLAN_COMPARISON extends LogKey
+ case object QUERY_PLAN_LENGTH_ACTUAL extends LogKey
+ case object QUERY_PLAN_LENGTH_MAX extends LogKey
+ case object QUERY_RUN_ID extends LogKey
+ case object RANGE extends LogKey
+ case object RDD_ID extends LogKey
+ case object READ_LIMIT extends LogKey
+ case object REASON extends LogKey
+ case object REATTACHABLE extends LogKey
+ case object RECEIVED_BLOCK_INFO extends LogKey
+ case object RECEIVED_BLOCK_TRACKER_LOG_EVENT extends LogKey
+ case object RECEIVER_ID extends LogKey
+ case object RECEIVER_IDS extends LogKey
+ case object RECORDS extends LogKey
+ case object REDACTED_STATEMENT extends LogKey
+ case object REDUCE_ID extends LogKey
+ case object RELATION_NAME extends LogKey
+ case object RELATIVE_TOLERANCE extends LogKey
+ case object REMAINING_PARTITIONS extends LogKey
+ case object REPORT_DETAILS extends LogKey
+ case object RESOURCE extends LogKey
+ case object RESOURCE_NAME extends LogKey
+ case object RESOURCE_PROFILE_ID extends LogKey
+ case object RESOURCE_PROFILE_IDS extends LogKey
+ case object RETRY_COUNT extends LogKey
+ case object RETRY_INTERVAL extends LogKey
+ case object RIGHT_EXPR extends LogKey
+ case object RMSE extends LogKey
+ case object RPC_ENDPOINT_REF extends LogKey
+ case object RULE_BATCH_NAME extends LogKey
+ case object RULE_NAME extends LogKey
+ case object RULE_NUMBER_OF_RUNS extends LogKey
+ case object RUN_ID extends LogKey
+ case object SCHEMA extends LogKey
+ case object SCHEMA2 extends LogKey
+ case object SERVICE_NAME extends LogKey
+ case object SESSION_HOLD_INFO extends LogKey
+ case object SESSION_ID extends LogKey
+ case object SESSION_KEY extends LogKey
+ case object SHARD_ID extends LogKey
+ case object SHUFFLE_BLOCK_INFO extends LogKey
+ case object SHUFFLE_ID extends LogKey
+ case object SHUFFLE_MERGE_ID extends LogKey
+ case object SHUFFLE_SERVICE_NAME extends LogKey
+ case object SIZE extends LogKey
+ case object SLEEP_TIME extends LogKey
+ case object SLIDE_DURATION extends LogKey
+ case object SMALLEST_CLUSTER_INDEX extends LogKey
+ case object SPARK_DATA_STREAM extends LogKey
+ case object SPARK_PLAN_ID extends LogKey
+ case object SQL_TEXT extends LogKey
+ case object SRC_PATH extends LogKey
+ case object STAGE_ATTEMPT extends LogKey
+ case object STAGE_ID extends LogKey
+ case object START_INDEX extends LogKey
+ case object STATEMENT_ID extends LogKey
+ case object STATE_STORE_PROVIDER extends LogKey
+ case object STATUS extends LogKey
+ case object STDERR extends LogKey
+ case object STORAGE_LEVEL extends LogKey
+ case object STORAGE_LEVEL_DESERIALIZED extends LogKey
+ case object STORAGE_LEVEL_REPLICATION extends LogKey
+ case object STORE_ID extends LogKey
+ case object STREAM_ID extends LogKey
+ case object STREAM_NAME extends LogKey
+ case object SUBMISSION_ID extends LogKey
+ case object SUBSAMPLING_RATE extends LogKey
+ case object SUB_QUERY extends LogKey
+ case object TABLE_NAME extends LogKey
+ case object TABLE_TYPES extends LogKey
+ case object TARGET_PATH extends LogKey
+ case object TASK_ATTEMPT_ID extends LogKey
+ case object TASK_ID extends LogKey
+ case object TASK_NAME extends LogKey
+ case object TASK_SET_NAME extends LogKey
+ case object TASK_STATE extends LogKey
+ case object TEMP_FILE extends LogKey
+ case object TEMP_PATH extends LogKey
+ case object TEST_SIZE extends LogKey
+ case object THREAD extends LogKey
+ case object THREAD_NAME extends LogKey
+ case object TID extends LogKey
+ case object TIME extends LogKey
+ case object TIMEOUT extends LogKey
+ case object TIMER extends LogKey
+ case object TIMER_LABEL extends LogKey
Review Comment:
prefix sounds good
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47963][CORE] Make the external Spark ecosystem can use structured logging mechanisms [spark]
Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on code in PR #46193:
URL: https://github.com/apache/spark/pull/46193#discussion_r1578930058
##########
common/utils/src/test/scala/org/apache/spark/util/StructuredLoggingSuite.scala:
##########
@@ -144,6 +146,22 @@ trait LoggingSuiteBase
}
}
+ // An enumeration value that is not defined in `LogKey`, you can define it anywhere
+ private val externalLogKey = LogKey.VALUE
+ private val externalLog = log"${MDC(externalLogKey, "External log message.")}"
+ test("Logging with external LogKey") {
Review Comment:
Yeah
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47963][CORE] Make the external Spark ecosystem can use structured logging mechanisms [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #46193:
URL: https://github.com/apache/spark/pull/46193#discussion_r1577602188
##########
common/utils/src/test/scala/org/apache/spark/util/StructuredLoggingSuite.scala:
##########
@@ -144,6 +146,22 @@ trait LoggingSuiteBase
}
}
+ // An enumeration value that is not defined in `LogKey`, you can define it anywhere
+ private val externalLogKey = LogKey.VALUE
+ private val externalLog = log"${MDC(externalLogKey, "External log message.")}"
+ test("Logging with external LogKey") {
Review Comment:
Perhaps we also need to consider namespace isolation. Currently, we can define keys with the same name in multiple places, but we cannot distinguish from the `MessageWithContext` whether they are defined internally in Spark or externally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47963][CORE][TESTS] Add an external LogKey usage case in UT [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #46193:
URL: https://github.com/apache/spark/pull/46193#discussion_r1577206238
##########
common/utils/src/test/scala/org/apache/spark/util/StructuredLoggingSuite.scala:
##########
@@ -144,6 +146,22 @@ trait LoggingSuiteBase
}
}
+ // An enumeration value that is not defined in `LogKey`, you can define it anywhere
+ private val externalLogKey = LogKey.VALUE
+ private val externalLog = log"${MDC(externalLogKey, "External log message.")}"
+ test("Logging with external LogKey") {
Review Comment:
friendly ping @gengliangwang, what's your take on this? Is `StructuredLogging` a feature that users can utilize? If so, how can we make it easier for users to extend the definition of `LogKey`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47963][CORE] Make the external Spark ecosystem can use structured logging mechanisms [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #46193:
URL: https://github.com/apache/spark/pull/46193#discussion_r1578788755
##########
common/utils/src/main/scala/org/apache/spark/internal/LogKey.scala:
##########
@@ -16,399 +16,399 @@
*/
package org.apache.spark.internal
+trait LogKey
+
/**
* Various keys used for mapped diagnostic contexts(MDC) in logging.
* All structured logging keys should be defined here for standardization.
*/
-object LogKey extends Enumeration {
- val ACCUMULATOR_ID = Value
- val ACTUAL_NUM_FILES = Value
- val ACTUAL_PARTITION_COLUMN = Value
- val ALPHA = Value
- val ANALYSIS_ERROR = Value
- val APP_ATTEMPT_ID = Value
- val APP_DESC = Value
- val APP_ID = Value
- val APP_NAME = Value
- val APP_STATE = Value
- val ARGS = Value
- val BACKUP_FILE = Value
- val BATCH_ID = Value
- val BATCH_WRITE = Value
- val BLOCK_ID = Value
- val BLOCK_MANAGER_ID = Value
- val BROADCAST_ID = Value
- val BUCKET = Value
- val BYTECODE_SIZE = Value
- val CACHED_TABLE_PARTITION_METADATA_SIZE = Value
- val CACHE_AUTO_REMOVED_SIZE = Value
- val CACHE_UNTIL_HIGHEST_CONSUMED_SIZE = Value
- val CACHE_UNTIL_LAST_PRODUCED_SIZE = Value
- val CALL_SITE_LONG_FORM = Value
- val CATALOG_NAME = Value
- val CATEGORICAL_FEATURES = Value
- val CHECKPOINT_FILE = Value
- val CHECKPOINT_TIME = Value
- val CHECKSUM_FILE_NUM = Value
- val CLASS_LOADER = Value
- val CLASS_NAME = Value
- val CLUSTER_CENTROIDS = Value
- val CLUSTER_ID = Value
- val CLUSTER_LABEL = Value
- val CLUSTER_LEVEL = Value
- val CLUSTER_WEIGHT = Value
- val CODEC_LEVEL = Value
- val CODEC_NAME = Value
- val CODEGEN_STAGE_ID = Value
- val COLUMN_DATA_TYPE_SOURCE = Value
- val COLUMN_DATA_TYPE_TARGET = Value
- val COLUMN_DEFAULT_VALUE = Value
- val COLUMN_NAME = Value
- val COMMAND = Value
- val COMMAND_OUTPUT = Value
- val COMPONENT = Value
- val CONFIG = Value
- val CONFIG2 = Value
- val CONFIG3 = Value
- val CONFIG4 = Value
- val CONFIG5 = Value
- val CONSUMER = Value
- val CONTAINER = Value
- val CONTAINER_ID = Value
- val CONTAINER_STATE = Value
- val COST = Value
- val COUNT = Value
- val CROSS_VALIDATION_METRIC = Value
- val CROSS_VALIDATION_METRICS = Value
- val CSV_HEADER_COLUMN_NAME = Value
- val CSV_HEADER_COLUMN_NAMES = Value
- val CSV_HEADER_LENGTH = Value
- val CSV_SCHEMA_FIELD_NAME = Value
- val CSV_SCHEMA_FIELD_NAMES = Value
- val CSV_SOURCE = Value
- val CURRENT_PATH = Value
- val DATA = Value
- val DATABASE_NAME = Value
- val DATAFRAME_CACHE_ENTRY = Value
- val DATAFRAME_ID = Value
- val DATA_FILE_NUM = Value
- val DATA_SOURCE = Value
- val DATA_SOURCES = Value
- val DATA_SOURCE_PROVIDER = Value
- val DEFAULT_ISOLATION_LEVEL = Value
- val DEFAULT_VALUE = Value
- val DELAY = Value
- val DELEGATE = Value
- val DELTA = Value
- val DESCRIPTION = Value
- val DESIRED_PARTITIONS_SIZE = Value
- val DIFF_DELTA = Value
- val DIVISIBLE_CLUSTER_INDICES_SIZE = Value
- val DRIVER_ID = Value
- val DROPPED_PARTITIONS = Value
- val DURATION = Value
- val EFFECTIVE_STORAGE_LEVEL = Value
- val ELAPSED_TIME = Value
- val ENCODING = Value
- val END_INDEX = Value
- val END_POINT = Value
- val ENGINE = Value
- val ERROR = Value
- val ESTIMATOR_PARAMETER_MAP = Value
- val EVENT_LOOP = Value
- val EVENT_QUEUE = Value
- val EXCEPTION = Value
- val EXECUTE_INFO = Value
- val EXECUTE_KEY = Value
- val EXECUTION_PLAN_LEAVES = Value
- val EXECUTOR_DESIRED_COUNT = Value
- val EXECUTOR_ENVS = Value
- val EXECUTOR_ENV_REGEX = Value
- val EXECUTOR_ID = Value
- val EXECUTOR_IDS = Value
- val EXECUTOR_LAUNCH_COMMANDS = Value
- val EXECUTOR_LAUNCH_COUNT = Value
- val EXECUTOR_RESOURCES = Value
- val EXECUTOR_STATE = Value
- val EXECUTOR_TARGET_COUNT = Value
- val EXIT_CODE = Value
- val EXPECTED_NUM_FILES = Value
- val EXPECTED_PARTITION_COLUMN = Value
- val EXPIRY_TIMESTAMP = Value
- val EXPR = Value
- val EXPR_TERMS = Value
- val EXTENDED_EXPLAIN_GENERATOR = Value
- val FAILURES = Value
- val FALLBACK_VERSION = Value
- val FEATURE_COLUMN = Value
- val FEATURE_DIMENSION = Value
- val FIELD_NAME = Value
- val FILE_ABSOLUTE_PATH = Value
- val FILE_FORMAT = Value
- val FILE_FORMAT2 = Value
- val FILE_NAME = Value
- val FILE_VERSION = Value
- val FINISH_TRIGGER_DURATION = Value
- val FROM_OFFSET = Value
- val FROM_TIME = Value
- val FUNCTION_NAME = Value
- val FUNCTION_PARAMETER = Value
- val GROUP_ID = Value
- val HADOOP_VERSION = Value
- val HASH_JOIN_KEYS = Value
- val HEARTBEAT_INTERVAL = Value
- val HISTORY_DIR = Value
- val HIVE_CLIENT_VERSION = Value
- val HIVE_METASTORE_VERSION = Value
- val HIVE_OPERATION_STATE = Value
- val HIVE_OPERATION_TYPE = Value
- val HOST = Value
- val HOST_PORT = Value
- val IDENTIFIER = Value
- val INCOMPATIBLE_TYPES = Value
- val INDEX = Value
- val INDEX_FILE_NUM = Value
- val INDEX_NAME = Value
- val INFERENCE_MODE = Value
- val INITIAL_CAPACITY = Value
- val INITIAL_HEARTBEAT_INTERVAL = Value
- val INIT_MODE = Value
- val INTERVAL = Value
- val ISOLATION_LEVEL = Value
- val JOB_ID = Value
- val JOIN_CONDITION = Value
- val JOIN_CONDITION_SUB_EXPR = Value
- val K8S_CONTEXT = Value
- val KAFKA_PULLS_COUNT = Value
- val KAFKA_RECORDS_PULLED_COUNT = Value
- val KEY = Value
- val KEYTAB = Value
- val LABEL_COLUMN = Value
- val LARGEST_CLUSTER_INDEX = Value
- val LAST_ACCESS_TIME = Value
- val LAST_VALID_TIME = Value
- val LATEST_BATCH_ID = Value
- val LATEST_COMMITTED_BATCH_ID = Value
- val LEARNING_RATE = Value
- val LEFT_EXPR = Value
- val LINE = Value
- val LINE_NUM = Value
- val LISTENER = Value
- val LOAD_FACTOR = Value
- val LOAD_TIME = Value
- val LOGICAL_PLAN_COLUMNS = Value
- val LOGICAL_PLAN_LEAVES = Value
- val LOG_TYPE = Value
- val LOWER_BOUND = Value
- val MALFORMATTED_STIRNG = Value
- val MASTER_URL = Value
- val MAX_ATTEMPTS = Value
- val MAX_CACHE_UNTIL_HIGHEST_CONSUMED_SIZE = Value
- val MAX_CACHE_UNTIL_LAST_PRODUCED_SIZE = Value
- val MAX_CAPACITY = Value
- val MAX_CATEGORIES = Value
- val MAX_EXECUTOR_FAILURES = Value
- val MAX_FILE_VERSION = Value
- val MAX_MEMORY_SIZE = Value
- val MAX_PARTITIONS_SIZE = Value
- val MAX_SIZE = Value
- val MAX_TABLE_PARTITION_METADATA_SIZE = Value
- val MEMORY_SIZE = Value
- val MERGE_DIR_NAME = Value
- val MESSAGE = Value
- val METHOD_NAME = Value
- val MIN_FREQUENT_PATTERN_COUNT = Value
- val MIN_POINT_PER_CLUSTER = Value
- val MIN_SIZE = Value
- val MODEL_WEIGHTS = Value
- val NAMESPACE = Value
- val NEW_FEATURE_COLUMN_NAME = Value
- val NEW_LABEL_COLUMN_NAME = Value
- val NEW_PATH = Value
- val NEW_VALUE = Value
- val NODES = Value
- val NODE_LOCATION = Value
- val NORM = Value
- val NUM_BIN = Value
- val NUM_CLASSES = Value
- val NUM_COLUMNS = Value
- val NUM_EXAMPLES = Value
- val NUM_FEATURES = Value
- val NUM_FILES = Value
- val NUM_FREQUENT_ITEMS = Value
- val NUM_ITERATIONS = Value
- val NUM_LOCAL_FREQUENT_PATTERN = Value
- val NUM_PARTITION = Value
- val NUM_POINT = Value
- val NUM_PREFIXES = Value
- val NUM_SEQUENCES = Value
- val OBJECT_ID = Value
- val OFFSET = Value
- val OFFSETS = Value
- val OLD_BLOCK_MANAGER_ID = Value
- val OLD_VALUE = Value
- val OPTIMIZED_PLAN_COLUMNS = Value
- val OPTIMIZER_CLASS_NAME = Value
- val OPTIONS = Value
- val OP_ID = Value
- val OP_TYPE = Value
- val OVERHEAD_MEMORY_SIZE = Value
- val PARSE_MODE = Value
- val PARTITIONED_FILE_READER = Value
- val PARTITIONS_SIZE = Value
- val PARTITION_ID = Value
- val PARTITION_SPECIFICATION = Value
- val PARTITION_SPECS = Value
- val PATH = Value
- val PATHS = Value
- val PIPELINE_STAGE_UID = Value
- val POD_COUNT = Value
- val POD_ID = Value
- val POD_NAME = Value
- val POD_NAMESPACE = Value
- val POD_PHASE = Value
- val POD_SHARED_SLOT_COUNT = Value
- val POD_STATE = Value
- val POD_TARGET_COUNT = Value
- val POLICY = Value
- val PORT = Value
- val PRINCIPAL = Value
- val PROCESSING_TIME = Value
- val PRODUCER_ID = Value
- val PROVIDER = Value
- val PVC_METADATA_NAME = Value
- val QUERY_CACHE_VALUE = Value
- val QUERY_HINT = Value
- val QUERY_ID = Value
- val QUERY_PLAN = Value
- val QUERY_PLAN_COMPARISON = Value
- val QUERY_PLAN_LENGTH_ACTUAL = Value
- val QUERY_PLAN_LENGTH_MAX = Value
- val QUERY_RUN_ID = Value
- val RANGE = Value
- val RDD_ID = Value
- val READ_LIMIT = Value
- val REASON = Value
- val REATTACHABLE = Value
- val RECEIVED_BLOCK_INFO = Value
- val RECEIVED_BLOCK_TRACKER_LOG_EVENT = Value
- val RECEIVER_ID = Value
- val RECEIVER_IDS = Value
- val RECORDS = Value
- val REDACTED_STATEMENT = Value
- val REDUCE_ID = Value
- val RELATION_NAME = Value
- val RELATIVE_TOLERANCE = Value
- val REMAINING_PARTITIONS = Value
- val REPORT_DETAILS = Value
- val RESOURCE = Value
- val RESOURCE_NAME = Value
- val RESOURCE_PROFILE_ID = Value
- val RESOURCE_PROFILE_IDS = Value
- val RETRY_COUNT = Value
- val RETRY_INTERVAL = Value
- val RIGHT_EXPR = Value
- val RMSE = Value
- val RPC_ENDPOINT_REF = Value
- val RULE_BATCH_NAME = Value
- val RULE_NAME = Value
- val RULE_NUMBER_OF_RUNS = Value
- val RUN_ID = Value
- val SCHEMA = Value
- val SCHEMA2 = Value
- val SERVICE_NAME = Value
- val SESSION_HOLD_INFO = Value
- val SESSION_ID = Value
- val SESSION_KEY = Value
- val SHARD_ID = Value
- val SHUFFLE_BLOCK_INFO = Value
- val SHUFFLE_ID = Value
- val SHUFFLE_MERGE_ID = Value
- val SHUFFLE_SERVICE_NAME = Value
- val SIZE = Value
- val SLEEP_TIME = Value
- val SLIDE_DURATION = Value
- val SMALLEST_CLUSTER_INDEX = Value
- val SPARK_DATA_STREAM = Value
- val SPARK_PLAN_ID = Value
- val SQL_TEXT = Value
- val SRC_PATH = Value
- val STAGE_ATTEMPT = Value
- val STAGE_ID = Value
- val START_INDEX = Value
- val STATEMENT_ID = Value
- val STATE_STORE_PROVIDER = Value
- val STATUS = Value
- val STDERR = Value
- val STORAGE_LEVEL = Value
- val STORAGE_LEVEL_DESERIALIZED = Value
- val STORAGE_LEVEL_REPLICATION = Value
- val STORE_ID = Value
- val STREAM_ID = Value
- val STREAM_NAME = Value
- val SUBMISSION_ID = Value
- val SUBSAMPLING_RATE = Value
- val SUB_QUERY = Value
- val TABLE_NAME = Value
- val TABLE_TYPES = Value
- val TARGET_PATH = Value
- val TASK_ATTEMPT_ID = Value
- val TASK_ID = Value
- val TASK_NAME = Value
- val TASK_SET_NAME = Value
- val TASK_STATE = Value
- val TEMP_FILE = Value
- val TEMP_PATH = Value
- val TEST_SIZE = Value
- val THREAD = Value
- val THREAD_NAME = Value
- val TID = Value
- val TIME = Value
- val TIMEOUT = Value
- val TIMER = Value
- val TIMER_LABEL = Value
- val TIME_UNITS = Value
- val TIP = Value
- val TOKEN_REGEX = Value
- val TOPIC = Value
- val TOPIC_PARTITION = Value
- val TOPIC_PARTITIONS = Value
- val TOPIC_PARTITION_OFFSET = Value
- val TOPIC_PARTITION_OFFSET_RANGE = Value
- val TOTAL = Value
- val TOTAL_EFFECTIVE_TIME = Value
- val TOTAL_RECORDS_READ = Value
- val TOTAL_TIME = Value
- val TOTAL_TIME_READ = Value
- val TO_TIME = Value
- val TRAINING_SIZE = Value
- val TRAIN_VALIDATION_SPLIT_METRIC = Value
- val TRAIN_VALIDATION_SPLIT_METRICS = Value
- val TRAIN_WORD_COUNT = Value
- val TREE_NODE = Value
- val TRIGGER_INTERVAL = Value
- val UI_FILTER = Value
- val UI_FILTER_PARAMS = Value
- val UI_PROXY_BASE = Value
- val UNSUPPORTED_EXPR = Value
- val UNSUPPORTED_HINT_REASON = Value
- val UNTIL_OFFSET = Value
- val UPPER_BOUND = Value
- val URI = Value
- val USER_ID = Value
- val USER_NAME = Value
- val VALUE = Value
- val VIRTUAL_CORES = Value
- val VOCAB_SIZE = Value
- val WAIT_RESULT_TIME = Value
- val WAIT_SEND_TIME = Value
- val WAIT_TIME = Value
- val WATERMARK_CONSTRAINT = Value
- val WEIGHTED_NUM = Value
- val WORKER_URL = Value
- val WRITE_AHEAD_LOG_INFO = Value
- val WRITE_JOB_UUID = Value
- val XSD_PATH = Value
-
- type LogKey = Value
+object LogKeys {
+ case object ACCUMULATOR_ID extends LogKey
+ case object ACTUAL_NUM_FILES extends LogKey
+ case object ACTUAL_PARTITION_COLUMN extends LogKey
+ case object ALPHA extends LogKey
+ case object ANALYSIS_ERROR extends LogKey
+ case object APP_ATTEMPT_ID extends LogKey
+ case object APP_DESC extends LogKey
+ case object APP_ID extends LogKey
+ case object APP_NAME extends LogKey
+ case object APP_STATE extends LogKey
+ case object ARGS extends LogKey
+ case object BACKUP_FILE extends LogKey
+ case object BATCH_ID extends LogKey
+ case object BATCH_WRITE extends LogKey
+ case object BLOCK_ID extends LogKey
+ case object BLOCK_MANAGER_ID extends LogKey
+ case object BROADCAST_ID extends LogKey
+ case object BUCKET extends LogKey
+ case object BYTECODE_SIZE extends LogKey
+ case object CACHED_TABLE_PARTITION_METADATA_SIZE extends LogKey
+ case object CACHE_AUTO_REMOVED_SIZE extends LogKey
+ case object CACHE_UNTIL_HIGHEST_CONSUMED_SIZE extends LogKey
+ case object CACHE_UNTIL_LAST_PRODUCED_SIZE extends LogKey
+ case object CALL_SITE_LONG_FORM extends LogKey
+ case object CATALOG_NAME extends LogKey
+ case object CATEGORICAL_FEATURES extends LogKey
+ case object CHECKPOINT_FILE extends LogKey
+ case object CHECKPOINT_TIME extends LogKey
+ case object CHECKSUM_FILE_NUM extends LogKey
+ case object CLASS_LOADER extends LogKey
+ case object CLASS_NAME extends LogKey
+ case object CLUSTER_CENTROIDS extends LogKey
+ case object CLUSTER_ID extends LogKey
+ case object CLUSTER_LABEL extends LogKey
+ case object CLUSTER_LEVEL extends LogKey
+ case object CLUSTER_WEIGHT extends LogKey
+ case object CODEC_LEVEL extends LogKey
+ case object CODEC_NAME extends LogKey
+ case object CODEGEN_STAGE_ID extends LogKey
+ case object COLUMN_DATA_TYPE_SOURCE extends LogKey
+ case object COLUMN_DATA_TYPE_TARGET extends LogKey
+ case object COLUMN_DEFAULT_VALUE extends LogKey
+ case object COLUMN_NAME extends LogKey
+ case object COMMAND extends LogKey
+ case object COMMAND_OUTPUT extends LogKey
+ case object COMPONENT extends LogKey
+ case object CONFIG extends LogKey
+ case object CONFIG2 extends LogKey
+ case object CONFIG3 extends LogKey
+ case object CONFIG4 extends LogKey
+ case object CONFIG5 extends LogKey
+ case object CONSUMER extends LogKey
+ case object CONTAINER extends LogKey
+ case object CONTAINER_ID extends LogKey
+ case object CONTAINER_STATE extends LogKey
+ case object COST extends LogKey
+ case object COUNT extends LogKey
+ case object CROSS_VALIDATION_METRIC extends LogKey
+ case object CROSS_VALIDATION_METRICS extends LogKey
+ case object CSV_HEADER_COLUMN_NAME extends LogKey
+ case object CSV_HEADER_COLUMN_NAMES extends LogKey
+ case object CSV_HEADER_LENGTH extends LogKey
+ case object CSV_SCHEMA_FIELD_NAME extends LogKey
+ case object CSV_SCHEMA_FIELD_NAMES extends LogKey
+ case object CSV_SOURCE extends LogKey
+ case object CURRENT_PATH extends LogKey
+ case object DATA extends LogKey
+ case object DATABASE_NAME extends LogKey
+ case object DATAFRAME_CACHE_ENTRY extends LogKey
+ case object DATAFRAME_ID extends LogKey
+ case object DATA_FILE_NUM extends LogKey
+ case object DATA_SOURCE extends LogKey
+ case object DATA_SOURCES extends LogKey
+ case object DATA_SOURCE_PROVIDER extends LogKey
+ case object DEFAULT_ISOLATION_LEVEL extends LogKey
+ case object DEFAULT_VALUE extends LogKey
+ case object DELAY extends LogKey
+ case object DELEGATE extends LogKey
+ case object DELTA extends LogKey
+ case object DESCRIPTION extends LogKey
+ case object DESIRED_PARTITIONS_SIZE extends LogKey
+ case object DIFF_DELTA extends LogKey
+ case object DIVISIBLE_CLUSTER_INDICES_SIZE extends LogKey
+ case object DRIVER_ID extends LogKey
+ case object DROPPED_PARTITIONS extends LogKey
+ case object DURATION extends LogKey
+ case object EFFECTIVE_STORAGE_LEVEL extends LogKey
+ case object ELAPSED_TIME extends LogKey
+ case object ENCODING extends LogKey
+ case object END_INDEX extends LogKey
+ case object END_POINT extends LogKey
+ case object ENGINE extends LogKey
+ case object ERROR extends LogKey
+ case object ESTIMATOR_PARAMETER_MAP extends LogKey
+ case object EVENT_LOOP extends LogKey
+ case object EVENT_QUEUE extends LogKey
+ case object EXCEPTION extends LogKey
+ case object EXECUTE_INFO extends LogKey
+ case object EXECUTE_KEY extends LogKey
+ case object EXECUTION_PLAN_LEAVES extends LogKey
+ case object EXECUTOR_DESIRED_COUNT extends LogKey
+ case object EXECUTOR_ENVS extends LogKey
+ case object EXECUTOR_ENV_REGEX extends LogKey
+ case object EXECUTOR_ID extends LogKey
+ case object EXECUTOR_IDS extends LogKey
+ case object EXECUTOR_LAUNCH_COMMANDS extends LogKey
+ case object EXECUTOR_LAUNCH_COUNT extends LogKey
+ case object EXECUTOR_RESOURCES extends LogKey
+ case object EXECUTOR_STATE extends LogKey
+ case object EXECUTOR_TARGET_COUNT extends LogKey
+ case object EXIT_CODE extends LogKey
+ case object EXPECTED_NUM_FILES extends LogKey
+ case object EXPECTED_PARTITION_COLUMN extends LogKey
+ case object EXPIRY_TIMESTAMP extends LogKey
+ case object EXPR extends LogKey
+ case object EXPR_TERMS extends LogKey
+ case object EXTENDED_EXPLAIN_GENERATOR extends LogKey
+ case object FAILURES extends LogKey
+ case object FALLBACK_VERSION extends LogKey
+ case object FEATURE_COLUMN extends LogKey
+ case object FEATURE_DIMENSION extends LogKey
+ case object FIELD_NAME extends LogKey
+ case object FILE_ABSOLUTE_PATH extends LogKey
+ case object FILE_FORMAT extends LogKey
+ case object FILE_FORMAT2 extends LogKey
+ case object FILE_NAME extends LogKey
+ case object FILE_VERSION extends LogKey
+ case object FINISH_TRIGGER_DURATION extends LogKey
+ case object FROM_OFFSET extends LogKey
+ case object FROM_TIME extends LogKey
+ case object FUNCTION_NAME extends LogKey
+ case object FUNCTION_PARAMETER extends LogKey
+ case object GROUP_ID extends LogKey
+ case object HADOOP_VERSION extends LogKey
+ case object HASH_JOIN_KEYS extends LogKey
+ case object HEARTBEAT_INTERVAL extends LogKey
+ case object HISTORY_DIR extends LogKey
+ case object HIVE_CLIENT_VERSION extends LogKey
+ case object HIVE_METASTORE_VERSION extends LogKey
+ case object HIVE_OPERATION_STATE extends LogKey
+ case object HIVE_OPERATION_TYPE extends LogKey
+ case object HOST extends LogKey
+ case object HOST_PORT extends LogKey
+ case object IDENTIFIER extends LogKey
+ case object INCOMPATIBLE_TYPES extends LogKey
+ case object INDEX extends LogKey
+ case object INDEX_FILE_NUM extends LogKey
+ case object INDEX_NAME extends LogKey
+ case object INFERENCE_MODE extends LogKey
+ case object INITIAL_CAPACITY extends LogKey
+ case object INITIAL_HEARTBEAT_INTERVAL extends LogKey
+ case object INIT_MODE extends LogKey
+ case object INTERVAL extends LogKey
+ case object ISOLATION_LEVEL extends LogKey
+ case object JOB_ID extends LogKey
+ case object JOIN_CONDITION extends LogKey
+ case object JOIN_CONDITION_SUB_EXPR extends LogKey
+ case object K8S_CONTEXT extends LogKey
+ case object KAFKA_PULLS_COUNT extends LogKey
+ case object KAFKA_RECORDS_PULLED_COUNT extends LogKey
+ case object KEY extends LogKey
+ case object KEYTAB extends LogKey
+ case object LABEL_COLUMN extends LogKey
+ case object LARGEST_CLUSTER_INDEX extends LogKey
+ case object LAST_ACCESS_TIME extends LogKey
+ case object LAST_VALID_TIME extends LogKey
+ case object LATEST_BATCH_ID extends LogKey
+ case object LATEST_COMMITTED_BATCH_ID extends LogKey
+ case object LEARNING_RATE extends LogKey
+ case object LEFT_EXPR extends LogKey
+ case object LINE extends LogKey
+ case object LINE_NUM extends LogKey
+ case object LISTENER extends LogKey
+ case object LOAD_FACTOR extends LogKey
+ case object LOAD_TIME extends LogKey
+ case object LOGICAL_PLAN_COLUMNS extends LogKey
+ case object LOGICAL_PLAN_LEAVES extends LogKey
+ case object LOG_TYPE extends LogKey
+ case object LOWER_BOUND extends LogKey
+ case object MALFORMATTED_STIRNG extends LogKey
+ case object MASTER_URL extends LogKey
+ case object MAX_ATTEMPTS extends LogKey
+ case object MAX_CACHE_UNTIL_HIGHEST_CONSUMED_SIZE extends LogKey
+ case object MAX_CACHE_UNTIL_LAST_PRODUCED_SIZE extends LogKey
+ case object MAX_CAPACITY extends LogKey
+ case object MAX_CATEGORIES extends LogKey
+ case object MAX_EXECUTOR_FAILURES extends LogKey
+ case object MAX_FILE_VERSION extends LogKey
+ case object MAX_MEMORY_SIZE extends LogKey
+ case object MAX_PARTITIONS_SIZE extends LogKey
+ case object MAX_SIZE extends LogKey
+ case object MAX_TABLE_PARTITION_METADATA_SIZE extends LogKey
+ case object MEMORY_SIZE extends LogKey
+ case object MERGE_DIR_NAME extends LogKey
+ case object MESSAGE extends LogKey
+ case object METHOD_NAME extends LogKey
+ case object MIN_FREQUENT_PATTERN_COUNT extends LogKey
+ case object MIN_POINT_PER_CLUSTER extends LogKey
+ case object MIN_SIZE extends LogKey
+ case object MODEL_WEIGHTS extends LogKey
+ case object NAMESPACE extends LogKey
+ case object NEW_FEATURE_COLUMN_NAME extends LogKey
+ case object NEW_LABEL_COLUMN_NAME extends LogKey
+ case object NEW_PATH extends LogKey
+ case object NEW_VALUE extends LogKey
+ case object NODES extends LogKey
+ case object NODE_LOCATION extends LogKey
+ case object NORM extends LogKey
+ case object NUM_BIN extends LogKey
+ case object NUM_CLASSES extends LogKey
+ case object NUM_COLUMNS extends LogKey
+ case object NUM_EXAMPLES extends LogKey
+ case object NUM_FEATURES extends LogKey
+ case object NUM_FILES extends LogKey
+ case object NUM_FREQUENT_ITEMS extends LogKey
+ case object NUM_ITERATIONS extends LogKey
+ case object NUM_LOCAL_FREQUENT_PATTERN extends LogKey
+ case object NUM_PARTITION extends LogKey
+ case object NUM_POINT extends LogKey
+ case object NUM_PREFIXES extends LogKey
+ case object NUM_SEQUENCES extends LogKey
+ case object OBJECT_ID extends LogKey
+ case object OFFSET extends LogKey
+ case object OFFSETS extends LogKey
+ case object OLD_BLOCK_MANAGER_ID extends LogKey
+ case object OLD_VALUE extends LogKey
+ case object OPTIMIZED_PLAN_COLUMNS extends LogKey
+ case object OPTIMIZER_CLASS_NAME extends LogKey
+ case object OPTIONS extends LogKey
+ case object OP_ID extends LogKey
+ case object OP_TYPE extends LogKey
+ case object OVERHEAD_MEMORY_SIZE extends LogKey
+ case object PARSE_MODE extends LogKey
+ case object PARTITIONED_FILE_READER extends LogKey
+ case object PARTITIONS_SIZE extends LogKey
+ case object PARTITION_ID extends LogKey
+ case object PARTITION_SPECIFICATION extends LogKey
+ case object PARTITION_SPECS extends LogKey
+ case object PATH extends LogKey
+ case object PATHS extends LogKey
+ case object PIPELINE_STAGE_UID extends LogKey
+ case object POD_COUNT extends LogKey
+ case object POD_ID extends LogKey
+ case object POD_NAME extends LogKey
+ case object POD_NAMESPACE extends LogKey
+ case object POD_PHASE extends LogKey
+ case object POD_SHARED_SLOT_COUNT extends LogKey
+ case object POD_STATE extends LogKey
+ case object POD_TARGET_COUNT extends LogKey
+ case object POLICY extends LogKey
+ case object PORT extends LogKey
+ case object PRINCIPAL extends LogKey
+ case object PROCESSING_TIME extends LogKey
+ case object PRODUCER_ID extends LogKey
+ case object PROVIDER extends LogKey
+ case object PVC_METADATA_NAME extends LogKey
+ case object QUERY_CACHE_VALUE extends LogKey
+ case object QUERY_HINT extends LogKey
+ case object QUERY_ID extends LogKey
+ case object QUERY_PLAN extends LogKey
+ case object QUERY_PLAN_COMPARISON extends LogKey
+ case object QUERY_PLAN_LENGTH_ACTUAL extends LogKey
+ case object QUERY_PLAN_LENGTH_MAX extends LogKey
+ case object QUERY_RUN_ID extends LogKey
+ case object RANGE extends LogKey
+ case object RDD_ID extends LogKey
+ case object READ_LIMIT extends LogKey
+ case object REASON extends LogKey
+ case object REATTACHABLE extends LogKey
+ case object RECEIVED_BLOCK_INFO extends LogKey
+ case object RECEIVED_BLOCK_TRACKER_LOG_EVENT extends LogKey
+ case object RECEIVER_ID extends LogKey
+ case object RECEIVER_IDS extends LogKey
+ case object RECORDS extends LogKey
+ case object REDACTED_STATEMENT extends LogKey
+ case object REDUCE_ID extends LogKey
+ case object RELATION_NAME extends LogKey
+ case object RELATIVE_TOLERANCE extends LogKey
+ case object REMAINING_PARTITIONS extends LogKey
+ case object REPORT_DETAILS extends LogKey
+ case object RESOURCE extends LogKey
+ case object RESOURCE_NAME extends LogKey
+ case object RESOURCE_PROFILE_ID extends LogKey
+ case object RESOURCE_PROFILE_IDS extends LogKey
+ case object RETRY_COUNT extends LogKey
+ case object RETRY_INTERVAL extends LogKey
+ case object RIGHT_EXPR extends LogKey
+ case object RMSE extends LogKey
+ case object RPC_ENDPOINT_REF extends LogKey
+ case object RULE_BATCH_NAME extends LogKey
+ case object RULE_NAME extends LogKey
+ case object RULE_NUMBER_OF_RUNS extends LogKey
+ case object RUN_ID extends LogKey
+ case object SCHEMA extends LogKey
+ case object SCHEMA2 extends LogKey
+ case object SERVICE_NAME extends LogKey
+ case object SESSION_HOLD_INFO extends LogKey
+ case object SESSION_ID extends LogKey
+ case object SESSION_KEY extends LogKey
+ case object SHARD_ID extends LogKey
+ case object SHUFFLE_BLOCK_INFO extends LogKey
+ case object SHUFFLE_ID extends LogKey
+ case object SHUFFLE_MERGE_ID extends LogKey
+ case object SHUFFLE_SERVICE_NAME extends LogKey
+ case object SIZE extends LogKey
+ case object SLEEP_TIME extends LogKey
+ case object SLIDE_DURATION extends LogKey
+ case object SMALLEST_CLUSTER_INDEX extends LogKey
+ case object SPARK_DATA_STREAM extends LogKey
+ case object SPARK_PLAN_ID extends LogKey
+ case object SQL_TEXT extends LogKey
+ case object SRC_PATH extends LogKey
+ case object STAGE_ATTEMPT extends LogKey
+ case object STAGE_ID extends LogKey
+ case object START_INDEX extends LogKey
+ case object STATEMENT_ID extends LogKey
+ case object STATE_STORE_PROVIDER extends LogKey
+ case object STATUS extends LogKey
+ case object STDERR extends LogKey
+ case object STORAGE_LEVEL extends LogKey
+ case object STORAGE_LEVEL_DESERIALIZED extends LogKey
+ case object STORAGE_LEVEL_REPLICATION extends LogKey
+ case object STORE_ID extends LogKey
+ case object STREAM_ID extends LogKey
+ case object STREAM_NAME extends LogKey
+ case object SUBMISSION_ID extends LogKey
+ case object SUBSAMPLING_RATE extends LogKey
+ case object SUB_QUERY extends LogKey
+ case object TABLE_NAME extends LogKey
+ case object TABLE_TYPES extends LogKey
+ case object TARGET_PATH extends LogKey
+ case object TASK_ATTEMPT_ID extends LogKey
+ case object TASK_ID extends LogKey
+ case object TASK_NAME extends LogKey
+ case object TASK_SET_NAME extends LogKey
+ case object TASK_STATE extends LogKey
+ case object TEMP_FILE extends LogKey
+ case object TEMP_PATH extends LogKey
+ case object TEST_SIZE extends LogKey
+ case object THREAD extends LogKey
+ case object THREAD_NAME extends LogKey
+ case object TID extends LogKey
+ case object TIME extends LogKey
+ case object TIMEOUT extends LogKey
+ case object TIMER extends LogKey
+ case object TIMER_LABEL extends LogKey
Review Comment:
`TIMER_LABEL` and `TIME_UNITS` are not being used, should we temporarily clean them up?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47963][CORE] Make the external Spark ecosystem can use structured logging mechanisms [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #46193:
URL: https://github.com/apache/spark/pull/46193#discussion_r1578786930
##########
common/utils/src/main/scala/org/apache/spark/internal/LogKey.scala:
##########
@@ -16,399 +16,399 @@
*/
package org.apache.spark.internal
+trait LogKey
Review Comment:
Relevant explanatory comments should be added for `LogKey`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47963][CORE] Make the external Spark ecosystem can use structured logging mechanisms [spark]
Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang commented on code in PR #46193:
URL: https://github.com/apache/spark/pull/46193#discussion_r1578624819
##########
common/utils/src/test/scala/org/apache/spark/util/StructuredLoggingSuite.scala:
##########
@@ -144,6 +146,22 @@ trait LoggingSuiteBase
}
}
+ // An enumeration value that is not defined in `LogKey`, you can define it anywhere
+ private val externalLogKey = LogKey.VALUE
+ private val externalLog = log"${MDC(externalLogKey, "External log message.")}"
+ test("Logging with external LogKey") {
Review Comment:
Yeah creating a trait seems better. I have some preference in the naming. See my other comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47963][CORE] Make the external Spark ecosystem can use structured logging mechanisms [spark]
Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on code in PR #46193:
URL: https://github.com/apache/spark/pull/46193#discussion_r1578641891
##########
common/utils/src/main/scala/org/apache/spark/internal/LogKey.scala:
##########
@@ -16,393 +16,393 @@
*/
package org.apache.spark.internal
+trait ILogKey
+
/**
* Various keys used for mapped diagnostic contexts(MDC) in logging.
* All structured logging keys should be defined here for standardization.
*/
-object LogKey extends Enumeration {
- val ACCUMULATOR_ID = Value
- val ACTUAL_NUM_FILES = Value
- val ACTUAL_PARTITION_COLUMN = Value
- val ALPHA = Value
- val ANALYSIS_ERROR = Value
- val APP_ATTEMPT_ID = Value
- val APP_DESC = Value
- val APP_ID = Value
- val APP_NAME = Value
- val APP_STATE = Value
- val ARGS = Value
- val BACKUP_FILE = Value
- val BATCH_ID = Value
- val BLOCK_ID = Value
- val BLOCK_MANAGER_ID = Value
- val BROADCAST_ID = Value
- val BUCKET = Value
- val BYTECODE_SIZE = Value
- val CACHED_TABLE_PARTITION_METADATA_SIZE = Value
- val CACHE_AUTO_REMOVED_SIZE = Value
- val CACHE_UNTIL_HIGHEST_CONSUMED_SIZE = Value
- val CACHE_UNTIL_LAST_PRODUCED_SIZE = Value
- val CALL_SITE_LONG_FORM = Value
- val CATALOG_NAME = Value
- val CATEGORICAL_FEATURES = Value
- val CHECKPOINT_FILE = Value
- val CHECKPOINT_TIME = Value
- val CHECKSUM_FILE_NUM = Value
- val CLASS_LOADER = Value
- val CLASS_NAME = Value
- val CLUSTER_CENTROIDS = Value
- val CLUSTER_ID = Value
- val CLUSTER_LABEL = Value
- val CLUSTER_LEVEL = Value
- val CLUSTER_WEIGHT = Value
- val CODEC_LEVEL = Value
- val CODEC_NAME = Value
- val CODEGEN_STAGE_ID = Value
- val COLUMN_DATA_TYPE_SOURCE = Value
- val COLUMN_DATA_TYPE_TARGET = Value
- val COLUMN_DEFAULT_VALUE = Value
- val COLUMN_NAME = Value
- val COMMAND = Value
- val COMMAND_OUTPUT = Value
- val COMPONENT = Value
- val CONFIG = Value
- val CONFIG2 = Value
- val CONFIG3 = Value
- val CONFIG4 = Value
- val CONFIG5 = Value
- val CONSUMER = Value
- val CONTAINER = Value
- val CONTAINER_ID = Value
- val CONTAINER_STATE = Value
- val COST = Value
- val COUNT = Value
- val CROSS_VALIDATION_METRIC = Value
- val CROSS_VALIDATION_METRICS = Value
- val CSV_HEADER_COLUMN_NAME = Value
- val CSV_HEADER_COLUMN_NAMES = Value
- val CSV_HEADER_LENGTH = Value
- val CSV_SCHEMA_FIELD_NAME = Value
- val CSV_SCHEMA_FIELD_NAMES = Value
- val CSV_SOURCE = Value
- val CURRENT_PATH = Value
- val DATA = Value
- val DATABASE_NAME = Value
- val DATAFRAME_CACHE_ENTRY = Value
- val DATAFRAME_ID = Value
- val DATA_FILE_NUM = Value
- val DATA_SOURCE = Value
- val DATA_SOURCES = Value
- val DATA_SOURCE_PROVIDER = Value
- val DEFAULT_ISOLATION_LEVEL = Value
- val DEFAULT_VALUE = Value
- val DELAY = Value
- val DELEGATE = Value
- val DELTA = Value
- val DESCRIPTION = Value
- val DESIRED_PARTITIONS_SIZE = Value
- val DIFF_DELTA = Value
- val DIVISIBLE_CLUSTER_INDICES_SIZE = Value
- val DRIVER_ID = Value
- val DROPPED_PARTITIONS = Value
- val DURATION = Value
- val EFFECTIVE_STORAGE_LEVEL = Value
- val ELAPSED_TIME = Value
- val ENCODING = Value
- val END_INDEX = Value
- val END_POINT = Value
- val ENGINE = Value
- val ERROR = Value
- val ESTIMATOR_PARAMETER_MAP = Value
- val EVENT_LOOP = Value
- val EVENT_QUEUE = Value
- val EXECUTE_INFO = Value
- val EXECUTE_KEY = Value
- val EXECUTION_PLAN_LEAVES = Value
- val EXECUTOR_DESIRED_COUNT = Value
- val EXECUTOR_ENVS = Value
- val EXECUTOR_ENV_REGEX = Value
- val EXECUTOR_ID = Value
- val EXECUTOR_IDS = Value
- val EXECUTOR_LAUNCH_COMMANDS = Value
- val EXECUTOR_LAUNCH_COUNT = Value
- val EXECUTOR_RESOURCES = Value
- val EXECUTOR_STATE = Value
- val EXECUTOR_TARGET_COUNT = Value
- val EXIT_CODE = Value
- val EXPECTED_NUM_FILES = Value
- val EXPECTED_PARTITION_COLUMN = Value
- val EXPIRY_TIMESTAMP = Value
- val EXPR = Value
- val EXPR_TERMS = Value
- val EXTENDED_EXPLAIN_GENERATOR = Value
- val FAILURES = Value
- val FALLBACK_VERSION = Value
- val FEATURE_COLUMN = Value
- val FEATURE_DIMENSION = Value
- val FIELD_NAME = Value
- val FILE_ABSOLUTE_PATH = Value
- val FILE_FORMAT = Value
- val FILE_FORMAT2 = Value
- val FILE_NAME = Value
- val FILE_VERSION = Value
- val FINISH_TRIGGER_DURATION = Value
- val FROM_OFFSET = Value
- val FROM_TIME = Value
- val FUNCTION_NAME = Value
- val FUNCTION_PARAMETER = Value
- val GROUP_ID = Value
- val HADOOP_VERSION = Value
- val HASH_JOIN_KEYS = Value
- val HEARTBEAT_INTERVAL = Value
- val HISTORY_DIR = Value
- val HIVE_CLIENT_VERSION = Value
- val HIVE_METASTORE_VERSION = Value
- val HIVE_OPERATION_STATE = Value
- val HIVE_OPERATION_TYPE = Value
- val HOST = Value
- val HOST_PORT = Value
- val INCOMPATIBLE_TYPES = Value
- val INDEX = Value
- val INDEX_FILE_NUM = Value
- val INDEX_NAME = Value
- val INFERENCE_MODE = Value
- val INITIAL_CAPACITY = Value
- val INITIAL_HEARTBEAT_INTERVAL = Value
- val INIT_MODE = Value
- val INTERVAL = Value
- val ISOLATION_LEVEL = Value
- val JOB_ID = Value
- val JOIN_CONDITION = Value
- val JOIN_CONDITION_SUB_EXPR = Value
- val K8S_CONTEXT = Value
- val KAFKA_PULLS_COUNT = Value
- val KAFKA_RECORDS_PULLED_COUNT = Value
- val KEY = Value
- val KEYTAB = Value
- val LABEL_COLUMN = Value
- val LARGEST_CLUSTER_INDEX = Value
- val LAST_ACCESS_TIME = Value
- val LAST_VALID_TIME = Value
- val LATEST_BATCH_ID = Value
- val LATEST_COMMITTED_BATCH_ID = Value
- val LEARNING_RATE = Value
- val LEFT_EXPR = Value
- val LINE = Value
- val LINE_NUM = Value
- val LISTENER = Value
- val LOAD_FACTOR = Value
- val LOAD_TIME = Value
- val LOGICAL_PLAN_COLUMNS = Value
- val LOGICAL_PLAN_LEAVES = Value
- val LOG_TYPE = Value
- val LOWER_BOUND = Value
- val MALFORMATTED_STIRNG = Value
- val MASTER_URL = Value
- val MAX_ATTEMPTS = Value
- val MAX_CACHE_UNTIL_HIGHEST_CONSUMED_SIZE = Value
- val MAX_CACHE_UNTIL_LAST_PRODUCED_SIZE = Value
- val MAX_CAPACITY = Value
- val MAX_CATEGORIES = Value
- val MAX_EXECUTOR_FAILURES = Value
- val MAX_FILE_VERSION = Value
- val MAX_MEMORY_SIZE = Value
- val MAX_PARTITIONS_SIZE = Value
- val MAX_SIZE = Value
- val MAX_TABLE_PARTITION_METADATA_SIZE = Value
- val MEMORY_SIZE = Value
- val MERGE_DIR_NAME = Value
- val MESSAGE = Value
- val METHOD_NAME = Value
- val MIN_FREQUENT_PATTERN_COUNT = Value
- val MIN_POINT_PER_CLUSTER = Value
- val MIN_SIZE = Value
- val MODEL_WEIGHTS = Value
- val NAMESPACE = Value
- val NEW_FEATURE_COLUMN_NAME = Value
- val NEW_LABEL_COLUMN_NAME = Value
- val NEW_PATH = Value
- val NEW_VALUE = Value
- val NODES = Value
- val NODE_LOCATION = Value
- val NORM = Value
- val NUM_BIN = Value
- val NUM_CLASSES = Value
- val NUM_COLUMNS = Value
- val NUM_EXAMPLES = Value
- val NUM_FEATURES = Value
- val NUM_FILES = Value
- val NUM_FREQUENT_ITEMS = Value
- val NUM_ITERATIONS = Value
- val NUM_LOCAL_FREQUENT_PATTERN = Value
- val NUM_PARTITION = Value
- val NUM_POINT = Value
- val NUM_PREFIXES = Value
- val NUM_SEQUENCES = Value
- val OBJECT_ID = Value
- val OFFSET = Value
- val OFFSETS = Value
- val OLD_BLOCK_MANAGER_ID = Value
- val OLD_VALUE = Value
- val OPTIMIZED_PLAN_COLUMNS = Value
- val OPTIMIZER_CLASS_NAME = Value
- val OPTIONS = Value
- val OP_ID = Value
- val OP_TYPE = Value
- val OVERHEAD_MEMORY_SIZE = Value
- val PARSE_MODE = Value
- val PARTITIONED_FILE_READER = Value
- val PARTITIONS_SIZE = Value
- val PARTITION_ID = Value
- val PARTITION_SPECIFICATION = Value
- val PARTITION_SPECS = Value
- val PATH = Value
- val PATHS = Value
- val PIPELINE_STAGE_UID = Value
- val POD_COUNT = Value
- val POD_ID = Value
- val POD_NAME = Value
- val POD_NAMESPACE = Value
- val POD_PHASE = Value
- val POD_SHARED_SLOT_COUNT = Value
- val POD_STATE = Value
- val POD_TARGET_COUNT = Value
- val POLICY = Value
- val PORT = Value
- val PRINCIPAL = Value
- val PROCESSING_TIME = Value
- val PRODUCER_ID = Value
- val PROVIDER = Value
- val PVC_METADATA_NAME = Value
- val QUERY_CACHE_VALUE = Value
- val QUERY_HINT = Value
- val QUERY_ID = Value
- val QUERY_PLAN = Value
- val QUERY_PLAN_COMPARISON = Value
- val QUERY_PLAN_LENGTH_ACTUAL = Value
- val QUERY_PLAN_LENGTH_MAX = Value
- val QUERY_RUN_ID = Value
- val RANGE = Value
- val RDD_ID = Value
- val READ_LIMIT = Value
- val REASON = Value
- val REATTACHABLE = Value
- val RECEIVED_BLOCK_INFO = Value
- val RECEIVED_BLOCK_TRACKER_LOG_EVENT = Value
- val RECEIVER_ID = Value
- val RECEIVER_IDS = Value
- val RECORDS = Value
- val REDACTED_STATEMENT = Value
- val REDUCE_ID = Value
- val RELATION_NAME = Value
- val RELATIVE_TOLERANCE = Value
- val REMAINING_PARTITIONS = Value
- val REPORT_DETAILS = Value
- val RESOURCE = Value
- val RESOURCE_NAME = Value
- val RESOURCE_PROFILE_ID = Value
- val RESOURCE_PROFILE_IDS = Value
- val RETRY_COUNT = Value
- val RETRY_INTERVAL = Value
- val RIGHT_EXPR = Value
- val RMSE = Value
- val RPC_ENDPOINT_REF = Value
- val RULE_BATCH_NAME = Value
- val RULE_NAME = Value
- val RULE_NUMBER_OF_RUNS = Value
- val RUN_ID = Value
- val SCHEMA = Value
- val SCHEMA2 = Value
- val SERVICE_NAME = Value
- val SESSION_HOLD_INFO = Value
- val SESSION_ID = Value
- val SESSION_KEY = Value
- val SHARD_ID = Value
- val SHUFFLE_BLOCK_INFO = Value
- val SHUFFLE_ID = Value
- val SHUFFLE_MERGE_ID = Value
- val SHUFFLE_SERVICE_NAME = Value
- val SIZE = Value
- val SLEEP_TIME = Value
- val SLIDE_DURATION = Value
- val SMALLEST_CLUSTER_INDEX = Value
- val SPARK_DATA_STREAM = Value
- val SPARK_PLAN_ID = Value
- val SQL_TEXT = Value
- val SRC_PATH = Value
- val STAGE_ID = Value
- val START_INDEX = Value
- val STATEMENT_ID = Value
- val STATE_STORE_PROVIDER = Value
- val STATUS = Value
- val STORAGE_LEVEL = Value
- val STORAGE_LEVEL_DESERIALIZED = Value
- val STORAGE_LEVEL_REPLICATION = Value
- val STORE_ID = Value
- val STREAM_ID = Value
- val STREAM_NAME = Value
- val SUBMISSION_ID = Value
- val SUBSAMPLING_RATE = Value
- val SUB_QUERY = Value
- val TABLE_NAME = Value
- val TABLE_TYPES = Value
- val TARGET_PATH = Value
- val TASK_ATTEMPT_ID = Value
- val TASK_ID = Value
- val TASK_NAME = Value
- val TASK_SET_NAME = Value
- val TASK_STATE = Value
- val TEMP_FILE = Value
- val TEMP_PATH = Value
- val TEST_SIZE = Value
- val THREAD = Value
- val THREAD_NAME = Value
- val TID = Value
- val TIME = Value
- val TIMEOUT = Value
- val TIMER = Value
- val TIMER_LABEL = Value
- val TIME_UNITS = Value
- val TIP = Value
- val TOKEN_REGEX = Value
- val TOPIC = Value
- val TOPIC_PARTITION = Value
- val TOPIC_PARTITIONS = Value
- val TOPIC_PARTITION_OFFSET = Value
- val TOPIC_PARTITION_OFFSET_RANGE = Value
- val TOTAL = Value
- val TOTAL_EFFECTIVE_TIME = Value
- val TOTAL_RECORDS_READ = Value
- val TOTAL_TIME = Value
- val TOTAL_TIME_READ = Value
- val TO_TIME = Value
- val TRAINING_SIZE = Value
- val TRAIN_VALIDATION_SPLIT_METRIC = Value
- val TRAIN_VALIDATION_SPLIT_METRICS = Value
- val TRAIN_WORD_COUNT = Value
- val TREE_NODE = Value
- val TRIGGER_INTERVAL = Value
- val UI_FILTER = Value
- val UI_FILTER_PARAMS = Value
- val UI_PROXY_BASE = Value
- val UNSUPPORTED_EXPR = Value
- val UNSUPPORTED_HINT_REASON = Value
- val UNTIL_OFFSET = Value
- val UPPER_BOUND = Value
- val URI = Value
- val USER_ID = Value
- val USER_NAME = Value
- val VALUE = Value
- val VIRTUAL_CORES = Value
- val VOCAB_SIZE = Value
- val WAIT_RESULT_TIME = Value
- val WAIT_SEND_TIME = Value
- val WAIT_TIME = Value
- val WATERMARK_CONSTRAINT = Value
- val WEIGHTED_NUM = Value
- val WORKER_URL = Value
- val WRITE_AHEAD_LOG_INFO = Value
- val XSD_PATH = Value
-
- type LogKey = Value
+object LogKey extends ILogKey {
Review Comment:
> Yes, it is totally fine. I also tried it on my local. Looks good.
Okay, let me handle it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47963][CORE] Make the external Spark ecosystem can use structured logging mechanisms [spark]
Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang commented on PR #46193:
URL: https://github.com/apache/spark/pull/46193#issuecomment-2078113658
@panbingkun Sorry, I have to merge https://github.com/apache/spark/pull/46192 since it was created before this one. Please resolve the conflicts and I will merge this one. Thanks as always.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47963][CORE] Make the external Spark ecosystem can use structured logging mechanisms [spark]
Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang commented on PR #46193:
URL: https://github.com/apache/spark/pull/46193#issuecomment-2079790129
Thanks, merging to master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47963][CORE] Make the external Spark ecosystem can use structured logging mechanisms [spark]
Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang commented on code in PR #46193:
URL: https://github.com/apache/spark/pull/46193#discussion_r1578624300
##########
common/utils/src/main/scala/org/apache/spark/internal/LogKey.scala:
##########
@@ -16,393 +16,393 @@
*/
package org.apache.spark.internal
+trait ILogKey
+
/**
* Various keys used for mapped diagnostic contexts(MDC) in logging.
* All structured logging keys should be defined here for standardization.
*/
-object LogKey extends Enumeration {
- val ACCUMULATOR_ID = Value
- val ACTUAL_NUM_FILES = Value
- val ACTUAL_PARTITION_COLUMN = Value
- val ALPHA = Value
- val ANALYSIS_ERROR = Value
- val APP_ATTEMPT_ID = Value
- val APP_DESC = Value
- val APP_ID = Value
- val APP_NAME = Value
- val APP_STATE = Value
- val ARGS = Value
- val BACKUP_FILE = Value
- val BATCH_ID = Value
- val BLOCK_ID = Value
- val BLOCK_MANAGER_ID = Value
- val BROADCAST_ID = Value
- val BUCKET = Value
- val BYTECODE_SIZE = Value
- val CACHED_TABLE_PARTITION_METADATA_SIZE = Value
- val CACHE_AUTO_REMOVED_SIZE = Value
- val CACHE_UNTIL_HIGHEST_CONSUMED_SIZE = Value
- val CACHE_UNTIL_LAST_PRODUCED_SIZE = Value
- val CALL_SITE_LONG_FORM = Value
- val CATALOG_NAME = Value
- val CATEGORICAL_FEATURES = Value
- val CHECKPOINT_FILE = Value
- val CHECKPOINT_TIME = Value
- val CHECKSUM_FILE_NUM = Value
- val CLASS_LOADER = Value
- val CLASS_NAME = Value
- val CLUSTER_CENTROIDS = Value
- val CLUSTER_ID = Value
- val CLUSTER_LABEL = Value
- val CLUSTER_LEVEL = Value
- val CLUSTER_WEIGHT = Value
- val CODEC_LEVEL = Value
- val CODEC_NAME = Value
- val CODEGEN_STAGE_ID = Value
- val COLUMN_DATA_TYPE_SOURCE = Value
- val COLUMN_DATA_TYPE_TARGET = Value
- val COLUMN_DEFAULT_VALUE = Value
- val COLUMN_NAME = Value
- val COMMAND = Value
- val COMMAND_OUTPUT = Value
- val COMPONENT = Value
- val CONFIG = Value
- val CONFIG2 = Value
- val CONFIG3 = Value
- val CONFIG4 = Value
- val CONFIG5 = Value
- val CONSUMER = Value
- val CONTAINER = Value
- val CONTAINER_ID = Value
- val CONTAINER_STATE = Value
- val COST = Value
- val COUNT = Value
- val CROSS_VALIDATION_METRIC = Value
- val CROSS_VALIDATION_METRICS = Value
- val CSV_HEADER_COLUMN_NAME = Value
- val CSV_HEADER_COLUMN_NAMES = Value
- val CSV_HEADER_LENGTH = Value
- val CSV_SCHEMA_FIELD_NAME = Value
- val CSV_SCHEMA_FIELD_NAMES = Value
- val CSV_SOURCE = Value
- val CURRENT_PATH = Value
- val DATA = Value
- val DATABASE_NAME = Value
- val DATAFRAME_CACHE_ENTRY = Value
- val DATAFRAME_ID = Value
- val DATA_FILE_NUM = Value
- val DATA_SOURCE = Value
- val DATA_SOURCES = Value
- val DATA_SOURCE_PROVIDER = Value
- val DEFAULT_ISOLATION_LEVEL = Value
- val DEFAULT_VALUE = Value
- val DELAY = Value
- val DELEGATE = Value
- val DELTA = Value
- val DESCRIPTION = Value
- val DESIRED_PARTITIONS_SIZE = Value
- val DIFF_DELTA = Value
- val DIVISIBLE_CLUSTER_INDICES_SIZE = Value
- val DRIVER_ID = Value
- val DROPPED_PARTITIONS = Value
- val DURATION = Value
- val EFFECTIVE_STORAGE_LEVEL = Value
- val ELAPSED_TIME = Value
- val ENCODING = Value
- val END_INDEX = Value
- val END_POINT = Value
- val ENGINE = Value
- val ERROR = Value
- val ESTIMATOR_PARAMETER_MAP = Value
- val EVENT_LOOP = Value
- val EVENT_QUEUE = Value
- val EXECUTE_INFO = Value
- val EXECUTE_KEY = Value
- val EXECUTION_PLAN_LEAVES = Value
- val EXECUTOR_DESIRED_COUNT = Value
- val EXECUTOR_ENVS = Value
- val EXECUTOR_ENV_REGEX = Value
- val EXECUTOR_ID = Value
- val EXECUTOR_IDS = Value
- val EXECUTOR_LAUNCH_COMMANDS = Value
- val EXECUTOR_LAUNCH_COUNT = Value
- val EXECUTOR_RESOURCES = Value
- val EXECUTOR_STATE = Value
- val EXECUTOR_TARGET_COUNT = Value
- val EXIT_CODE = Value
- val EXPECTED_NUM_FILES = Value
- val EXPECTED_PARTITION_COLUMN = Value
- val EXPIRY_TIMESTAMP = Value
- val EXPR = Value
- val EXPR_TERMS = Value
- val EXTENDED_EXPLAIN_GENERATOR = Value
- val FAILURES = Value
- val FALLBACK_VERSION = Value
- val FEATURE_COLUMN = Value
- val FEATURE_DIMENSION = Value
- val FIELD_NAME = Value
- val FILE_ABSOLUTE_PATH = Value
- val FILE_FORMAT = Value
- val FILE_FORMAT2 = Value
- val FILE_NAME = Value
- val FILE_VERSION = Value
- val FINISH_TRIGGER_DURATION = Value
- val FROM_OFFSET = Value
- val FROM_TIME = Value
- val FUNCTION_NAME = Value
- val FUNCTION_PARAMETER = Value
- val GROUP_ID = Value
- val HADOOP_VERSION = Value
- val HASH_JOIN_KEYS = Value
- val HEARTBEAT_INTERVAL = Value
- val HISTORY_DIR = Value
- val HIVE_CLIENT_VERSION = Value
- val HIVE_METASTORE_VERSION = Value
- val HIVE_OPERATION_STATE = Value
- val HIVE_OPERATION_TYPE = Value
- val HOST = Value
- val HOST_PORT = Value
- val INCOMPATIBLE_TYPES = Value
- val INDEX = Value
- val INDEX_FILE_NUM = Value
- val INDEX_NAME = Value
- val INFERENCE_MODE = Value
- val INITIAL_CAPACITY = Value
- val INITIAL_HEARTBEAT_INTERVAL = Value
- val INIT_MODE = Value
- val INTERVAL = Value
- val ISOLATION_LEVEL = Value
- val JOB_ID = Value
- val JOIN_CONDITION = Value
- val JOIN_CONDITION_SUB_EXPR = Value
- val K8S_CONTEXT = Value
- val KAFKA_PULLS_COUNT = Value
- val KAFKA_RECORDS_PULLED_COUNT = Value
- val KEY = Value
- val KEYTAB = Value
- val LABEL_COLUMN = Value
- val LARGEST_CLUSTER_INDEX = Value
- val LAST_ACCESS_TIME = Value
- val LAST_VALID_TIME = Value
- val LATEST_BATCH_ID = Value
- val LATEST_COMMITTED_BATCH_ID = Value
- val LEARNING_RATE = Value
- val LEFT_EXPR = Value
- val LINE = Value
- val LINE_NUM = Value
- val LISTENER = Value
- val LOAD_FACTOR = Value
- val LOAD_TIME = Value
- val LOGICAL_PLAN_COLUMNS = Value
- val LOGICAL_PLAN_LEAVES = Value
- val LOG_TYPE = Value
- val LOWER_BOUND = Value
- val MALFORMATTED_STIRNG = Value
- val MASTER_URL = Value
- val MAX_ATTEMPTS = Value
- val MAX_CACHE_UNTIL_HIGHEST_CONSUMED_SIZE = Value
- val MAX_CACHE_UNTIL_LAST_PRODUCED_SIZE = Value
- val MAX_CAPACITY = Value
- val MAX_CATEGORIES = Value
- val MAX_EXECUTOR_FAILURES = Value
- val MAX_FILE_VERSION = Value
- val MAX_MEMORY_SIZE = Value
- val MAX_PARTITIONS_SIZE = Value
- val MAX_SIZE = Value
- val MAX_TABLE_PARTITION_METADATA_SIZE = Value
- val MEMORY_SIZE = Value
- val MERGE_DIR_NAME = Value
- val MESSAGE = Value
- val METHOD_NAME = Value
- val MIN_FREQUENT_PATTERN_COUNT = Value
- val MIN_POINT_PER_CLUSTER = Value
- val MIN_SIZE = Value
- val MODEL_WEIGHTS = Value
- val NAMESPACE = Value
- val NEW_FEATURE_COLUMN_NAME = Value
- val NEW_LABEL_COLUMN_NAME = Value
- val NEW_PATH = Value
- val NEW_VALUE = Value
- val NODES = Value
- val NODE_LOCATION = Value
- val NORM = Value
- val NUM_BIN = Value
- val NUM_CLASSES = Value
- val NUM_COLUMNS = Value
- val NUM_EXAMPLES = Value
- val NUM_FEATURES = Value
- val NUM_FILES = Value
- val NUM_FREQUENT_ITEMS = Value
- val NUM_ITERATIONS = Value
- val NUM_LOCAL_FREQUENT_PATTERN = Value
- val NUM_PARTITION = Value
- val NUM_POINT = Value
- val NUM_PREFIXES = Value
- val NUM_SEQUENCES = Value
- val OBJECT_ID = Value
- val OFFSET = Value
- val OFFSETS = Value
- val OLD_BLOCK_MANAGER_ID = Value
- val OLD_VALUE = Value
- val OPTIMIZED_PLAN_COLUMNS = Value
- val OPTIMIZER_CLASS_NAME = Value
- val OPTIONS = Value
- val OP_ID = Value
- val OP_TYPE = Value
- val OVERHEAD_MEMORY_SIZE = Value
- val PARSE_MODE = Value
- val PARTITIONED_FILE_READER = Value
- val PARTITIONS_SIZE = Value
- val PARTITION_ID = Value
- val PARTITION_SPECIFICATION = Value
- val PARTITION_SPECS = Value
- val PATH = Value
- val PATHS = Value
- val PIPELINE_STAGE_UID = Value
- val POD_COUNT = Value
- val POD_ID = Value
- val POD_NAME = Value
- val POD_NAMESPACE = Value
- val POD_PHASE = Value
- val POD_SHARED_SLOT_COUNT = Value
- val POD_STATE = Value
- val POD_TARGET_COUNT = Value
- val POLICY = Value
- val PORT = Value
- val PRINCIPAL = Value
- val PROCESSING_TIME = Value
- val PRODUCER_ID = Value
- val PROVIDER = Value
- val PVC_METADATA_NAME = Value
- val QUERY_CACHE_VALUE = Value
- val QUERY_HINT = Value
- val QUERY_ID = Value
- val QUERY_PLAN = Value
- val QUERY_PLAN_COMPARISON = Value
- val QUERY_PLAN_LENGTH_ACTUAL = Value
- val QUERY_PLAN_LENGTH_MAX = Value
- val QUERY_RUN_ID = Value
- val RANGE = Value
- val RDD_ID = Value
- val READ_LIMIT = Value
- val REASON = Value
- val REATTACHABLE = Value
- val RECEIVED_BLOCK_INFO = Value
- val RECEIVED_BLOCK_TRACKER_LOG_EVENT = Value
- val RECEIVER_ID = Value
- val RECEIVER_IDS = Value
- val RECORDS = Value
- val REDACTED_STATEMENT = Value
- val REDUCE_ID = Value
- val RELATION_NAME = Value
- val RELATIVE_TOLERANCE = Value
- val REMAINING_PARTITIONS = Value
- val REPORT_DETAILS = Value
- val RESOURCE = Value
- val RESOURCE_NAME = Value
- val RESOURCE_PROFILE_ID = Value
- val RESOURCE_PROFILE_IDS = Value
- val RETRY_COUNT = Value
- val RETRY_INTERVAL = Value
- val RIGHT_EXPR = Value
- val RMSE = Value
- val RPC_ENDPOINT_REF = Value
- val RULE_BATCH_NAME = Value
- val RULE_NAME = Value
- val RULE_NUMBER_OF_RUNS = Value
- val RUN_ID = Value
- val SCHEMA = Value
- val SCHEMA2 = Value
- val SERVICE_NAME = Value
- val SESSION_HOLD_INFO = Value
- val SESSION_ID = Value
- val SESSION_KEY = Value
- val SHARD_ID = Value
- val SHUFFLE_BLOCK_INFO = Value
- val SHUFFLE_ID = Value
- val SHUFFLE_MERGE_ID = Value
- val SHUFFLE_SERVICE_NAME = Value
- val SIZE = Value
- val SLEEP_TIME = Value
- val SLIDE_DURATION = Value
- val SMALLEST_CLUSTER_INDEX = Value
- val SPARK_DATA_STREAM = Value
- val SPARK_PLAN_ID = Value
- val SQL_TEXT = Value
- val SRC_PATH = Value
- val STAGE_ID = Value
- val START_INDEX = Value
- val STATEMENT_ID = Value
- val STATE_STORE_PROVIDER = Value
- val STATUS = Value
- val STORAGE_LEVEL = Value
- val STORAGE_LEVEL_DESERIALIZED = Value
- val STORAGE_LEVEL_REPLICATION = Value
- val STORE_ID = Value
- val STREAM_ID = Value
- val STREAM_NAME = Value
- val SUBMISSION_ID = Value
- val SUBSAMPLING_RATE = Value
- val SUB_QUERY = Value
- val TABLE_NAME = Value
- val TABLE_TYPES = Value
- val TARGET_PATH = Value
- val TASK_ATTEMPT_ID = Value
- val TASK_ID = Value
- val TASK_NAME = Value
- val TASK_SET_NAME = Value
- val TASK_STATE = Value
- val TEMP_FILE = Value
- val TEMP_PATH = Value
- val TEST_SIZE = Value
- val THREAD = Value
- val THREAD_NAME = Value
- val TID = Value
- val TIME = Value
- val TIMEOUT = Value
- val TIMER = Value
- val TIMER_LABEL = Value
- val TIME_UNITS = Value
- val TIP = Value
- val TOKEN_REGEX = Value
- val TOPIC = Value
- val TOPIC_PARTITION = Value
- val TOPIC_PARTITIONS = Value
- val TOPIC_PARTITION_OFFSET = Value
- val TOPIC_PARTITION_OFFSET_RANGE = Value
- val TOTAL = Value
- val TOTAL_EFFECTIVE_TIME = Value
- val TOTAL_RECORDS_READ = Value
- val TOTAL_TIME = Value
- val TOTAL_TIME_READ = Value
- val TO_TIME = Value
- val TRAINING_SIZE = Value
- val TRAIN_VALIDATION_SPLIT_METRIC = Value
- val TRAIN_VALIDATION_SPLIT_METRICS = Value
- val TRAIN_WORD_COUNT = Value
- val TREE_NODE = Value
- val TRIGGER_INTERVAL = Value
- val UI_FILTER = Value
- val UI_FILTER_PARAMS = Value
- val UI_PROXY_BASE = Value
- val UNSUPPORTED_EXPR = Value
- val UNSUPPORTED_HINT_REASON = Value
- val UNTIL_OFFSET = Value
- val UPPER_BOUND = Value
- val URI = Value
- val USER_ID = Value
- val USER_NAME = Value
- val VALUE = Value
- val VIRTUAL_CORES = Value
- val VOCAB_SIZE = Value
- val WAIT_RESULT_TIME = Value
- val WAIT_SEND_TIME = Value
- val WAIT_TIME = Value
- val WATERMARK_CONSTRAINT = Value
- val WEIGHTED_NUM = Value
- val WORKER_URL = Value
- val WRITE_AHEAD_LOG_INFO = Value
- val XSD_PATH = Value
-
- type LogKey = Value
+object LogKey extends ILogKey {
Review Comment:
Rename as LogKeys? Also, there seems no need to extend any trait.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47963][CORE] Make the external Spark ecosystem can use structured logging mechanisms [spark]
Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on code in PR #46193:
URL: https://github.com/apache/spark/pull/46193#discussion_r1578780993
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala:
##########
@@ -20,15 +20,15 @@ package org.apache.spark.sql.catalyst.optimizer
import scala.collection.mutable
import org.apache.spark.SparkException
-import org.apache.spark.internal.LogKey._
+import org.apache.spark.internal.LogKeys._
import org.apache.spark.internal.MDC
import org.apache.spark.sql.catalyst.SQLConfHelper
import org.apache.spark.sql.catalyst.analysis._
import org.apache.spark.sql.catalyst.catalog.{InMemoryCatalog, SessionCatalog}
import org.apache.spark.sql.catalyst.expressions._
import org.apache.spark.sql.catalyst.expressions.aggregate._
import org.apache.spark.sql.catalyst.plans._
-import org.apache.spark.sql.catalyst.plans.logical.{RepartitionOperation, _}
+import org.apache.spark.sql.catalyst.plans.logical._
Review Comment:
This is not related to the renaming of LogKey -> LogKeys, but it is almost more reasonable to change it to this way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47963][CORE][TESTS] Add an external LogKey usage case in UT [spark]
Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on PR #46193:
URL: https://github.com/apache/spark/pull/46193#issuecomment-2073684346
cc @gengliangwang @LuciferYang
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47963][CORE] Make the external Spark ecosystem can use structured logging mechanisms [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #46193:
URL: https://github.com/apache/spark/pull/46193#discussion_r1577167593
##########
common/utils/src/test/scala/org/apache/spark/util/StructuredLoggingSuite.scala:
##########
@@ -144,6 +146,22 @@ trait LoggingSuiteBase
}
}
+ // An enumeration value that is not defined in `LogKey`, you can define it anywhere
+ private val externalLogKey = LogKey.VALUE
+ private val externalLog = log"${MDC(externalLogKey, "External log message.")}"
+ test("Logging with external LogKey") {
Review Comment:
In https://github.com/apache/spark/pull/46186/files, I try to relax the type of `MDC.key` to `Enumeration#Value`, which allows us to extend the use of other enumeration types as `MDC.key` to enhance a bit of extensibility.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47963][CORE] Make the external Spark ecosystem can use structured logging mechanisms [spark]
Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang commented on code in PR #46193:
URL: https://github.com/apache/spark/pull/46193#discussion_r1578640764
##########
common/utils/src/main/scala/org/apache/spark/internal/LogKey.scala:
##########
@@ -16,393 +16,393 @@
*/
package org.apache.spark.internal
+trait ILogKey
+
/**
* Various keys used for mapped diagnostic contexts(MDC) in logging.
* All structured logging keys should be defined here for standardization.
*/
-object LogKey extends Enumeration {
- val ACCUMULATOR_ID = Value
- val ACTUAL_NUM_FILES = Value
- val ACTUAL_PARTITION_COLUMN = Value
- val ALPHA = Value
- val ANALYSIS_ERROR = Value
- val APP_ATTEMPT_ID = Value
- val APP_DESC = Value
- val APP_ID = Value
- val APP_NAME = Value
- val APP_STATE = Value
- val ARGS = Value
- val BACKUP_FILE = Value
- val BATCH_ID = Value
- val BLOCK_ID = Value
- val BLOCK_MANAGER_ID = Value
- val BROADCAST_ID = Value
- val BUCKET = Value
- val BYTECODE_SIZE = Value
- val CACHED_TABLE_PARTITION_METADATA_SIZE = Value
- val CACHE_AUTO_REMOVED_SIZE = Value
- val CACHE_UNTIL_HIGHEST_CONSUMED_SIZE = Value
- val CACHE_UNTIL_LAST_PRODUCED_SIZE = Value
- val CALL_SITE_LONG_FORM = Value
- val CATALOG_NAME = Value
- val CATEGORICAL_FEATURES = Value
- val CHECKPOINT_FILE = Value
- val CHECKPOINT_TIME = Value
- val CHECKSUM_FILE_NUM = Value
- val CLASS_LOADER = Value
- val CLASS_NAME = Value
- val CLUSTER_CENTROIDS = Value
- val CLUSTER_ID = Value
- val CLUSTER_LABEL = Value
- val CLUSTER_LEVEL = Value
- val CLUSTER_WEIGHT = Value
- val CODEC_LEVEL = Value
- val CODEC_NAME = Value
- val CODEGEN_STAGE_ID = Value
- val COLUMN_DATA_TYPE_SOURCE = Value
- val COLUMN_DATA_TYPE_TARGET = Value
- val COLUMN_DEFAULT_VALUE = Value
- val COLUMN_NAME = Value
- val COMMAND = Value
- val COMMAND_OUTPUT = Value
- val COMPONENT = Value
- val CONFIG = Value
- val CONFIG2 = Value
- val CONFIG3 = Value
- val CONFIG4 = Value
- val CONFIG5 = Value
- val CONSUMER = Value
- val CONTAINER = Value
- val CONTAINER_ID = Value
- val CONTAINER_STATE = Value
- val COST = Value
- val COUNT = Value
- val CROSS_VALIDATION_METRIC = Value
- val CROSS_VALIDATION_METRICS = Value
- val CSV_HEADER_COLUMN_NAME = Value
- val CSV_HEADER_COLUMN_NAMES = Value
- val CSV_HEADER_LENGTH = Value
- val CSV_SCHEMA_FIELD_NAME = Value
- val CSV_SCHEMA_FIELD_NAMES = Value
- val CSV_SOURCE = Value
- val CURRENT_PATH = Value
- val DATA = Value
- val DATABASE_NAME = Value
- val DATAFRAME_CACHE_ENTRY = Value
- val DATAFRAME_ID = Value
- val DATA_FILE_NUM = Value
- val DATA_SOURCE = Value
- val DATA_SOURCES = Value
- val DATA_SOURCE_PROVIDER = Value
- val DEFAULT_ISOLATION_LEVEL = Value
- val DEFAULT_VALUE = Value
- val DELAY = Value
- val DELEGATE = Value
- val DELTA = Value
- val DESCRIPTION = Value
- val DESIRED_PARTITIONS_SIZE = Value
- val DIFF_DELTA = Value
- val DIVISIBLE_CLUSTER_INDICES_SIZE = Value
- val DRIVER_ID = Value
- val DROPPED_PARTITIONS = Value
- val DURATION = Value
- val EFFECTIVE_STORAGE_LEVEL = Value
- val ELAPSED_TIME = Value
- val ENCODING = Value
- val END_INDEX = Value
- val END_POINT = Value
- val ENGINE = Value
- val ERROR = Value
- val ESTIMATOR_PARAMETER_MAP = Value
- val EVENT_LOOP = Value
- val EVENT_QUEUE = Value
- val EXECUTE_INFO = Value
- val EXECUTE_KEY = Value
- val EXECUTION_PLAN_LEAVES = Value
- val EXECUTOR_DESIRED_COUNT = Value
- val EXECUTOR_ENVS = Value
- val EXECUTOR_ENV_REGEX = Value
- val EXECUTOR_ID = Value
- val EXECUTOR_IDS = Value
- val EXECUTOR_LAUNCH_COMMANDS = Value
- val EXECUTOR_LAUNCH_COUNT = Value
- val EXECUTOR_RESOURCES = Value
- val EXECUTOR_STATE = Value
- val EXECUTOR_TARGET_COUNT = Value
- val EXIT_CODE = Value
- val EXPECTED_NUM_FILES = Value
- val EXPECTED_PARTITION_COLUMN = Value
- val EXPIRY_TIMESTAMP = Value
- val EXPR = Value
- val EXPR_TERMS = Value
- val EXTENDED_EXPLAIN_GENERATOR = Value
- val FAILURES = Value
- val FALLBACK_VERSION = Value
- val FEATURE_COLUMN = Value
- val FEATURE_DIMENSION = Value
- val FIELD_NAME = Value
- val FILE_ABSOLUTE_PATH = Value
- val FILE_FORMAT = Value
- val FILE_FORMAT2 = Value
- val FILE_NAME = Value
- val FILE_VERSION = Value
- val FINISH_TRIGGER_DURATION = Value
- val FROM_OFFSET = Value
- val FROM_TIME = Value
- val FUNCTION_NAME = Value
- val FUNCTION_PARAMETER = Value
- val GROUP_ID = Value
- val HADOOP_VERSION = Value
- val HASH_JOIN_KEYS = Value
- val HEARTBEAT_INTERVAL = Value
- val HISTORY_DIR = Value
- val HIVE_CLIENT_VERSION = Value
- val HIVE_METASTORE_VERSION = Value
- val HIVE_OPERATION_STATE = Value
- val HIVE_OPERATION_TYPE = Value
- val HOST = Value
- val HOST_PORT = Value
- val INCOMPATIBLE_TYPES = Value
- val INDEX = Value
- val INDEX_FILE_NUM = Value
- val INDEX_NAME = Value
- val INFERENCE_MODE = Value
- val INITIAL_CAPACITY = Value
- val INITIAL_HEARTBEAT_INTERVAL = Value
- val INIT_MODE = Value
- val INTERVAL = Value
- val ISOLATION_LEVEL = Value
- val JOB_ID = Value
- val JOIN_CONDITION = Value
- val JOIN_CONDITION_SUB_EXPR = Value
- val K8S_CONTEXT = Value
- val KAFKA_PULLS_COUNT = Value
- val KAFKA_RECORDS_PULLED_COUNT = Value
- val KEY = Value
- val KEYTAB = Value
- val LABEL_COLUMN = Value
- val LARGEST_CLUSTER_INDEX = Value
- val LAST_ACCESS_TIME = Value
- val LAST_VALID_TIME = Value
- val LATEST_BATCH_ID = Value
- val LATEST_COMMITTED_BATCH_ID = Value
- val LEARNING_RATE = Value
- val LEFT_EXPR = Value
- val LINE = Value
- val LINE_NUM = Value
- val LISTENER = Value
- val LOAD_FACTOR = Value
- val LOAD_TIME = Value
- val LOGICAL_PLAN_COLUMNS = Value
- val LOGICAL_PLAN_LEAVES = Value
- val LOG_TYPE = Value
- val LOWER_BOUND = Value
- val MALFORMATTED_STIRNG = Value
- val MASTER_URL = Value
- val MAX_ATTEMPTS = Value
- val MAX_CACHE_UNTIL_HIGHEST_CONSUMED_SIZE = Value
- val MAX_CACHE_UNTIL_LAST_PRODUCED_SIZE = Value
- val MAX_CAPACITY = Value
- val MAX_CATEGORIES = Value
- val MAX_EXECUTOR_FAILURES = Value
- val MAX_FILE_VERSION = Value
- val MAX_MEMORY_SIZE = Value
- val MAX_PARTITIONS_SIZE = Value
- val MAX_SIZE = Value
- val MAX_TABLE_PARTITION_METADATA_SIZE = Value
- val MEMORY_SIZE = Value
- val MERGE_DIR_NAME = Value
- val MESSAGE = Value
- val METHOD_NAME = Value
- val MIN_FREQUENT_PATTERN_COUNT = Value
- val MIN_POINT_PER_CLUSTER = Value
- val MIN_SIZE = Value
- val MODEL_WEIGHTS = Value
- val NAMESPACE = Value
- val NEW_FEATURE_COLUMN_NAME = Value
- val NEW_LABEL_COLUMN_NAME = Value
- val NEW_PATH = Value
- val NEW_VALUE = Value
- val NODES = Value
- val NODE_LOCATION = Value
- val NORM = Value
- val NUM_BIN = Value
- val NUM_CLASSES = Value
- val NUM_COLUMNS = Value
- val NUM_EXAMPLES = Value
- val NUM_FEATURES = Value
- val NUM_FILES = Value
- val NUM_FREQUENT_ITEMS = Value
- val NUM_ITERATIONS = Value
- val NUM_LOCAL_FREQUENT_PATTERN = Value
- val NUM_PARTITION = Value
- val NUM_POINT = Value
- val NUM_PREFIXES = Value
- val NUM_SEQUENCES = Value
- val OBJECT_ID = Value
- val OFFSET = Value
- val OFFSETS = Value
- val OLD_BLOCK_MANAGER_ID = Value
- val OLD_VALUE = Value
- val OPTIMIZED_PLAN_COLUMNS = Value
- val OPTIMIZER_CLASS_NAME = Value
- val OPTIONS = Value
- val OP_ID = Value
- val OP_TYPE = Value
- val OVERHEAD_MEMORY_SIZE = Value
- val PARSE_MODE = Value
- val PARTITIONED_FILE_READER = Value
- val PARTITIONS_SIZE = Value
- val PARTITION_ID = Value
- val PARTITION_SPECIFICATION = Value
- val PARTITION_SPECS = Value
- val PATH = Value
- val PATHS = Value
- val PIPELINE_STAGE_UID = Value
- val POD_COUNT = Value
- val POD_ID = Value
- val POD_NAME = Value
- val POD_NAMESPACE = Value
- val POD_PHASE = Value
- val POD_SHARED_SLOT_COUNT = Value
- val POD_STATE = Value
- val POD_TARGET_COUNT = Value
- val POLICY = Value
- val PORT = Value
- val PRINCIPAL = Value
- val PROCESSING_TIME = Value
- val PRODUCER_ID = Value
- val PROVIDER = Value
- val PVC_METADATA_NAME = Value
- val QUERY_CACHE_VALUE = Value
- val QUERY_HINT = Value
- val QUERY_ID = Value
- val QUERY_PLAN = Value
- val QUERY_PLAN_COMPARISON = Value
- val QUERY_PLAN_LENGTH_ACTUAL = Value
- val QUERY_PLAN_LENGTH_MAX = Value
- val QUERY_RUN_ID = Value
- val RANGE = Value
- val RDD_ID = Value
- val READ_LIMIT = Value
- val REASON = Value
- val REATTACHABLE = Value
- val RECEIVED_BLOCK_INFO = Value
- val RECEIVED_BLOCK_TRACKER_LOG_EVENT = Value
- val RECEIVER_ID = Value
- val RECEIVER_IDS = Value
- val RECORDS = Value
- val REDACTED_STATEMENT = Value
- val REDUCE_ID = Value
- val RELATION_NAME = Value
- val RELATIVE_TOLERANCE = Value
- val REMAINING_PARTITIONS = Value
- val REPORT_DETAILS = Value
- val RESOURCE = Value
- val RESOURCE_NAME = Value
- val RESOURCE_PROFILE_ID = Value
- val RESOURCE_PROFILE_IDS = Value
- val RETRY_COUNT = Value
- val RETRY_INTERVAL = Value
- val RIGHT_EXPR = Value
- val RMSE = Value
- val RPC_ENDPOINT_REF = Value
- val RULE_BATCH_NAME = Value
- val RULE_NAME = Value
- val RULE_NUMBER_OF_RUNS = Value
- val RUN_ID = Value
- val SCHEMA = Value
- val SCHEMA2 = Value
- val SERVICE_NAME = Value
- val SESSION_HOLD_INFO = Value
- val SESSION_ID = Value
- val SESSION_KEY = Value
- val SHARD_ID = Value
- val SHUFFLE_BLOCK_INFO = Value
- val SHUFFLE_ID = Value
- val SHUFFLE_MERGE_ID = Value
- val SHUFFLE_SERVICE_NAME = Value
- val SIZE = Value
- val SLEEP_TIME = Value
- val SLIDE_DURATION = Value
- val SMALLEST_CLUSTER_INDEX = Value
- val SPARK_DATA_STREAM = Value
- val SPARK_PLAN_ID = Value
- val SQL_TEXT = Value
- val SRC_PATH = Value
- val STAGE_ID = Value
- val START_INDEX = Value
- val STATEMENT_ID = Value
- val STATE_STORE_PROVIDER = Value
- val STATUS = Value
- val STORAGE_LEVEL = Value
- val STORAGE_LEVEL_DESERIALIZED = Value
- val STORAGE_LEVEL_REPLICATION = Value
- val STORE_ID = Value
- val STREAM_ID = Value
- val STREAM_NAME = Value
- val SUBMISSION_ID = Value
- val SUBSAMPLING_RATE = Value
- val SUB_QUERY = Value
- val TABLE_NAME = Value
- val TABLE_TYPES = Value
- val TARGET_PATH = Value
- val TASK_ATTEMPT_ID = Value
- val TASK_ID = Value
- val TASK_NAME = Value
- val TASK_SET_NAME = Value
- val TASK_STATE = Value
- val TEMP_FILE = Value
- val TEMP_PATH = Value
- val TEST_SIZE = Value
- val THREAD = Value
- val THREAD_NAME = Value
- val TID = Value
- val TIME = Value
- val TIMEOUT = Value
- val TIMER = Value
- val TIMER_LABEL = Value
- val TIME_UNITS = Value
- val TIP = Value
- val TOKEN_REGEX = Value
- val TOPIC = Value
- val TOPIC_PARTITION = Value
- val TOPIC_PARTITIONS = Value
- val TOPIC_PARTITION_OFFSET = Value
- val TOPIC_PARTITION_OFFSET_RANGE = Value
- val TOTAL = Value
- val TOTAL_EFFECTIVE_TIME = Value
- val TOTAL_RECORDS_READ = Value
- val TOTAL_TIME = Value
- val TOTAL_TIME_READ = Value
- val TO_TIME = Value
- val TRAINING_SIZE = Value
- val TRAIN_VALIDATION_SPLIT_METRIC = Value
- val TRAIN_VALIDATION_SPLIT_METRICS = Value
- val TRAIN_WORD_COUNT = Value
- val TREE_NODE = Value
- val TRIGGER_INTERVAL = Value
- val UI_FILTER = Value
- val UI_FILTER_PARAMS = Value
- val UI_PROXY_BASE = Value
- val UNSUPPORTED_EXPR = Value
- val UNSUPPORTED_HINT_REASON = Value
- val UNTIL_OFFSET = Value
- val UPPER_BOUND = Value
- val URI = Value
- val USER_ID = Value
- val USER_NAME = Value
- val VALUE = Value
- val VIRTUAL_CORES = Value
- val VOCAB_SIZE = Value
- val WAIT_RESULT_TIME = Value
- val WAIT_SEND_TIME = Value
- val WAIT_TIME = Value
- val WATERMARK_CONSTRAINT = Value
- val WEIGHTED_NUM = Value
- val WORKER_URL = Value
- val WRITE_AHEAD_LOG_INFO = Value
- val XSD_PATH = Value
-
- type LogKey = Value
+object LogKey extends ILogKey {
Review Comment:
Yes, it is totally fine. I also tried it on my local. Looks good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47963][CORE] Make the external Spark ecosystem can use structured logging mechanisms [spark]
Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang commented on code in PR #46193:
URL: https://github.com/apache/spark/pull/46193#discussion_r1578608646
##########
common/utils/src/test/scala/org/apache/spark/util/StructuredLoggingSuite.scala:
##########
@@ -144,6 +146,22 @@ trait LoggingSuiteBase
}
}
+ // An enumeration value that is not defined in `LogKey`, you can define it anywhere
+ private val externalLogKey = LogKey.VALUE
+ private val externalLog = log"${MDC(externalLogKey, "External log message.")}"
+ test("Logging with external LogKey") {
Review Comment:
We should definitely allow external libraries to extend the log keys. But I don't have a good suggestion for this one now. Either making it as a trait or making the keys as strings are ok.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47963][CORE] Make the external Spark ecosystem can use structured logging mechanisms [spark]
Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang commented on code in PR #46193:
URL: https://github.com/apache/spark/pull/46193#discussion_r1578773809
##########
common/utils/src/main/scala/org/apache/spark/internal/README.md:
##########
@@ -2,7 +2,7 @@
## LogKey
-LogKeys serve as identifiers for mapped diagnostic contexts (MDC) within logs. Follow these guidelines when adding new LogKeys:
+LogKey serve as identifiers for mapped diagnostic contexts (MDC) within logs. Follow these guidelines when adding new LogKey:
Review Comment:
```suggestion
`LogKey`s serve as identifiers for mapped diagnostic contexts (MDC) within logs. Follow these guidelines when adding a new LogKey:
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47963][CORE] Make the external Spark ecosystem can use structured logging mechanisms [spark]
Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on code in PR #46193:
URL: https://github.com/apache/spark/pull/46193#discussion_r1578769212
##########
common/utils/src/test/scala/org/apache/spark/util/StructuredLoggingSuite.scala:
##########
@@ -144,6 +146,21 @@ trait LoggingSuiteBase
}
}
+ private val externalSystemCustomLog =
Review Comment:
Add a UT to test how external third-party can use `LogKey`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47963][CORE] Make the external Spark ecosystem can use structured logging mechanisms [spark]
Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang commented on code in PR #46193:
URL: https://github.com/apache/spark/pull/46193#discussion_r1578623991
##########
common/utils/src/main/scala/org/apache/spark/internal/LogKey.scala:
##########
@@ -16,393 +16,393 @@
*/
package org.apache.spark.internal
+trait ILogKey
Review Comment:
rename as LogKey?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47963][CORE] Make the external Spark ecosystem can use structured logging mechanisms [spark]
Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on code in PR #46193:
URL: https://github.com/apache/spark/pull/46193#discussion_r1578806516
##########
common/utils/src/main/scala/org/apache/spark/internal/LogKey.scala:
##########
@@ -16,399 +16,399 @@
*/
package org.apache.spark.internal
+trait LogKey
+
/**
* Various keys used for mapped diagnostic contexts(MDC) in logging.
* All structured logging keys should be defined here for standardization.
*/
-object LogKey extends Enumeration {
- val ACCUMULATOR_ID = Value
- val ACTUAL_NUM_FILES = Value
- val ACTUAL_PARTITION_COLUMN = Value
- val ALPHA = Value
- val ANALYSIS_ERROR = Value
- val APP_ATTEMPT_ID = Value
- val APP_DESC = Value
- val APP_ID = Value
- val APP_NAME = Value
- val APP_STATE = Value
- val ARGS = Value
- val BACKUP_FILE = Value
- val BATCH_ID = Value
- val BATCH_WRITE = Value
- val BLOCK_ID = Value
- val BLOCK_MANAGER_ID = Value
- val BROADCAST_ID = Value
- val BUCKET = Value
- val BYTECODE_SIZE = Value
- val CACHED_TABLE_PARTITION_METADATA_SIZE = Value
- val CACHE_AUTO_REMOVED_SIZE = Value
- val CACHE_UNTIL_HIGHEST_CONSUMED_SIZE = Value
- val CACHE_UNTIL_LAST_PRODUCED_SIZE = Value
- val CALL_SITE_LONG_FORM = Value
- val CATALOG_NAME = Value
- val CATEGORICAL_FEATURES = Value
- val CHECKPOINT_FILE = Value
- val CHECKPOINT_TIME = Value
- val CHECKSUM_FILE_NUM = Value
- val CLASS_LOADER = Value
- val CLASS_NAME = Value
- val CLUSTER_CENTROIDS = Value
- val CLUSTER_ID = Value
- val CLUSTER_LABEL = Value
- val CLUSTER_LEVEL = Value
- val CLUSTER_WEIGHT = Value
- val CODEC_LEVEL = Value
- val CODEC_NAME = Value
- val CODEGEN_STAGE_ID = Value
- val COLUMN_DATA_TYPE_SOURCE = Value
- val COLUMN_DATA_TYPE_TARGET = Value
- val COLUMN_DEFAULT_VALUE = Value
- val COLUMN_NAME = Value
- val COMMAND = Value
- val COMMAND_OUTPUT = Value
- val COMPONENT = Value
- val CONFIG = Value
- val CONFIG2 = Value
- val CONFIG3 = Value
- val CONFIG4 = Value
- val CONFIG5 = Value
- val CONSUMER = Value
- val CONTAINER = Value
- val CONTAINER_ID = Value
- val CONTAINER_STATE = Value
- val COST = Value
- val COUNT = Value
- val CROSS_VALIDATION_METRIC = Value
- val CROSS_VALIDATION_METRICS = Value
- val CSV_HEADER_COLUMN_NAME = Value
- val CSV_HEADER_COLUMN_NAMES = Value
- val CSV_HEADER_LENGTH = Value
- val CSV_SCHEMA_FIELD_NAME = Value
- val CSV_SCHEMA_FIELD_NAMES = Value
- val CSV_SOURCE = Value
- val CURRENT_PATH = Value
- val DATA = Value
- val DATABASE_NAME = Value
- val DATAFRAME_CACHE_ENTRY = Value
- val DATAFRAME_ID = Value
- val DATA_FILE_NUM = Value
- val DATA_SOURCE = Value
- val DATA_SOURCES = Value
- val DATA_SOURCE_PROVIDER = Value
- val DEFAULT_ISOLATION_LEVEL = Value
- val DEFAULT_VALUE = Value
- val DELAY = Value
- val DELEGATE = Value
- val DELTA = Value
- val DESCRIPTION = Value
- val DESIRED_PARTITIONS_SIZE = Value
- val DIFF_DELTA = Value
- val DIVISIBLE_CLUSTER_INDICES_SIZE = Value
- val DRIVER_ID = Value
- val DROPPED_PARTITIONS = Value
- val DURATION = Value
- val EFFECTIVE_STORAGE_LEVEL = Value
- val ELAPSED_TIME = Value
- val ENCODING = Value
- val END_INDEX = Value
- val END_POINT = Value
- val ENGINE = Value
- val ERROR = Value
- val ESTIMATOR_PARAMETER_MAP = Value
- val EVENT_LOOP = Value
- val EVENT_QUEUE = Value
- val EXCEPTION = Value
- val EXECUTE_INFO = Value
- val EXECUTE_KEY = Value
- val EXECUTION_PLAN_LEAVES = Value
- val EXECUTOR_DESIRED_COUNT = Value
- val EXECUTOR_ENVS = Value
- val EXECUTOR_ENV_REGEX = Value
- val EXECUTOR_ID = Value
- val EXECUTOR_IDS = Value
- val EXECUTOR_LAUNCH_COMMANDS = Value
- val EXECUTOR_LAUNCH_COUNT = Value
- val EXECUTOR_RESOURCES = Value
- val EXECUTOR_STATE = Value
- val EXECUTOR_TARGET_COUNT = Value
- val EXIT_CODE = Value
- val EXPECTED_NUM_FILES = Value
- val EXPECTED_PARTITION_COLUMN = Value
- val EXPIRY_TIMESTAMP = Value
- val EXPR = Value
- val EXPR_TERMS = Value
- val EXTENDED_EXPLAIN_GENERATOR = Value
- val FAILURES = Value
- val FALLBACK_VERSION = Value
- val FEATURE_COLUMN = Value
- val FEATURE_DIMENSION = Value
- val FIELD_NAME = Value
- val FILE_ABSOLUTE_PATH = Value
- val FILE_FORMAT = Value
- val FILE_FORMAT2 = Value
- val FILE_NAME = Value
- val FILE_VERSION = Value
- val FINISH_TRIGGER_DURATION = Value
- val FROM_OFFSET = Value
- val FROM_TIME = Value
- val FUNCTION_NAME = Value
- val FUNCTION_PARAMETER = Value
- val GROUP_ID = Value
- val HADOOP_VERSION = Value
- val HASH_JOIN_KEYS = Value
- val HEARTBEAT_INTERVAL = Value
- val HISTORY_DIR = Value
- val HIVE_CLIENT_VERSION = Value
- val HIVE_METASTORE_VERSION = Value
- val HIVE_OPERATION_STATE = Value
- val HIVE_OPERATION_TYPE = Value
- val HOST = Value
- val HOST_PORT = Value
- val IDENTIFIER = Value
- val INCOMPATIBLE_TYPES = Value
- val INDEX = Value
- val INDEX_FILE_NUM = Value
- val INDEX_NAME = Value
- val INFERENCE_MODE = Value
- val INITIAL_CAPACITY = Value
- val INITIAL_HEARTBEAT_INTERVAL = Value
- val INIT_MODE = Value
- val INTERVAL = Value
- val ISOLATION_LEVEL = Value
- val JOB_ID = Value
- val JOIN_CONDITION = Value
- val JOIN_CONDITION_SUB_EXPR = Value
- val K8S_CONTEXT = Value
- val KAFKA_PULLS_COUNT = Value
- val KAFKA_RECORDS_PULLED_COUNT = Value
- val KEY = Value
- val KEYTAB = Value
- val LABEL_COLUMN = Value
- val LARGEST_CLUSTER_INDEX = Value
- val LAST_ACCESS_TIME = Value
- val LAST_VALID_TIME = Value
- val LATEST_BATCH_ID = Value
- val LATEST_COMMITTED_BATCH_ID = Value
- val LEARNING_RATE = Value
- val LEFT_EXPR = Value
- val LINE = Value
- val LINE_NUM = Value
- val LISTENER = Value
- val LOAD_FACTOR = Value
- val LOAD_TIME = Value
- val LOGICAL_PLAN_COLUMNS = Value
- val LOGICAL_PLAN_LEAVES = Value
- val LOG_TYPE = Value
- val LOWER_BOUND = Value
- val MALFORMATTED_STIRNG = Value
- val MASTER_URL = Value
- val MAX_ATTEMPTS = Value
- val MAX_CACHE_UNTIL_HIGHEST_CONSUMED_SIZE = Value
- val MAX_CACHE_UNTIL_LAST_PRODUCED_SIZE = Value
- val MAX_CAPACITY = Value
- val MAX_CATEGORIES = Value
- val MAX_EXECUTOR_FAILURES = Value
- val MAX_FILE_VERSION = Value
- val MAX_MEMORY_SIZE = Value
- val MAX_PARTITIONS_SIZE = Value
- val MAX_SIZE = Value
- val MAX_TABLE_PARTITION_METADATA_SIZE = Value
- val MEMORY_SIZE = Value
- val MERGE_DIR_NAME = Value
- val MESSAGE = Value
- val METHOD_NAME = Value
- val MIN_FREQUENT_PATTERN_COUNT = Value
- val MIN_POINT_PER_CLUSTER = Value
- val MIN_SIZE = Value
- val MODEL_WEIGHTS = Value
- val NAMESPACE = Value
- val NEW_FEATURE_COLUMN_NAME = Value
- val NEW_LABEL_COLUMN_NAME = Value
- val NEW_PATH = Value
- val NEW_VALUE = Value
- val NODES = Value
- val NODE_LOCATION = Value
- val NORM = Value
- val NUM_BIN = Value
- val NUM_CLASSES = Value
- val NUM_COLUMNS = Value
- val NUM_EXAMPLES = Value
- val NUM_FEATURES = Value
- val NUM_FILES = Value
- val NUM_FREQUENT_ITEMS = Value
- val NUM_ITERATIONS = Value
- val NUM_LOCAL_FREQUENT_PATTERN = Value
- val NUM_PARTITION = Value
- val NUM_POINT = Value
- val NUM_PREFIXES = Value
- val NUM_SEQUENCES = Value
- val OBJECT_ID = Value
- val OFFSET = Value
- val OFFSETS = Value
- val OLD_BLOCK_MANAGER_ID = Value
- val OLD_VALUE = Value
- val OPTIMIZED_PLAN_COLUMNS = Value
- val OPTIMIZER_CLASS_NAME = Value
- val OPTIONS = Value
- val OP_ID = Value
- val OP_TYPE = Value
- val OVERHEAD_MEMORY_SIZE = Value
- val PARSE_MODE = Value
- val PARTITIONED_FILE_READER = Value
- val PARTITIONS_SIZE = Value
- val PARTITION_ID = Value
- val PARTITION_SPECIFICATION = Value
- val PARTITION_SPECS = Value
- val PATH = Value
- val PATHS = Value
- val PIPELINE_STAGE_UID = Value
- val POD_COUNT = Value
- val POD_ID = Value
- val POD_NAME = Value
- val POD_NAMESPACE = Value
- val POD_PHASE = Value
- val POD_SHARED_SLOT_COUNT = Value
- val POD_STATE = Value
- val POD_TARGET_COUNT = Value
- val POLICY = Value
- val PORT = Value
- val PRINCIPAL = Value
- val PROCESSING_TIME = Value
- val PRODUCER_ID = Value
- val PROVIDER = Value
- val PVC_METADATA_NAME = Value
- val QUERY_CACHE_VALUE = Value
- val QUERY_HINT = Value
- val QUERY_ID = Value
- val QUERY_PLAN = Value
- val QUERY_PLAN_COMPARISON = Value
- val QUERY_PLAN_LENGTH_ACTUAL = Value
- val QUERY_PLAN_LENGTH_MAX = Value
- val QUERY_RUN_ID = Value
- val RANGE = Value
- val RDD_ID = Value
- val READ_LIMIT = Value
- val REASON = Value
- val REATTACHABLE = Value
- val RECEIVED_BLOCK_INFO = Value
- val RECEIVED_BLOCK_TRACKER_LOG_EVENT = Value
- val RECEIVER_ID = Value
- val RECEIVER_IDS = Value
- val RECORDS = Value
- val REDACTED_STATEMENT = Value
- val REDUCE_ID = Value
- val RELATION_NAME = Value
- val RELATIVE_TOLERANCE = Value
- val REMAINING_PARTITIONS = Value
- val REPORT_DETAILS = Value
- val RESOURCE = Value
- val RESOURCE_NAME = Value
- val RESOURCE_PROFILE_ID = Value
- val RESOURCE_PROFILE_IDS = Value
- val RETRY_COUNT = Value
- val RETRY_INTERVAL = Value
- val RIGHT_EXPR = Value
- val RMSE = Value
- val RPC_ENDPOINT_REF = Value
- val RULE_BATCH_NAME = Value
- val RULE_NAME = Value
- val RULE_NUMBER_OF_RUNS = Value
- val RUN_ID = Value
- val SCHEMA = Value
- val SCHEMA2 = Value
- val SERVICE_NAME = Value
- val SESSION_HOLD_INFO = Value
- val SESSION_ID = Value
- val SESSION_KEY = Value
- val SHARD_ID = Value
- val SHUFFLE_BLOCK_INFO = Value
- val SHUFFLE_ID = Value
- val SHUFFLE_MERGE_ID = Value
- val SHUFFLE_SERVICE_NAME = Value
- val SIZE = Value
- val SLEEP_TIME = Value
- val SLIDE_DURATION = Value
- val SMALLEST_CLUSTER_INDEX = Value
- val SPARK_DATA_STREAM = Value
- val SPARK_PLAN_ID = Value
- val SQL_TEXT = Value
- val SRC_PATH = Value
- val STAGE_ATTEMPT = Value
- val STAGE_ID = Value
- val START_INDEX = Value
- val STATEMENT_ID = Value
- val STATE_STORE_PROVIDER = Value
- val STATUS = Value
- val STDERR = Value
- val STORAGE_LEVEL = Value
- val STORAGE_LEVEL_DESERIALIZED = Value
- val STORAGE_LEVEL_REPLICATION = Value
- val STORE_ID = Value
- val STREAM_ID = Value
- val STREAM_NAME = Value
- val SUBMISSION_ID = Value
- val SUBSAMPLING_RATE = Value
- val SUB_QUERY = Value
- val TABLE_NAME = Value
- val TABLE_TYPES = Value
- val TARGET_PATH = Value
- val TASK_ATTEMPT_ID = Value
- val TASK_ID = Value
- val TASK_NAME = Value
- val TASK_SET_NAME = Value
- val TASK_STATE = Value
- val TEMP_FILE = Value
- val TEMP_PATH = Value
- val TEST_SIZE = Value
- val THREAD = Value
- val THREAD_NAME = Value
- val TID = Value
- val TIME = Value
- val TIMEOUT = Value
- val TIMER = Value
- val TIMER_LABEL = Value
- val TIME_UNITS = Value
- val TIP = Value
- val TOKEN_REGEX = Value
- val TOPIC = Value
- val TOPIC_PARTITION = Value
- val TOPIC_PARTITIONS = Value
- val TOPIC_PARTITION_OFFSET = Value
- val TOPIC_PARTITION_OFFSET_RANGE = Value
- val TOTAL = Value
- val TOTAL_EFFECTIVE_TIME = Value
- val TOTAL_RECORDS_READ = Value
- val TOTAL_TIME = Value
- val TOTAL_TIME_READ = Value
- val TO_TIME = Value
- val TRAINING_SIZE = Value
- val TRAIN_VALIDATION_SPLIT_METRIC = Value
- val TRAIN_VALIDATION_SPLIT_METRICS = Value
- val TRAIN_WORD_COUNT = Value
- val TREE_NODE = Value
- val TRIGGER_INTERVAL = Value
- val UI_FILTER = Value
- val UI_FILTER_PARAMS = Value
- val UI_PROXY_BASE = Value
- val UNSUPPORTED_EXPR = Value
- val UNSUPPORTED_HINT_REASON = Value
- val UNTIL_OFFSET = Value
- val UPPER_BOUND = Value
- val URI = Value
- val USER_ID = Value
- val USER_NAME = Value
- val VALUE = Value
- val VIRTUAL_CORES = Value
- val VOCAB_SIZE = Value
- val WAIT_RESULT_TIME = Value
- val WAIT_SEND_TIME = Value
- val WAIT_TIME = Value
- val WATERMARK_CONSTRAINT = Value
- val WEIGHTED_NUM = Value
- val WORKER_URL = Value
- val WRITE_AHEAD_LOG_INFO = Value
- val WRITE_JOB_UUID = Value
- val XSD_PATH = Value
-
- type LogKey = Value
+object LogKeys {
+ case object ACCUMULATOR_ID extends LogKey
+ case object ACTUAL_NUM_FILES extends LogKey
+ case object ACTUAL_PARTITION_COLUMN extends LogKey
+ case object ALPHA extends LogKey
+ case object ANALYSIS_ERROR extends LogKey
+ case object APP_ATTEMPT_ID extends LogKey
+ case object APP_DESC extends LogKey
+ case object APP_ID extends LogKey
+ case object APP_NAME extends LogKey
+ case object APP_STATE extends LogKey
+ case object ARGS extends LogKey
+ case object BACKUP_FILE extends LogKey
+ case object BATCH_ID extends LogKey
+ case object BATCH_WRITE extends LogKey
+ case object BLOCK_ID extends LogKey
+ case object BLOCK_MANAGER_ID extends LogKey
+ case object BROADCAST_ID extends LogKey
+ case object BUCKET extends LogKey
+ case object BYTECODE_SIZE extends LogKey
+ case object CACHED_TABLE_PARTITION_METADATA_SIZE extends LogKey
+ case object CACHE_AUTO_REMOVED_SIZE extends LogKey
+ case object CACHE_UNTIL_HIGHEST_CONSUMED_SIZE extends LogKey
+ case object CACHE_UNTIL_LAST_PRODUCED_SIZE extends LogKey
+ case object CALL_SITE_LONG_FORM extends LogKey
+ case object CATALOG_NAME extends LogKey
+ case object CATEGORICAL_FEATURES extends LogKey
+ case object CHECKPOINT_FILE extends LogKey
+ case object CHECKPOINT_TIME extends LogKey
+ case object CHECKSUM_FILE_NUM extends LogKey
+ case object CLASS_LOADER extends LogKey
+ case object CLASS_NAME extends LogKey
+ case object CLUSTER_CENTROIDS extends LogKey
+ case object CLUSTER_ID extends LogKey
+ case object CLUSTER_LABEL extends LogKey
+ case object CLUSTER_LEVEL extends LogKey
+ case object CLUSTER_WEIGHT extends LogKey
+ case object CODEC_LEVEL extends LogKey
+ case object CODEC_NAME extends LogKey
+ case object CODEGEN_STAGE_ID extends LogKey
+ case object COLUMN_DATA_TYPE_SOURCE extends LogKey
+ case object COLUMN_DATA_TYPE_TARGET extends LogKey
+ case object COLUMN_DEFAULT_VALUE extends LogKey
+ case object COLUMN_NAME extends LogKey
+ case object COMMAND extends LogKey
+ case object COMMAND_OUTPUT extends LogKey
+ case object COMPONENT extends LogKey
+ case object CONFIG extends LogKey
+ case object CONFIG2 extends LogKey
+ case object CONFIG3 extends LogKey
+ case object CONFIG4 extends LogKey
+ case object CONFIG5 extends LogKey
+ case object CONSUMER extends LogKey
+ case object CONTAINER extends LogKey
+ case object CONTAINER_ID extends LogKey
+ case object CONTAINER_STATE extends LogKey
+ case object COST extends LogKey
+ case object COUNT extends LogKey
+ case object CROSS_VALIDATION_METRIC extends LogKey
+ case object CROSS_VALIDATION_METRICS extends LogKey
+ case object CSV_HEADER_COLUMN_NAME extends LogKey
+ case object CSV_HEADER_COLUMN_NAMES extends LogKey
+ case object CSV_HEADER_LENGTH extends LogKey
+ case object CSV_SCHEMA_FIELD_NAME extends LogKey
+ case object CSV_SCHEMA_FIELD_NAMES extends LogKey
+ case object CSV_SOURCE extends LogKey
+ case object CURRENT_PATH extends LogKey
+ case object DATA extends LogKey
+ case object DATABASE_NAME extends LogKey
+ case object DATAFRAME_CACHE_ENTRY extends LogKey
+ case object DATAFRAME_ID extends LogKey
+ case object DATA_FILE_NUM extends LogKey
+ case object DATA_SOURCE extends LogKey
+ case object DATA_SOURCES extends LogKey
+ case object DATA_SOURCE_PROVIDER extends LogKey
+ case object DEFAULT_ISOLATION_LEVEL extends LogKey
+ case object DEFAULT_VALUE extends LogKey
+ case object DELAY extends LogKey
+ case object DELEGATE extends LogKey
+ case object DELTA extends LogKey
+ case object DESCRIPTION extends LogKey
+ case object DESIRED_PARTITIONS_SIZE extends LogKey
+ case object DIFF_DELTA extends LogKey
+ case object DIVISIBLE_CLUSTER_INDICES_SIZE extends LogKey
+ case object DRIVER_ID extends LogKey
+ case object DROPPED_PARTITIONS extends LogKey
+ case object DURATION extends LogKey
+ case object EFFECTIVE_STORAGE_LEVEL extends LogKey
+ case object ELAPSED_TIME extends LogKey
+ case object ENCODING extends LogKey
+ case object END_INDEX extends LogKey
+ case object END_POINT extends LogKey
+ case object ENGINE extends LogKey
+ case object ERROR extends LogKey
+ case object ESTIMATOR_PARAMETER_MAP extends LogKey
+ case object EVENT_LOOP extends LogKey
+ case object EVENT_QUEUE extends LogKey
+ case object EXCEPTION extends LogKey
+ case object EXECUTE_INFO extends LogKey
+ case object EXECUTE_KEY extends LogKey
+ case object EXECUTION_PLAN_LEAVES extends LogKey
+ case object EXECUTOR_DESIRED_COUNT extends LogKey
+ case object EXECUTOR_ENVS extends LogKey
+ case object EXECUTOR_ENV_REGEX extends LogKey
+ case object EXECUTOR_ID extends LogKey
+ case object EXECUTOR_IDS extends LogKey
+ case object EXECUTOR_LAUNCH_COMMANDS extends LogKey
+ case object EXECUTOR_LAUNCH_COUNT extends LogKey
+ case object EXECUTOR_RESOURCES extends LogKey
+ case object EXECUTOR_STATE extends LogKey
+ case object EXECUTOR_TARGET_COUNT extends LogKey
+ case object EXIT_CODE extends LogKey
+ case object EXPECTED_NUM_FILES extends LogKey
+ case object EXPECTED_PARTITION_COLUMN extends LogKey
+ case object EXPIRY_TIMESTAMP extends LogKey
+ case object EXPR extends LogKey
+ case object EXPR_TERMS extends LogKey
+ case object EXTENDED_EXPLAIN_GENERATOR extends LogKey
+ case object FAILURES extends LogKey
+ case object FALLBACK_VERSION extends LogKey
+ case object FEATURE_COLUMN extends LogKey
+ case object FEATURE_DIMENSION extends LogKey
+ case object FIELD_NAME extends LogKey
+ case object FILE_ABSOLUTE_PATH extends LogKey
+ case object FILE_FORMAT extends LogKey
+ case object FILE_FORMAT2 extends LogKey
+ case object FILE_NAME extends LogKey
+ case object FILE_VERSION extends LogKey
+ case object FINISH_TRIGGER_DURATION extends LogKey
+ case object FROM_OFFSET extends LogKey
+ case object FROM_TIME extends LogKey
+ case object FUNCTION_NAME extends LogKey
+ case object FUNCTION_PARAMETER extends LogKey
+ case object GROUP_ID extends LogKey
+ case object HADOOP_VERSION extends LogKey
+ case object HASH_JOIN_KEYS extends LogKey
+ case object HEARTBEAT_INTERVAL extends LogKey
+ case object HISTORY_DIR extends LogKey
+ case object HIVE_CLIENT_VERSION extends LogKey
+ case object HIVE_METASTORE_VERSION extends LogKey
+ case object HIVE_OPERATION_STATE extends LogKey
+ case object HIVE_OPERATION_TYPE extends LogKey
+ case object HOST extends LogKey
+ case object HOST_PORT extends LogKey
+ case object IDENTIFIER extends LogKey
+ case object INCOMPATIBLE_TYPES extends LogKey
+ case object INDEX extends LogKey
+ case object INDEX_FILE_NUM extends LogKey
+ case object INDEX_NAME extends LogKey
+ case object INFERENCE_MODE extends LogKey
+ case object INITIAL_CAPACITY extends LogKey
+ case object INITIAL_HEARTBEAT_INTERVAL extends LogKey
+ case object INIT_MODE extends LogKey
+ case object INTERVAL extends LogKey
+ case object ISOLATION_LEVEL extends LogKey
+ case object JOB_ID extends LogKey
+ case object JOIN_CONDITION extends LogKey
+ case object JOIN_CONDITION_SUB_EXPR extends LogKey
+ case object K8S_CONTEXT extends LogKey
+ case object KAFKA_PULLS_COUNT extends LogKey
+ case object KAFKA_RECORDS_PULLED_COUNT extends LogKey
+ case object KEY extends LogKey
+ case object KEYTAB extends LogKey
+ case object LABEL_COLUMN extends LogKey
+ case object LARGEST_CLUSTER_INDEX extends LogKey
+ case object LAST_ACCESS_TIME extends LogKey
+ case object LAST_VALID_TIME extends LogKey
+ case object LATEST_BATCH_ID extends LogKey
+ case object LATEST_COMMITTED_BATCH_ID extends LogKey
+ case object LEARNING_RATE extends LogKey
+ case object LEFT_EXPR extends LogKey
+ case object LINE extends LogKey
+ case object LINE_NUM extends LogKey
+ case object LISTENER extends LogKey
+ case object LOAD_FACTOR extends LogKey
+ case object LOAD_TIME extends LogKey
+ case object LOGICAL_PLAN_COLUMNS extends LogKey
+ case object LOGICAL_PLAN_LEAVES extends LogKey
+ case object LOG_TYPE extends LogKey
+ case object LOWER_BOUND extends LogKey
+ case object MALFORMATTED_STIRNG extends LogKey
+ case object MASTER_URL extends LogKey
+ case object MAX_ATTEMPTS extends LogKey
+ case object MAX_CACHE_UNTIL_HIGHEST_CONSUMED_SIZE extends LogKey
+ case object MAX_CACHE_UNTIL_LAST_PRODUCED_SIZE extends LogKey
+ case object MAX_CAPACITY extends LogKey
+ case object MAX_CATEGORIES extends LogKey
+ case object MAX_EXECUTOR_FAILURES extends LogKey
+ case object MAX_FILE_VERSION extends LogKey
+ case object MAX_MEMORY_SIZE extends LogKey
+ case object MAX_PARTITIONS_SIZE extends LogKey
+ case object MAX_SIZE extends LogKey
+ case object MAX_TABLE_PARTITION_METADATA_SIZE extends LogKey
+ case object MEMORY_SIZE extends LogKey
+ case object MERGE_DIR_NAME extends LogKey
+ case object MESSAGE extends LogKey
+ case object METHOD_NAME extends LogKey
+ case object MIN_FREQUENT_PATTERN_COUNT extends LogKey
+ case object MIN_POINT_PER_CLUSTER extends LogKey
+ case object MIN_SIZE extends LogKey
+ case object MODEL_WEIGHTS extends LogKey
+ case object NAMESPACE extends LogKey
+ case object NEW_FEATURE_COLUMN_NAME extends LogKey
+ case object NEW_LABEL_COLUMN_NAME extends LogKey
+ case object NEW_PATH extends LogKey
+ case object NEW_VALUE extends LogKey
+ case object NODES extends LogKey
+ case object NODE_LOCATION extends LogKey
+ case object NORM extends LogKey
+ case object NUM_BIN extends LogKey
+ case object NUM_CLASSES extends LogKey
+ case object NUM_COLUMNS extends LogKey
+ case object NUM_EXAMPLES extends LogKey
+ case object NUM_FEATURES extends LogKey
+ case object NUM_FILES extends LogKey
+ case object NUM_FREQUENT_ITEMS extends LogKey
+ case object NUM_ITERATIONS extends LogKey
+ case object NUM_LOCAL_FREQUENT_PATTERN extends LogKey
+ case object NUM_PARTITION extends LogKey
+ case object NUM_POINT extends LogKey
+ case object NUM_PREFIXES extends LogKey
+ case object NUM_SEQUENCES extends LogKey
+ case object OBJECT_ID extends LogKey
+ case object OFFSET extends LogKey
+ case object OFFSETS extends LogKey
+ case object OLD_BLOCK_MANAGER_ID extends LogKey
+ case object OLD_VALUE extends LogKey
+ case object OPTIMIZED_PLAN_COLUMNS extends LogKey
+ case object OPTIMIZER_CLASS_NAME extends LogKey
+ case object OPTIONS extends LogKey
+ case object OP_ID extends LogKey
+ case object OP_TYPE extends LogKey
+ case object OVERHEAD_MEMORY_SIZE extends LogKey
+ case object PARSE_MODE extends LogKey
+ case object PARTITIONED_FILE_READER extends LogKey
+ case object PARTITIONS_SIZE extends LogKey
+ case object PARTITION_ID extends LogKey
+ case object PARTITION_SPECIFICATION extends LogKey
+ case object PARTITION_SPECS extends LogKey
+ case object PATH extends LogKey
+ case object PATHS extends LogKey
+ case object PIPELINE_STAGE_UID extends LogKey
+ case object POD_COUNT extends LogKey
+ case object POD_ID extends LogKey
+ case object POD_NAME extends LogKey
+ case object POD_NAMESPACE extends LogKey
+ case object POD_PHASE extends LogKey
+ case object POD_SHARED_SLOT_COUNT extends LogKey
+ case object POD_STATE extends LogKey
+ case object POD_TARGET_COUNT extends LogKey
+ case object POLICY extends LogKey
+ case object PORT extends LogKey
+ case object PRINCIPAL extends LogKey
+ case object PROCESSING_TIME extends LogKey
+ case object PRODUCER_ID extends LogKey
+ case object PROVIDER extends LogKey
+ case object PVC_METADATA_NAME extends LogKey
+ case object QUERY_CACHE_VALUE extends LogKey
+ case object QUERY_HINT extends LogKey
+ case object QUERY_ID extends LogKey
+ case object QUERY_PLAN extends LogKey
+ case object QUERY_PLAN_COMPARISON extends LogKey
+ case object QUERY_PLAN_LENGTH_ACTUAL extends LogKey
+ case object QUERY_PLAN_LENGTH_MAX extends LogKey
+ case object QUERY_RUN_ID extends LogKey
+ case object RANGE extends LogKey
+ case object RDD_ID extends LogKey
+ case object READ_LIMIT extends LogKey
+ case object REASON extends LogKey
+ case object REATTACHABLE extends LogKey
+ case object RECEIVED_BLOCK_INFO extends LogKey
+ case object RECEIVED_BLOCK_TRACKER_LOG_EVENT extends LogKey
+ case object RECEIVER_ID extends LogKey
+ case object RECEIVER_IDS extends LogKey
+ case object RECORDS extends LogKey
+ case object REDACTED_STATEMENT extends LogKey
+ case object REDUCE_ID extends LogKey
+ case object RELATION_NAME extends LogKey
+ case object RELATIVE_TOLERANCE extends LogKey
+ case object REMAINING_PARTITIONS extends LogKey
+ case object REPORT_DETAILS extends LogKey
+ case object RESOURCE extends LogKey
+ case object RESOURCE_NAME extends LogKey
+ case object RESOURCE_PROFILE_ID extends LogKey
+ case object RESOURCE_PROFILE_IDS extends LogKey
+ case object RETRY_COUNT extends LogKey
+ case object RETRY_INTERVAL extends LogKey
+ case object RIGHT_EXPR extends LogKey
+ case object RMSE extends LogKey
+ case object RPC_ENDPOINT_REF extends LogKey
+ case object RULE_BATCH_NAME extends LogKey
+ case object RULE_NAME extends LogKey
+ case object RULE_NUMBER_OF_RUNS extends LogKey
+ case object RUN_ID extends LogKey
+ case object SCHEMA extends LogKey
+ case object SCHEMA2 extends LogKey
+ case object SERVICE_NAME extends LogKey
+ case object SESSION_HOLD_INFO extends LogKey
+ case object SESSION_ID extends LogKey
+ case object SESSION_KEY extends LogKey
+ case object SHARD_ID extends LogKey
+ case object SHUFFLE_BLOCK_INFO extends LogKey
+ case object SHUFFLE_ID extends LogKey
+ case object SHUFFLE_MERGE_ID extends LogKey
+ case object SHUFFLE_SERVICE_NAME extends LogKey
+ case object SIZE extends LogKey
+ case object SLEEP_TIME extends LogKey
+ case object SLIDE_DURATION extends LogKey
+ case object SMALLEST_CLUSTER_INDEX extends LogKey
+ case object SPARK_DATA_STREAM extends LogKey
+ case object SPARK_PLAN_ID extends LogKey
+ case object SQL_TEXT extends LogKey
+ case object SRC_PATH extends LogKey
+ case object STAGE_ATTEMPT extends LogKey
+ case object STAGE_ID extends LogKey
+ case object START_INDEX extends LogKey
+ case object STATEMENT_ID extends LogKey
+ case object STATE_STORE_PROVIDER extends LogKey
+ case object STATUS extends LogKey
+ case object STDERR extends LogKey
+ case object STORAGE_LEVEL extends LogKey
+ case object STORAGE_LEVEL_DESERIALIZED extends LogKey
+ case object STORAGE_LEVEL_REPLICATION extends LogKey
+ case object STORE_ID extends LogKey
+ case object STREAM_ID extends LogKey
+ case object STREAM_NAME extends LogKey
+ case object SUBMISSION_ID extends LogKey
+ case object SUBSAMPLING_RATE extends LogKey
+ case object SUB_QUERY extends LogKey
+ case object TABLE_NAME extends LogKey
+ case object TABLE_TYPES extends LogKey
+ case object TARGET_PATH extends LogKey
+ case object TASK_ATTEMPT_ID extends LogKey
+ case object TASK_ID extends LogKey
+ case object TASK_NAME extends LogKey
+ case object TASK_SET_NAME extends LogKey
+ case object TASK_STATE extends LogKey
+ case object TEMP_FILE extends LogKey
+ case object TEMP_PATH extends LogKey
+ case object TEST_SIZE extends LogKey
+ case object THREAD extends LogKey
+ case object THREAD_NAME extends LogKey
+ case object TID extends LogKey
+ case object TIME extends LogKey
+ case object TIMEOUT extends LogKey
+ case object TIMER extends LogKey
+ case object TIMER_LABEL extends LogKey
Review Comment:
Let me remove it.
When all pr for the structured logs is completed, I will also submit a PR to `improve` the names of this class, such as `NUM-FILES` and `WEIGHTED-NUM`. Should `NUM` be used as the prefix or suffix? We need to unify. @gengliangwang @LuciferYang
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47963][CORE][TESTS] Add an external LogKey usage case in UT [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #46193:
URL: https://github.com/apache/spark/pull/46193#discussion_r1577159936
##########
common/utils/src/test/scala/org/apache/spark/util/StructuredLoggingSuite.scala:
##########
@@ -144,6 +146,22 @@ trait LoggingSuiteBase
}
}
+ // An enumeration value that is not defined in `LogKey`, you can define it anywhere
+ private val externalLogKey = LogKey.VALUE
+ private val externalLog = log"${MDC(externalLogKey, "External log message.")}"
+ test("Logging with external LogKey") {
Review Comment:
For this case, the key for `externalLog.context` will be always ~an empty string~ "value". Is this expected?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47963][CORE] Make the external Spark ecosystem can use structured logging mechanisms [spark]
Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on PR #46193:
URL: https://github.com/apache/spark/pull/46193#issuecomment-2078352745
> @panbingkun Sorry, I have to merge #46192 since it was created before this one. Please resolve the conflicts and I will merge this one. Thanks as always.
Don't worry, I have solved it, 😄
Let's run GA first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org