You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2020/08/03 14:54:00 UTC
[GitHub] [hudi] tooptoop4 opened a new issue #1906: [SUPPORT] org.apache.hudi.common.util.FSUtils logging too verbose
tooptoop4 opened a new issue #1906:
URL: https://github.com/apache/hudi/issues/1906
for one simple spark-submit (COW data source) org.apache.hudi.common.util.FSUtils is logged on 21 lines, each line being 2700+ bytes
example of one line
2020-08-03 14:19:52,892 [main] INFO org.apache.hudi.common.util.FSUtils - Hadoop Configuration: fs.defaultFS: [file:///], Config:[Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml, __spark_hadoop_conf__.xml, file:/home/ec2-user/spark_home/conf/hive-site.xml], FileSystem: [S3AFileSystem{uri=s3a://mybucketo, workingDir=s3a://mybucketo/user/ec2-user, inputPolicy=normal, partSize=104857600, enableMultiObjectsDelete=true, maxKeys=5000, readAhead=65536, blockSize=33554432, multiPartThreshold=2147483647, serverSideEncryptionAlgorithm='AES256', blockFactory=org.apache.hadoop.fs.s3a.S3ADataBlocks$DiskBlockFactory@c247b02, boundedExecutor=BlockingThreadPoolExecutorService{SemaphoredDelegatingExecutor{permitCount=2405, available=2405, waiting=0}, activeCount=0}, unboundedExecutor=java.util.concurrent.ThreadPoolExecutor@78f1d29[Running, pool size = 1, active threads = 0, queued tasks = 0, comp
leted tasks = 1], statistics {11286 bytes read, 4304 bytes written, 160 read ops, 0 large read ops, 19 write ops}, metrics {{Context=S3AFileSystem} {FileSystemId=8624b3cf-eff7-4713-8405-95fc9bd35d90-mybucketo} {fsURI=s3a://mybucketo/private/sparkevents} {files_created=6} {files_copied=0} {files_copied_bytes=0} {files_deleted=1} {fake_directories_deleted=0} {directories_created=1} {directories_deleted=0} {ignored_errors=5} {op_copy_from_local_file=0} {op_exists=44} {op_get_file_status=138} {op_glob_status=0} {op_is_directory=36} {op_is_file=0} {op_list_files=1} {op_list_located_status=0} {op_list_status=21} {op_mkdirs=0} {op_rename=0} {object_copy_requests=0} {object_delete_requests=6} {object_list_requests=124} {object_continue_list_requests=0} {object_metadata_requests=240} {object_multipart_aborted=0} {object_put_bytes=4304} {object_put_requests=6} {object_put_requests_completed=6} {stream_write_failures=0} {stream_write_block_uploads=0} {stream_write_block_uploads_committed=0} {s
tream_write_block_uploads_aborted=0} {stream_write_total_time=0} {stream_write_total_data=4304} {object_put_requests_active=0} {object_put_bytes_pending=0} {stream_write_block_uploads_active=0} {stream_write_block_uploads_pending=5} {stream_write_block_uploads_data_pending=0} {stream_read_fully_operations=0} {stream_opened=22} {stream_bytes_skipped_on_seek=0} {stream_closed=22} {stream_bytes_backwards_on_seek=0} {stream_bytes_read=11286} {stream_read_operations_incomplete=22} {stream_bytes_discarded_in_abort=0} {stream_close_operations=22} {stream_read_operations=29} {stream_aborted=0} {stream_forward_seek_operations=0} {stream_backward_seek_operations=0} {stream_seek_operations=0} {stream_bytes_read_in_close=0} {stream_read_exceptions=0} }}]
some ideas:
1. "Hadoop Configuration" block be removed or split to separate log class
2. make it DEBUG level?
3. only print metrics that are non-0 ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar closed issue #1906: [SUPPORT] org.apache.hudi.common.util.FSUtils logging too verbose
Posted by GitBox <gi...@apache.org>.
bvaradar closed issue #1906:
URL: https://github.com/apache/hudi/issues/1906
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar commented on issue #1906: [SUPPORT] org.apache.hudi.common.util.FSUtils logging too verbose
Posted by GitBox <gi...@apache.org>.
bvaradar commented on issue #1906:
URL: https://github.com/apache/hudi/issues/1906#issuecomment-668644790
Added https://issues.apache.org/jira/browse/HUDI-1148
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org