You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2022/06/08 05:21:06 UTC
[GitHub] [ozone] szetszwo opened a new pull request, #3492: HDDS-6842. [Ozone-Streaming] Reduce watch requests in StreamCommitWatcher.
szetszwo opened a new pull request, #3492:
URL: https://github.com/apache/ozone/pull/3492
See https://issues.apache.org/jira/browse/HDDS-6842
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] captainzmc commented on pull request #3492: HDDS-6842. [Ozone-Streaming] Reduce watch requests in StreamCommitWatcher.
Posted by GitBox <gi...@apache.org>.
captainzmc commented on PR #3492:
URL: https://github.com/apache/ozone/pull/3492#issuecomment-1149862571
> @captainzmc , BTW, I moved some of the changes to #3493 so that we can merge it directly to the master branch.
OK, Let's start with #3493.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] szetszwo commented on pull request #3492: HDDS-6842. [Ozone-Streaming] Reduce the number of watch requests in StreamCommitWatcher.
Posted by GitBox <gi...@apache.org>.
szetszwo commented on PR #3492:
URL: https://github.com/apache/ozone/pull/3492#issuecomment-1165863332
TYPE could be ASYNC, STREAM_API_MAPPED_BYTE_BUFFER or STREAM_API_BYTE_ARRAY.
TYPE=STREAM will use STREAM_API_MAPPED_BYTE_BUFFER.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] szetszwo commented on pull request #3492: HDDS-6842. [Ozone-Streaming] Reduce the number of watch requests in StreamCommitWatcher.
Posted by GitBox <gi...@apache.org>.
szetszwo commented on PR #3492:
URL: https://github.com/apache/ozone/pull/3492#issuecomment-1163446878
@captainzmc , tried some single client benchmarks. Streaming is obvious better than Async after this and the related fixes.
- ASYNC[100 x 128m]
```
2022-06-22 17:55:26 INFO Print:53 - ASYNC[100 x 128m]: ELAPSED: 36789ms (36.789253S)
2022-06-22 17:57:06 INFO Print:53 - ASYNC[100 x 128m]: ELAPSED: 34980ms (34.980303S)
2022-06-22 17:52:30 INFO Print:53 - ASYNC[100 x 128m]: ELAPSED: 40386ms (40.386711S)
```
- STREAM[100 x 128m]
```
2022-06-22 17:50:15 INFO Print:53 - STREAM[100 x 128m]: ELAPSED: 15367ms (15.367493S)
2022-06-22 17:50:58 INFO Print:53 - STREAM[100 x 128m]: ELAPSED: 22082ms (22.082032S)
2022-06-22 17:51:33 INFO Print:53 - STREAM[100 x 128m]: ELAPSED: 20748ms (20.748417S)
```
- ASYNC[10 x 1g]
```
2022-06-22 16:03:12 INFO Print:53 - ASYNC[10 x 1g]: ELAPSED: 41268ms (41.268702S)
2022-06-22 16:04:31 INFO Print:53 - ASYNC[10 x 1g]: ELAPSED: 42623ms (42.623741S)
2022-06-22 16:06:12 INFO Print:53 - ASYNC[10 x 1g]: ELAPSED: 39027ms (39.027346S)
```
- STREAM[10 x 1g]
```
2022-06-22 16:01:19 INFO Print:53 - STREAM[10 x 1g]: ELAPSED: 20486ms (20.486308S)
2022-06-22 16:07:34 INFO Print:53 - STREAM[10 x 1g]: ELAPSED: 26253ms (26.253218S)
2022-06-22 16:08:35 INFO Print:53 - STREAM[10 x 1g]: ELAPSED: 19151ms (19.151242S)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] captainzmc commented on pull request #3492: HDDS-6842. [Ozone-Streaming] Reduce the number of watch requests in StreamCommitWatcher.
Posted by GitBox <gi...@apache.org>.
captainzmc commented on PR #3492:
URL: https://github.com/apache/ozone/pull/3492#issuecomment-1152019922
> Let's fix it separately. Do you know how to fix it?
Let me try to fix these.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] szetszwo commented on pull request #3492: HDDS-6842. [Ozone-Streaming] Reduce the number of watch requests in StreamCommitWatcher.
Posted by GitBox <gi...@apache.org>.
szetszwo commented on PR #3492:
URL: https://github.com/apache/ozone/pull/3492#issuecomment-1163566443
- 200x128m_ASYNC
```
2022-06-22 20:09:58 INFO Print:58 - 200x128m_ASYNC_c8m_t32: ELAPSED: 50301ms (50.301573S)
2022-06-22 20:11:32 INFO Print:58 - 200x128m_ASYNC_c8m_t32: ELAPSED: 49688ms (49.688644S)
2022-06-22 20:12:53 INFO Print:58 - 200x128m_ASYNC_c8m_t32: ELAPSED: 50884ms (50.884893S)
2022-06-22 20:15:28 INFO Print:58 - 200x128m_ASYNC_c8m_t64: ELAPSED: 49258ms (49.258734S)
```
- 200x128m_STREAM
```
2022-06-22 20:01:26 INFO Print:58 - 200x128m_STREAM_c8m_t32: ELAPSED: 28995ms (28.995231S)
2022-06-22 20:02:58 INFO Print:58 - 200x128m_STREAM_c8m_t32: ELAPSED: 22288ms (22.288911S)
2022-06-22 20:04:11 INFO Print:58 - 200x128m_STREAM_c8m_t32: ELAPSED: 26967ms (26.967777S)
2022-06-22 20:22:01 INFO Print:58 - 200x128m_STREAM_c8m_t64: ELAPSED: 28768ms (28.768423S)
```
1. Note that c8m means 8mb chunk size. t32 means 32 client threads.
2. Note also that using more client threads (t64) does not help.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] szetszwo merged pull request #3492: HDDS-6842. [Ozone-Streaming] Reduce the number of watch requests in StreamCommitWatcher.
Posted by GitBox <gi...@apache.org>.
szetszwo merged PR #3492:
URL: https://github.com/apache/ozone/pull/3492
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] szetszwo commented on pull request #3492: HDDS-6842. [Ozone-Streaming] Reduce the number of watch requests in StreamCommitWatcher.
Posted by GitBox <gi...@apache.org>.
szetszwo commented on PR #3492:
URL: https://github.com/apache/ozone/pull/3492#issuecomment-1165051150
2022-06-23 20:54:00 INFO Print:58 - 3clients_100x128m_ASYNC_c8m_t32: ELAPSED: 72004ms (1M12.004621S)
2022-06-23 20:58:30 INFO Print:55 - 3clients_100x128m_ASYNC_c8m_t32: ELAPSED: 75607ms (1M15.607768S)
2022-06-23 21:00:42 INFO Print:55 - 3clients_100x128m_ASYNC_c8m_t32: ELAPSED: 72097ms (1M12.097986S)
2022-06-23 20:49:58 INFO Print:58 - 3clients_100x128m_STREAM_c8m_t32: ELAPSED: 22666ms (22.666303S)
2022-06-23 21:02:01 INFO Print:55 - 3clients_100x128m_STREAM_c8m_t32: ELAPSED: 24922ms (24.922359S)
2022-06-23 21:03:58 INFO Print:55 - 3clients_100x128m_STREAM_c8m_t32: ELAPSED: 22948ms (22.948317S)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] szetszwo commented on pull request #3492: HDDS-6842. [Ozone-Streaming] Reduce the number of watch requests in StreamCommitWatcher.
Posted by GitBox <gi...@apache.org>.
szetszwo commented on PR #3492:
URL: https://github.com/apache/ozone/pull/3492#issuecomment-1164977011
Did some 3-client benchmarks:
- 3clients_200x128m_ASYNC: client07
```
2022-06-23 21:20:06 INFO Print:55 - 3clients_200x128m_ASYNC_c8m_t32: ELAPSED: 142303ms (2M22.303595S)
2022-06-23 21:23:37 INFO Print:55 - 3clients_200x128m_ASYNC_c8m_t32: ELAPSED: 145289ms (2M25.28923S)
2022-06-23 21:28:52 INFO Print:55 - 3clients_200x128m_ASYNC_c8m_t32: ELAPSED: 153096ms (2M33.096604S)
```
- 3clients_200x128m_STREAM: client07
```
2022-06-23 21:07:02 INFO Print:55 - 3clients_200x128m_STREAM_c8m_t32: ELAPSED: 41521ms (41.521613S)
2022-06-23 21:11:45 INFO Print:55 - 3clients_200x128m_STREAM_c8m_t32: ELAPSED: 41287ms (41.287784S)
2022-06-23 21:14:54 INFO Print:55 - 3clients_200x128m_STREAM_c8m_t32: ELAPSED: 40074ms (40.074274S)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] szetszwo commented on pull request #3492: HDDS-6842. [Ozone-Streaming] Reduce the number of watch requests in StreamCommitWatcher.
Posted by GitBox <gi...@apache.org>.
szetszwo commented on PR #3492:
URL: https://github.com/apache/ozone/pull/3492#issuecomment-1151584268
- STREAM[1 x 1g]
```
2022-06-09 20:10:50 INFO Print:53 - STREAM[1 x 1g]: ELAPSED: 4962ms (4.962496S)
2022-06-09 20:11:49 INFO Print:53 - STREAM[1 x 1g]: ELAPSED: 5289ms (5.289901S)
2022-06-09 20:12:35 INFO Print:53 - STREAM[1 x 1g]: ELAPSED: 5354ms (5.354376S)
2022-06-09 20:13:59 INFO Print:53 - STREAM[1 x 1g]: ELAPSED: 5257ms (5.257939S)
2022-06-09 20:14:39 INFO Print:53 - STREAM[1 x 1g]: ELAPSED: 4844ms (4.844022S)
```
- ASYNC[1 x 1g]
```
2022-06-09 20:15:04 INFO Print:53 - ASYNC[1 x 1g]: ELAPSED: 5208ms (5.208552S)
2022-06-09 20:16:04 INFO Print:53 - ASYNC[1 x 1g]: ELAPSED: 4805ms (4.805783S)
2022-06-09 20:16:24 INFO Print:53 - ASYNC[1 x 1g]: ELAPSED: 5201ms (5.201009S)
2022-06-09 20:17:13 INFO Print:53 - ASYNC[1 x 1g]: ELAPSED: 5105ms (5.105652S)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] szetszwo commented on pull request #3492: HDDS-6842. [Ozone-Streaming] Reduce the number of watch requests in StreamCommitWatcher.
Posted by GitBox <gi...@apache.org>.
szetszwo commented on PR #3492:
URL: https://github.com/apache/ozone/pull/3492#issuecomment-1165858017
@captainzmc , this is the benchmark program https://github.com/szetszwo/ozone-benchmark which is forked for your program and this is the conf https://github.com/szetszwo/ozone-benchmark/blob/master/benchmark-conf.xml
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] szetszwo commented on pull request #3492: HDDS-6842. [Ozone-Streaming] Reduce the number of watch requests in StreamCommitWatcher.
Posted by GitBox <gi...@apache.org>.
szetszwo commented on PR #3492:
URL: https://github.com/apache/ozone/pull/3492#issuecomment-1164977563
- 3clients_200x128m_ASYNC: client08
```
2022-06-23 21:20:06 INFO Print:55 - 3clients_200x128m_ASYNC_c8m_t32: ELAPSED: 142551ms (2M22.551907S)
2022-06-23 21:23:37 INFO Print:55 - 3clients_200x128m_ASYNC_c8m_t32: ELAPSED: 145216ms (2M25.216375S)
2022-06-23 21:28:51 INFO Print:55 - 3clients_200x128m_ASYNC_c8m_t32: ELAPSED: 152550ms (2M32.550649S)
```
- 3clients_200x128m_STREAM: client08
```
2022-06-23 21:07:04 INFO Print:55 - 3clients_200x128m_STREAM_c8m_t32: ELAPSED: 42897ms (42.897611S)
2022-06-23 21:11:45 INFO Print:55 - 3clients_200x128m_STREAM_c8m_t32: ELAPSED: 41408ms (41.408407S)
2022-06-23 21:14:54 INFO Print:55 - 3clients_200x128m_STREAM_c8m_t32: ELAPSED: 40609ms (40.609904S)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] szetszwo commented on pull request #3492: HDDS-6842. [Ozone-Streaming] Reduce watch requests in StreamCommitWatcher.
Posted by GitBox <gi...@apache.org>.
szetszwo commented on PR #3492:
URL: https://github.com/apache/ozone/pull/3492#issuecomment-1149820975
@captainzmc , I was mainly testing writing a single 1GB file. Before this improvement, Async took ~6s but Streaming took ~9s in my setup. After this Streaming is about the same as Async.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] szetszwo commented on pull request #3492: HDDS-6842. [Ozone-Streaming] Reduce the number of watch requests in StreamCommitWatcher.
Posted by GitBox <gi...@apache.org>.
szetszwo commented on PR #3492:
URL: https://github.com/apache/ozone/pull/3492#issuecomment-1164978082
- 3clients_200x128m_ASYNC: client09
```
2022-06-23 21:20:06 INFO Print:55 - 3clients_200x128m_ASYNC_c8m_t32: ELAPSED: 142575ms (2M22.575039S)
2022-06-23 21:23:37 INFO Print:55 - 3clients_200x128m_ASYNC_c8m_t32: ELAPSED: 145177ms (2M25.177483S)
2022-06-23 21:28:52 INFO Print:55 - 3clients_200x128m_ASYNC_c8m_t32: ELAPSED: 152991ms (2M32.991775S)
```
- 3clients_200x128m_STREAM: client09
```
2022-06-23 21:07:04 INFO Print:55 - 3clients_200x128m_STREAM_c8m_t32: ELAPSED: 43379ms (43.379795S)
2022-06-23 21:11:46 INFO Print:55 - 3clients_200x128m_STREAM_c8m_t32: ELAPSED: 42202ms (42.202172S)
2022-06-23 21:14:55 INFO Print:55 - 3clients_200x128m_STREAM_c8m_t32: ELAPSED: 40797ms (40.797086S)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] szetszwo commented on pull request #3492: HDDS-6842. [Ozone-Streaming] Reduce the number of watch requests in StreamCommitWatcher.
Posted by GitBox <gi...@apache.org>.
szetszwo commented on PR #3492:
URL: https://github.com/apache/ozone/pull/3492#issuecomment-1164796775
@captainzmc , Stream is faster than Async by more than a double below.
- 800x128m_ASYNC
```
2022-06-23 19:18:56 INFO Print:58 - 800x128m_ASYNC_c8m_t32: ELAPSED: 192891ms (3M12.891844S)
2022-06-23 19:24:07 INFO Print:58 - 800x128m_ASYNC_c8m_t32: ELAPSED: 198278ms (3M18.278792S)
2022-06-23 19:30:04 INFO Print:58 - 800x128m_ASYNC_c8m_t32: ELAPSED: 201877ms (3M21.877562S)
```
- 800x128m_STREAM
```
2022-06-23 19:13:49 INFO Print:58 - 800x128m_STREAM_c8m_t32: ELAPSED: 70130ms (1M10.130834S)
2022-06-23 19:32:59 INFO Print:58 - 800x128m_STREAM_c8m_t32: ELAPSED: 77290ms (1M17.290496S)
2022-06-23 19:35:38 INFO Print:58 - 800x128m_STREAM_c8m_t32: ELAPSED: 77012ms (1M17.012314S)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] captainzmc commented on pull request #3492: HDDS-6842. [Ozone-Streaming] Reduce the number of watch requests in StreamCommitWatcher.
Posted by GitBox <gi...@apache.org>.
captainzmc commented on PR #3492:
URL: https://github.com/apache/ozone/pull/3492#issuecomment-1165154046
Hi @szetszwo, Apart from chunk size, is there any other change of the streaming configuration?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] szetszwo commented on pull request #3492: HDDS-6842. [Ozone-Streaming] Reduce the number of watch requests in StreamCommitWatcher.
Posted by GitBox <gi...@apache.org>.
szetszwo commented on PR #3492:
URL: https://github.com/apache/ozone/pull/3492#issuecomment-1165860716
```
cd $BENCHMARK_HOME
export CLASSPATH=`mvn -s $MVN_SETTINGS exec:exec | grep ${PWD}/`
OM=...
CLIENTS=...
BENCHMARK_ID=`hostname`
TYPE=STREAM
FILE_NUM=200
FILE_SIZE=128m
CHUNK_SIZE=8m
THREAD_NUM=32
DATETIME=`date +%Y%m%d-%H%M`; \
F=output-3clients_${FILE_NUM}x${FILE_SIZE}_${TYPE}_c${CHUNK_SIZE}_t${THREAD_NUM}_${DATETIME}.txt; \
java -Xms64g -Xmx64g org.apache.hadoop.ozone.benchmark.Benchmark \
-om ${OM} -localDirs ${LOCAL_ROOT}/benchmark-dir/ \
-verify \
-type ${TYPE} -id ${BENCHMARK_ID} -threadNum ${THREAD_NUM} \
-fileNum ${FILE_NUM} -fileSize ${FILE_SIZE} -chunkSize ${CHUNK_SIZE} \
-clients ${CLIENTS} \
| tee ${F}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] szetszwo commented on pull request #3492: HDDS-6842. [Ozone-Streaming] Reduce the number of watch requests in StreamCommitWatcher.
Posted by GitBox <gi...@apache.org>.
szetszwo commented on PR #3492:
URL: https://github.com/apache/ozone/pull/3492#issuecomment-1163603562
- 400x128m_ASYNC
```
2022-06-22 20:39:41 INFO Print:58 - 400x128m_ASYNC_c8m_t32: ELAPSED: 96096ms (1M36.096616S)
2022-06-22 20:42:02 INFO Print:58 - 400x128m_ASYNC_c8m_t32: ELAPSED: 91303ms (1M31.30335S)
2022-06-22 20:45:00 INFO Print:58 - 400x128m_ASYNC_c8m_t32: ELAPSED: 91450ms (1M31.450918S)
2022-06-22 20:47:41 INFO Print:58 - 400x128m_ASYNC_c8m_t32: ELAPSED: 93955ms (1M33.955319S)
2022-06-22 20:53:07 INFO Print:58 - 400x128m_ASYNC_c1m_t32: ELAPSED: 95263ms (1M35.263701S)
```
- 400x128m_STREAM
```
2022-06-22 20:33:59 INFO Print:58 - 400x128m_STREAM_c8m_t32: ELAPSED: 45900ms (45.900112S)
2022-06-22 20:35:48 INFO Print:58 - 400x128m_STREAM_c8m_t32: ELAPSED: 48339ms (48.33955S)
2022-06-22 20:37:21 INFO Print:58 - 400x128m_STREAM_c8m_t32: ELAPSED: 48380ms (48.380559S)
2022-06-22 21:03:29 INFO Print:58 - 400x128m_STREAM_c8m_t32: ELAPSED: 45124ms (45.124395S)
2022-06-22 20:59:01 INFO Print:58 - 400x128m_STREAM_c1m_t32: ELAPSED: 68551ms (1M8.55105S)
```
3. When using 1mb chunk size (c1m), the Async performance is about the same as 8mb chunk size but the Stream is much slower.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] szetszwo commented on pull request #3492: HDDS-6842. [Ozone-Streaming] Reduce the number of watch requests in StreamCommitWatcher.
Posted by GitBox <gi...@apache.org>.
szetszwo commented on PR #3492:
URL: https://github.com/apache/ozone/pull/3492#issuecomment-1151577742
@captainzmc , just have verified that the performance of Stream and Async are about the same for writing a single 1GB file. Please review this. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] captainzmc commented on pull request #3492: HDDS-6842. [Ozone-Streaming] Reduce the number of watch requests in StreamCommitWatcher.
Posted by GitBox <gi...@apache.org>.
captainzmc commented on PR #3492:
URL: https://github.com/apache/ozone/pull/3492#issuecomment-1164021879
Thanks @szetszwo for the test, the streaming performance is very good from the test results, which can meet our expectations. It seems that the chunk size setting is important.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] szetszwo commented on pull request #3492: HDDS-6842. [Ozone-Streaming] Reduce watch requests in StreamCommitWatcher.
Posted by GitBox <gi...@apache.org>.
szetszwo commented on PR #3492:
URL: https://github.com/apache/ozone/pull/3492#issuecomment-1149824809
@captainzmc , BTW, I moved some of the changes to #3493 so that we can merge it directly to the master branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] captainzmc commented on pull request #3492: HDDS-6842. [Ozone-Streaming] Reduce the number of watch requests in StreamCommitWatcher.
Posted by GitBox <gi...@apache.org>.
captainzmc commented on PR #3492:
URL: https://github.com/apache/ozone/pull/3492#issuecomment-1166770452
Thanks @szetszwo . These are useful. Later we can collate these test data and the test process so that others can understand the streaming performance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] szetszwo commented on pull request #3492: HDDS-6842. [Ozone-Streaming] Reduce the number of watch requests in StreamCommitWatcher.
Posted by GitBox <gi...@apache.org>.
szetszwo commented on PR #3492:
URL: https://github.com/apache/ozone/pull/3492#issuecomment-1150621245
Will run some benchmark again in order to verify the improvement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] szetszwo commented on pull request #3492: HDDS-6842. [Ozone-Streaming] Reduce the number of watch requests in StreamCommitWatcher.
Posted by GitBox <gi...@apache.org>.
szetszwo commented on PR #3492:
URL: https://github.com/apache/ozone/pull/3492#issuecomment-1152016504
@captainzmc , thanks a lot for reviewing this.
> CI has some errors. However, these errors are not caused by this PR, ...
Let's fix it separately. Do you know how to fix it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] captainzmc commented on pull request #3492: HDDS-6842. [Ozone-Streaming] Reduce the number of watch requests in StreamCommitWatcher.
Posted by GitBox <gi...@apache.org>.
captainzmc commented on PR #3492:
URL: https://github.com/apache/ozone/pull/3492#issuecomment-1165142959
Thanks @szetszwo，Great progress！We will no longer worry about streaming being challenged when merging master. I and @guohao-rosicky will continue the development of HDDS-5869 in order to complete streaming functionality as soon as possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org