You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bluemarlin.apache.org by GitBox <gi...@apache.org> on 2022/01/06 21:14:17 UTC
[GitHub] [incubator-bluemarlin] radibnia77 opened a new issue #26: dlpredictor cannot process huge log files
radibnia77 opened a new issue #26:
URL: https://github.com/apache/incubator-bluemarlin/issues/26
If you have a different process for dlpredictor to process huge log files, please add it to
/Processes/dlpredictor/experiments/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bluemarlin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-bluemarlin] radibnia77 commented on issue #26: dlpredictor cannot process huge log files
Posted by GitBox <gi...@apache.org>.
radibnia77 commented on issue #26:
URL: https://github.com/apache/incubator-bluemarlin/issues/26#issuecomment-1009289738
I ran your version of the post-process (prediction) code.
This result is equivalent to the original code. I don't see any issue here.
```
+--------------------------------+---------+
|si |sum(impr)|
+--------------------------------+---------+
|d4d7362e879511e5bdec00163e291137|45321 |
|17dd6d8098bf11e5bdec00163e291137|54459 |
|s4z85pd1h8 |589565 |
|j1430itab9wj3b |726600 |
|a8syykhszz |797981 |
|k4werqx13k |3509653 |
|x2fpfbm8rt |4300115 |
|d9jucwkpr3 |4801039 |
|5cd1c663263511e6af7500163e291137|5392623 |
|z041bf6g4s |5514397 |
|71bcd2720e5011e79bc8fa163e05184e|9680103 |
|d971z9825e |21667294 |
|l2d4ec6csv |27452169 |
|w9fmyd5r0i |31180311 |
|w3wx3nv9ow5i97 |32777583 |
|f1iprgyl13 |37138436 |
|68bcd2720e5011e79bc8fa163e05184e|46111387 |
|a290af82884e11e5bdec00163e291137|64487036 |
|e351de37263311e6af7500163e291137|71461386 |
|l03493p0r3 |97185399 |
|7b0d7b55ab0c11e68b7900163e3e481d|97269075 |
|b6le0s4qo8 |113705360|
|66bcd2720e5011e79bc8fa163e05184e|139935126|
|x0ej5xhk60kjwq |159131471|
|a47eavw7ex |666862239|
+--------------------------------+---------+
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bluemarlin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-bluemarlin] radibnia77 closed issue #26: dlpredictor cannot process huge log files
Posted by GitBox <gi...@apache.org>.
radibnia77 closed issue #26:
URL: https://github.com/apache/incubator-bluemarlin/issues/26
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bluemarlin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-bluemarlin] radibnia77 commented on issue #26: dlpredictor cannot process huge log files
Posted by GitBox <gi...@apache.org>.
radibnia77 commented on issue #26:
URL: https://github.com/apache/incubator-bluemarlin/issues/26#issuecomment-1007854232
I am reviewing/running your code.
I noticed that line 56 and line 140 contradict each other in main_spark_data_process, you might need to look into that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bluemarlin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-bluemarlin] radibnia77 commented on issue #26: dlpredictor cannot process huge log files
Posted by GitBox <gi...@apache.org>.
radibnia77 commented on issue #26:
URL: https://github.com/apache/incubator-bluemarlin/issues/26#issuecomment-1007854232
I am reviewing/running your code.
I noticed that line 56 and line 140 contradict each other in main_spark_data_process, you might need to look into that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bluemarlin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-bluemarlin] rangaswamymr commented on issue #26: dlpredictor cannot process huge log files
Posted by GitBox <gi...@apache.org>.
rangaswamymr commented on issue #26:
URL: https://github.com/apache/incubator-bluemarlin/issues/26#issuecomment-1007875039
Line 140 is not there in final code we are using. It was added for some testing and i forgot to remove before sharing with you.
Also, i have removed few lines related to integrations. if the code doesn't work, you may have to change according to your integration mechanism with ES and model serving system.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bluemarlin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-bluemarlin] radibnia77 edited a comment on issue #26: dlpredictor cannot process huge log files
Posted by GitBox <gi...@apache.org>.
radibnia77 edited a comment on issue #26:
URL: https://github.com/apache/incubator-bluemarlin/issues/26#issuecomment-1009289738
I ran your version of the post-process (prediction) code.
This result is equivalent to the original code. I don't see any issue here.
The sum(impr) is for 9 days.
```
+--------------------------------+---------+
|si |sum(impr)|
+--------------------------------+---------+
|d4d7362e879511e5bdec00163e291137|45321 |
|17dd6d8098bf11e5bdec00163e291137|54459 |
|s4z85pd1h8 |589565 |
|j1430itab9wj3b |726600 |
|a8syykhszz |797981 |
|k4werqx13k |3509653 |
|x2fpfbm8rt |4300115 |
|d9jucwkpr3 |4801039 |
|5cd1c663263511e6af7500163e291137|5392623 |
|z041bf6g4s |5514397 |
|71bcd2720e5011e79bc8fa163e05184e|9680103 |
|d971z9825e |21667294 |
|l2d4ec6csv |27452169 |
|w9fmyd5r0i |31180311 |
|w3wx3nv9ow5i97 |32777583 |
|f1iprgyl13 |37138436 |
|68bcd2720e5011e79bc8fa163e05184e|46111387 |
|a290af82884e11e5bdec00163e291137|64487036 |
|e351de37263311e6af7500163e291137|71461386 |
|l03493p0r3 |97185399 |
|7b0d7b55ab0c11e68b7900163e3e481d|97269075 |
|b6le0s4qo8 |113705360|
|66bcd2720e5011e79bc8fa163e05184e|139935126|
|x0ej5xhk60kjwq |159131471|
|a47eavw7ex |666862239|
+--------------------------------+---------+
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bluemarlin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-bluemarlin] rangaswamymr commented on issue #26: dlpredictor cannot process huge log files
Posted by GitBox <gi...@apache.org>.
rangaswamymr commented on issue #26:
URL: https://github.com/apache/incubator-bluemarlin/issues/26#issuecomment-1007482751
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bluemarlin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-bluemarlin] rangaswamymr commented on issue #26: dlpredictor cannot process huge log files
Posted by GitBox <gi...@apache.org>.
rangaswamymr commented on issue #26:
URL: https://github.com/apache/incubator-bluemarlin/issues/26#issuecomment-1007482751
@radibnia77 I was having issues in processing large data of logs. made few changes to post processing scripts and updated code files are committed and pull request is created.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bluemarlin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-bluemarlin] radibnia77 edited a comment on issue #26: dlpredictor cannot process huge log files
Posted by GitBox <gi...@apache.org>.
radibnia77 edited a comment on issue #26:
URL: https://github.com/apache/incubator-bluemarlin/issues/26#issuecomment-1009289738
I ran your version of the post-process (prediction) code.
This result is equivalent to the original code. I don't see any issue here.
```
9 days
+--------------------------------+---------+
|si |sum(impr)|
+--------------------------------+---------+
|d4d7362e879511e5bdec00163e291137|45321 |
|17dd6d8098bf11e5bdec00163e291137|54459 |
|s4z85pd1h8 |589565 |
|j1430itab9wj3b |726600 |
|a8syykhszz |797981 |
|k4werqx13k |3509653 |
|x2fpfbm8rt |4300115 |
|d9jucwkpr3 |4801039 |
|5cd1c663263511e6af7500163e291137|5392623 |
|z041bf6g4s |5514397 |
|71bcd2720e5011e79bc8fa163e05184e|9680103 |
|d971z9825e |21667294 |
|l2d4ec6csv |27452169 |
|w9fmyd5r0i |31180311 |
|w3wx3nv9ow5i97 |32777583 |
|f1iprgyl13 |37138436 |
|68bcd2720e5011e79bc8fa163e05184e|46111387 |
|a290af82884e11e5bdec00163e291137|64487036 |
|e351de37263311e6af7500163e291137|71461386 |
|l03493p0r3 |97185399 |
|7b0d7b55ab0c11e68b7900163e3e481d|97269075 |
|b6le0s4qo8 |113705360|
|66bcd2720e5011e79bc8fa163e05184e|139935126|
|x0ej5xhk60kjwq |159131471|
|a47eavw7ex |666862239|
+--------------------------------+---------+
10 days
+--------------------------------+---------+
|si |sum(impr)|
+--------------------------------+---------+
|d4d7362e879511e5bdec00163e291137|50456 |
|17dd6d8098bf11e5bdec00163e291137|60510 |
|s4z85pd1h8 |652759 |
|j1430itab9wj3b |810858 |
|a8syykhszz |884783 |
|k4werqx13k |3875218 |
|x2fpfbm8rt |4761138 |
|d9jucwkpr3 |5361277 |
|5cd1c663263511e6af7500163e291137|6029947 |
|z041bf6g4s |6069176 |
|71bcd2720e5011e79bc8fa163e05184e|10839372 |
|d971z9825e |24394780 |
|l2d4ec6csv |31009705 |
|w9fmyd5r0i |34697410 |
|w3wx3nv9ow5i97 |36511714 |
|f1iprgyl13 |42004396 |
|68bcd2720e5011e79bc8fa163e05184e|51639283 |
|a290af82884e11e5bdec00163e291137|72495755 |
|e351de37263311e6af7500163e291137|80392701 |
|l03493p0r3 |107668983|
|7b0d7b55ab0c11e68b7900163e3e481d|108882986|
|b6le0s4qo8 |125738313|
|66bcd2720e5011e79bc8fa163e05184e|155659693|
|x0ej5xhk60kjwq |177651970|
|a47eavw7ex |742013720|
+--------------------------------+---------+
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@bluemarlin.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org