You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@unomi.apache.org by "Kevan Jahanshahi (Jira)" <ji...@apache.org> on 2023/03/29 16:41:00 UTC
[jira] [Comment Edited] (UNOMI-758) Bunch of fixes and improvements for Profile Export system
[ https://issues.apache.org/jira/browse/UNOMI-758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17706508#comment-17706508 ]
Kevan Jahanshahi edited comment on UNOMI-758 at 3/29/23 4:40 PM:
-----------------------------------------------------------------
h1. Truncated CSV export to FTP investigations:
useful set camel FTP operations to debug:
{code:java}
log:set TRACE org.apache.camel.component.file.remote.FtpOperations {code}
This give logs like:
{noformat}
18:36:37.655 INFO [pool-21-thread-1] Configure Recurrent Route 'Export :: Collect Data'
18:36:37.683 INFO [pool-21-thread-1] Route: 82f5e3b0-67d2-4a8a-afd6-eda36f478485 started and consuming from: timer://collectProfile?fixedRate=true&period=1h
18:36:40.590 INFO [Camel (camel-1) thread #4 - timer://collectProfile] Processing route 82f5e3b0-67d2-4a8a-afd6-eda36f478485 completed.
18:36:40.612 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] Connecting using FTPClient: org.apache.commons.net.ftp.FTPClient@6ddfb396
18:36:40.613 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] Connecting to ftp://kevan@192.168.1.16:21 using connection timeout: 10000
18:36:40.656 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] Using passive mode connections
18:36:40.657 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] Using SoTimeout=300000
18:36:40.657 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] Attempting to login user: kevan using password: ********
18:36:41.294 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] User kevan logged in: true
18:36:41.298 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] buildDirectory(data3TO/trash)
18:36:41.304 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] changeCurrentDirectory(/)
18:36:41.305 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] Changing directory: /
18:36:53.156 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] storeFile(data3TO/trash/profiles-export-2023032918.csv)
18:36:53.157 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] getCurrentDirectory()
18:36:53.170 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] Current dir: /
18:36:53.171 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] changeCurrentDirectory(data3TO/trash)
18:36:53.171 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] Changing directory: data3TO
18:36:53.176 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] Changing directory: trash
18:36:55.827 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] doStoreFile(profiles-export-2023032918.csv)
18:36:55.828 DEBUG [Camel (camel-1) thread #4 - timer://collectProfile] About to store file: profiles-export-2023032918.csv using stream: java.io.ByteArrayInputStream@3cc24cfa
18:36:55.829 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] Client appendFile: profiles-export-2023032918.csv
18:36:55.858 DEBUG [Camel (camel-1) thread #4 - timer://collectProfile] Took 0.030 seconds (30 millis) to store file: profiles-export-2023032918.csv and FTP client returned: true
18:36:55.859 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] changeCurrentDirectory(/)
18:36:55.860 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] Changing directory: /{noformat}
was (Author: jkevan):
h1. Truncated CSV export to FTP investigations:
useful set camel FTP operations to debug:
{code:java}
log:set TRACE org.apache.camel.component.file.remote.FtpOperations {code}
This give logs like:
{noformat}
18:36:37.655 INFO [pool-21-thread-1] Configure Recurrent Route 'Export :: Collect Data'
18:36:37.683 INFO [pool-21-thread-1] Route: 82f5e3b0-67d2-4a8a-afd6-eda36f478485 started and consuming from: timer://collectProfile?fixedRate=true&period=1h
18:36:40.590 INFO [Camel (camel-1) thread #4 - timer://collectProfile] Processing route 82f5e3b0-67d2-4a8a-afd6-eda36f478485 completed.
18:36:40.612 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] Connecting using FTPClient: org.apache.commons.net.ftp.FTPClient@6ddfb396
18:36:40.613 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] Connecting to ftp://kevan@192.168.1.16:21 using connection timeout: 10000
18:36:40.656 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] Using passive mode connections
18:36:40.657 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] Using SoTimeout=300000
18:36:40.657 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] Attempting to login user: kevan using password: ********
18:36:41.294 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] User kevan logged in: true
18:36:41.298 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] buildDirectory(data3TO/trash)
18:36:41.304 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] changeCurrentDirectory(/)
18:36:41.305 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] Changing directory: /
18:36:53.156 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] storeFile(data3TO/trash/profiles-export-2023032918.csv)
18:36:53.157 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] getCurrentDirectory()
18:36:53.170 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] Current dir: /
18:36:53.171 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] changeCurrentDirectory(data3TO/trash)
18:36:53.171 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] Changing directory: data3TO
18:36:53.176 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] Changing directory: trash
18:36:55.827 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] doStoreFile(profiles-export-2023032918.csv)
18:36:55.828 DEBUG [Camel (camel-1) thread #4 - timer://collectProfile] About to store file: profiles-export-2023032918.csv using stream: java.io.ByteArrayInputStream@3cc24cfa
18:36:55.829 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] Client appendFile: profiles-export-2023032918.csv
18:36:55.858 DEBUG [Camel (camel-1) thread #4 - timer://collectProfile] Took 0.030 seconds (30 millis) to store file: profiles-export-2023032918.csv and FTP client returned: true
18:36:55.859 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] changeCurrentDirectory(/)
18:36:55.860 TRACE [Camel (camel-1) thread #4 - timer://collectProfile] Changing directory: /{noformat}
> Bunch of fixes and improvements for Profile Export system
> ---------------------------------------------------------
>
> Key: UNOMI-758
> URL: https://issues.apache.org/jira/browse/UNOMI-758
> Project: Apache Unomi
> Issue Type: Bug
> Reporter: David Griffon
> Assignee: David Griffon
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> h2. Issue 1: make the camel route update asynchron to avoid blocking the requests until camel route is killed.
> When a route is updated, we stop all execution of the current route.
> This stop can take time, and can lead to a request timeout.
> The goal of this task is to stop the execution asynchronously.
> h2. Issue 2: provide clear error messages in case something fail during camel route creation/update
> currently we are hiding the Exceptions:
> inside the class {*}org.apache.unomi.router.services.ExportConfigurationServiceImpl.java{*}, there is no stack trace:
> {noformat}
> 2022-10-26T16:15:56,667 | ERROR | qtp824972552-99936 | ExportConfigurationServiceImpl | 183 - org.apache.unomi.router-service - 1.4.4.jahia | Error when trying to save/update running Apache Camel Route: d9fa4556-a967-4410-aa1f-869ad66dbda5 {noformat}
> h2. Issue 3: investigate on truncated CSV export to FTP
> Sometimes we have CSV file that is truncated when exporting to FTP.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)