You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@streampipes.apache.org by udeho <ud...@student.kit.edu> on 2020/12/03 10:01:55 UTC
Re: [BUG] Accessing Images from Data Lake Identifiers with
Underscores
Hi Johannes,
I would also assume that an encoding with backslashes is the easiest solution.
Maybe using IDs instead of encoding the file route could be an option to avoid encoding issues.
But I do not know whether this can be done without to much effort.
Best,
Tim
On Nov. 28 2020, at 10:24 am, Johannes Tex <te...@apache.org> wrote:
> Hello, Tim,
>
> You are right. We encode some chars of the image path location when we save the image in the Datalake [1]. This was necessary that the can images via an http route [2].
> I think the easiest way would be to change the encoding. Maybe we should encode with '%2F', this is the encoding for slashes. But I am not sure if we can still access the images with [2]. Do you have another idea? The best solution would be a solution where the encoding can be removed.
> Johannes
> [1] https://github.com/apache/incubator-streampipes-extensions/blob/dev/streampipes-sinks-internal-jvm/src/main/java/org/apache/streampipes/sinks/internal/jvm/datalake/DataLake.java
> [2] https://github.com/apache/incubator-streampipes/blob/dev/streampipes-rest/src/main/java/org/apache/streampipes/rest/impl/datalake/DataLakeResourceV3.java
>
> On 2020/11/24 14:24:21, udeho <ud...@student.kit.edu> wrote:
> > Hi all,
> >
> > For a small object detection use case I have stored some images in the internal data lake to label them with the labeling tool.
> > I discovered that the use of underscores in the data lake identifier prevents the user from accessing the images later in the data explorer and the labeling tool.
> > As far as I can see, the underscore is already used as a delimiter in the http-request, which leads to a conflict if the identifier also contains an underscore.
> >
> > Best regards,
> > Tim
> >
> >
>
Re: RE: [BUG] Accessing Images from Data Lake Identifiers with Underscores
Posted by Johannes Tex <te...@apache.org>.
Hi,
yes, another and better possibility would be to use IDs. In this case we would have to store the path and ID together so that we can access the image again.
To read the image, the response time will increase (database look up), but I don't think that will be a problem.
I opened a ticket for that [1].
Johannes
[1] https://issues.apache.org/jira/browse/STREAMPIPES-265
On 2020/12/03 10:07:07, "Dominik Riemer" <ri...@apache.org> wrote:
> Hi,
>
> I'd also prefer something which works with an ID.
> Some application servers disallow encoded slashes by default, for the backend, we need to override this (see ALLOW_ENCODED_SLASH) in [1].
> Just in case you ever spend hours finding out why your encoded slash is not working 😉
>
> Dominik
>
> [1] https://github.com/apache/incubator-streampipes/blob/dev/streampipes-backend/src/main/java/org/apache/streampipes/backend/StreamPipesBackendApplication.java
>
>
> -----Original Message-----
> From: udeho <ud...@student.kit.edu>
> Sent: Thursday, December 3, 2020 11:02 AM
> To: dev@streampipes.apache.org
> Subject: Re: [BUG] Accessing Images from Data Lake Identifiers with Underscores
>
> Hi Johannes,
>
> I would also assume that an encoding with backslashes is the easiest solution.
> Maybe using IDs instead of encoding the file route could be an option to avoid encoding issues.
> But I do not know whether this can be done without to much effort.
>
> Best,
> Tim
>
> On Nov. 28 2020, at 10:24 am, Johannes Tex <te...@apache.org> wrote:
> > Hello, Tim,
> >
> > You are right. We encode some chars of the image path location when we save the image in the Datalake [1]. This was necessary that the can images via an http route [2].
> > I think the easiest way would be to change the encoding. Maybe we should encode with '%2F', this is the encoding for slashes. But I am not sure if we can still access the images with [2]. Do you have another idea? The best solution would be a solution where the encoding can be removed.
> > Johannes
> > [1] https://github.com/apache/incubator-streampipes-extensions/blob/dev/streampipes-sinks-internal-jvm/src/main/java/org/apache/streampipes/sinks/internal/jvm/datalake/DataLake.java
> > [2] https://github.com/apache/incubator-streampipes/blob/dev/streampipes-rest/src/main/java/org/apache/streampipes/rest/impl/datalake/DataLakeResourceV3.java
> >
> > On 2020/11/24 14:24:21, udeho <ud...@student.kit.edu> wrote:
> > > Hi all,
> > >
> > > For a small object detection use case I have stored some images in the internal data lake to label them with the labeling tool.
> > > I discovered that the use of underscores in the data lake identifier prevents the user from accessing the images later in the data explorer and the labeling tool.
> > > As far as I can see, the underscore is already used as a delimiter in the http-request, which leads to a conflict if the identifier also contains an underscore.
> > >
> > > Best regards,
> > > Tim
> > >
> > >
> >
>
>
>
RE: [BUG] Accessing Images from Data Lake Identifiers with Underscores
Posted by Dominik Riemer <ri...@apache.org>.
Hi,
I'd also prefer something which works with an ID.
Some application servers disallow encoded slashes by default, for the backend, we need to override this (see ALLOW_ENCODED_SLASH) in [1].
Just in case you ever spend hours finding out why your encoded slash is not working 😉
Dominik
[1] https://github.com/apache/incubator-streampipes/blob/dev/streampipes-backend/src/main/java/org/apache/streampipes/backend/StreamPipesBackendApplication.java
-----Original Message-----
From: udeho <ud...@student.kit.edu>
Sent: Thursday, December 3, 2020 11:02 AM
To: dev@streampipes.apache.org
Subject: Re: [BUG] Accessing Images from Data Lake Identifiers with Underscores
Hi Johannes,
I would also assume that an encoding with backslashes is the easiest solution.
Maybe using IDs instead of encoding the file route could be an option to avoid encoding issues.
But I do not know whether this can be done without to much effort.
Best,
Tim
On Nov. 28 2020, at 10:24 am, Johannes Tex <te...@apache.org> wrote:
> Hello, Tim,
>
> You are right. We encode some chars of the image path location when we save the image in the Datalake [1]. This was necessary that the can images via an http route [2].
> I think the easiest way would be to change the encoding. Maybe we should encode with '%2F', this is the encoding for slashes. But I am not sure if we can still access the images with [2]. Do you have another idea? The best solution would be a solution where the encoding can be removed.
> Johannes
> [1] https://github.com/apache/incubator-streampipes-extensions/blob/dev/streampipes-sinks-internal-jvm/src/main/java/org/apache/streampipes/sinks/internal/jvm/datalake/DataLake.java
> [2] https://github.com/apache/incubator-streampipes/blob/dev/streampipes-rest/src/main/java/org/apache/streampipes/rest/impl/datalake/DataLakeResourceV3.java
>
> On 2020/11/24 14:24:21, udeho <ud...@student.kit.edu> wrote:
> > Hi all,
> >
> > For a small object detection use case I have stored some images in the internal data lake to label them with the labeling tool.
> > I discovered that the use of underscores in the data lake identifier prevents the user from accessing the images later in the data explorer and the labeling tool.
> > As far as I can see, the underscore is already used as a delimiter in the http-request, which leads to a conflict if the identifier also contains an underscore.
> >
> > Best regards,
> > Tim
> >
> >
>