You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@streampipes.apache.org by Dominik Riemer <ri...@apache.org> on 2021/01/18 07:33:16 UTC
Changes in data explorer
Hi all,
I did some changes to the data explorer management and extracted all queries
for historic data to an individual module (streampipes-data-explorer). While
doing this, I also split the previously existing DataLakeManagementv3 class
into several smaller units to make it more readable.
The change is currently in the STREAMPIPES-272 branch [1], but I think we
can merge this soon for the next release.
Dominik
[1]
https://github.com/apache/incubator-streampipes/tree/STREAMPIPES-272/streamp
ipes-data-explorer/src/main/java/org/apache/streampipes/dataexplorer
Re: Changes in data explorer
Posted by Dominik Riemer <ri...@apache.org>.
Hi Philipp,
to be honest, I don't know yet - but I'd say there will be some data model that links a data stream to the corresponding index. This is probably similar to how we currently handle "visualizable pipelines".
Dominik
On 2021/01/18 21:35:20, Philipp Zehnder <ze...@fzi.de> wrote:
> Hi,
> very cool, the API is now much more clear.
>
> How will we handle the read of data from the data lake in the future?
> Currently the index name is used, but I guess we will change this to the stream id,
> right? How will this look like?
>
> Philipp
>
> Am 18.01.2021 um 08:33 schrieb Dominik Riemer <ri...@apache.org>:
>
> Hi all,
>
>
>
> I did some changes to the data explorer management and extracted all queries
> for historic data to an individual module (streampipes-data-explorer). While
> doing this, I also split the previously existing DataLakeManagementv3 class
> into several smaller units to make it more readable.
>
> The change is currently in the STREAMPIPES-272 branch [1], but I think we
> can merge this soon for the next release.
>
>
>
> Dominik
>
>
>
> [1]
> https://github.com/apache/incubator-streampipes/tree/STREAMPIPES-272/streamp
> ipes-data-explorer/src/main/java/org/apache/streampipes/dataexplorer
>
>
>
>
Re: Changes in data explorer
Posted by Philipp Zehnder <ze...@fzi.de>.
Hi,
very cool, the API is now much more clear.
How will we handle the read of data from the data lake in the future?
Currently the index name is used, but I guess we will change this to the stream id,
right? How will this look like?
Philipp
Am 18.01.2021 um 08:33 schrieb Dominik Riemer <ri...@apache.org>:
Hi all,
I did some changes to the data explorer management and extracted all queries
for historic data to an individual module (streampipes-data-explorer). While
doing this, I also split the previously existing DataLakeManagementv3 class
into several smaller units to make it more readable.
The change is currently in the STREAMPIPES-272 branch [1], but I think we
can merge this soon for the next release.
Dominik
[1]
https://github.com/apache/incubator-streampipes/tree/STREAMPIPES-272/streamp
ipes-data-explorer/src/main/java/org/apache/streampipes/dataexplorer