[GitHub] [streampipes] tenthe edited a comment on the discussion: StreamPipes Structure

[GitBox <gi...@apache.org>]

GitHub user tenthe edited a comment on the discussion: StreamPipes Structure

# Overview StreamPipes Modules
I have looked at all the modules we have in our Java codebase and tried to summarize their main intent. I also added some discussion points that I noticed.

- **client** / **client-python**
	- Clients to interact with the StreamPipes API
- **commons**
    - Constants, Exceptions, Networking, Parser, Random(Generators), ZIP
- **config**
    - Configurations that can be changed by users, as well as configuratioins for the core service
- **connect-management**
	- Contains the adapter management, that is required in the service extensions during runtime
    - `DISCUSS` Should we rename the classes?
- **data-explorer**
	-  Application logic to query data from the time-series database
    - `DISCUSS` Should we change the module name?
    - `DISCUSS` Which classes should be moved to the **REST modules**
    - `DISCUSS` In this module we have a strong coupling to the InfluxDB
- **data-explorer-commons**
    -  Configurations, InfluxStore, ImageStore, TimeSeriesStore
    - `DISCUSS`  Can we move the code to module **data-explorer**
- **data-export**
	- Responsible to manage the export / import of StreamPipes configurations
    - `DISCUSS` Think about a different name. Can be confused with export of data explorer data.
- **dataformat**
    - Those modules contain all the implementation for the data formats
    - (cbor, fst, json, smile)
- **extensions**
    - Contains all modules with adapters, processors, and sinks
- **extensions-api**
    - Interfaces for the extensions (See module below)
- **extensions-management**
    - Application logic relevant for the modules that are implemented in **extensions**
- **integration-tests**
    - Contain tests for services that have third party requirements
- **logging**
	- `DISCUSS` Do we still need this module or is it legacy code
- **mail**
    - Logic to send emails via the system
- **maven-plugin**
    - Maven plugin to extract pipeline element information from code and provide it for the website documentation
- **measurement-units**
    - logic to deal with different units
- **messaging**
    - Different implementation for internally used message brokers
    - (jms, kafka, mqtt, nats)
- **model**
    - Contains all model classes of StreamPipes
    - This model is also used to automatically generate the model for TypeScript
- **model-client**
    - Model classes that are only required by the UI
    - This model is also used to automatically generate the model for TypeScript
- **model-shared**
    - Only contains annotations `TsIgnore` & `TsModel`
- performance-tests `REMOVED`
- **pipeline-management**
	- Contains application logic for the pipeline editor
- **platform-service**
    - Currently contains Resources for the data lake
    - `DISCUSS` Move those classes into other modules. How will we structure the REST modules?
- **resource-management**
    - `DISCUSS` What is managed in this module and what could be moved to `pipeline-management` or `connect-management`
- **rest**
    - The `impl` package currenlty contains the most REST endpoints
    - **rest-core-base**
        - Only contains two classes `AbstractAuthGuardedRestResource` & `AbstactRestResource`
    - **rest-extensions**
        - Rest enpoints of **service extensions**
        - Currently mainly endpoints to manage connect & pipelines
    - **rest-shared**
        - (annotation, impl, serializer, util )
- **sdk**
	- builder, extractor, helpers, 
    - Application logic to extend StreamPipes functionalities
- **sdk-bundle**
    - Contains no classes, is only used to budle several packages
    - `DISCUSS` Do we need this or can we move this to  **sdk** instead
- **security-jwt**
    - Logic to handle json web tokens
- **serializers-json**
    - Contains one class, the `JacksonSerializer`
    - `DISCUSS` Do we require a seperate module for this?
- **service-base**
    - Base module for StreamPipes services (**service core** and **service extension**)
- **service-core**
    - Is executed when StreamPipes is started
    - Contains migration scripts and starts up REST endpooint
- **service-discovery**
    - Only contains class `SpServiceDiscovery`
	- **service-discovery-api**
		- Interface of the service discovery 
	- **service-discovery-consul**
		- Consul implementation of the service discovery interface
- **service-extensions**
    - Module for extension services containing (adapters, processing elements, sinks, functions) during runtime
    - Contains `ExtensionsModelSubmitter` that is implemented by all extension services
- **sources**
    - Abstract classes `AbstractAdapterIncludedStream` & `AbstractAlreadyExistingStream`
    - Required for streams that are not created by connect adapters
    - `DISCUSS` Should we continue to support this or only rely on the connect API?
- **storage-api**
    - Interfaces of all classes that can be persistet in the database
- **storage-couchdb**
    - CouchDB implementation of the **storage-api**
- **storage-management**
    - Contains two classes `StorageDispatcher` & `StorageManager`
- **test-utils**
	- Contains utility classes to ease the creation of unit tests
- **user-management**
	- Responsible for user management within StreamPipes
	- Packages: authentication, encryption, jwt, model, service
- **vocabulary**
    - Contains classes with different vocabularies
- **wrapper**
    - Base module for the different wrappers
    - **wrapper-distributed**
        - Base module for the distributed wrappers
		- Contains one abstract class `DistributedRuntime`
    - **wrapper-flink**
        - Wrapper for Flink
    - **wrapper-kafka-streams**
        - Wrapper for Kafka Streams
    - **wrapper-python**
        - Python wrapper
        - `DISCUSS` Is this module still relevant for the new Python integration?
    - **wrapper-siddhi**
        - Wrapper for Siddhi engine
    - **wrapper-standalone**
        - Module for standalone wrappers that are not distributed

GitHub link: https://github.com/apache/streampipes/discussions/1038#discussioncomment-4600352

----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org