You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sdap.apache.org by GitBox <gi...@apache.org> on 2020/08/05 20:43:17 UTC
[GitHub] [incubator-sdap-ingester] eamonford opened a new pull request #15: SDAP-277: Improved error handling in Granule Ingester
eamonford opened a new pull request #15:
URL: https://github.com/apache/incubator-sdap-ingester/pull/15
https://issues.apache.org/jira/browse/SDAP-277
This PR accomplishes 3 things:
1) Any dropped network connections will attempt to reconnect
2) Any unrecoverable errors or dropped connections that cannot be reconnected will result in requeueing unprocessed RabbitMQ messages and the app gracefully exiting
2) All unrecoverable errors will log a descriptive message before exit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sdap-ingester] eamonford merged pull request #15: SDAP-277: Improved error handling in Granule Ingester
Posted by GitBox <gi...@apache.org>.
eamonford merged pull request #15:
URL: https://github.com/apache/incubator-sdap-ingester/pull/15
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sdap-ingester] eamonford commented on a change in pull request #15: SDAP-277: Improved error handling in Granule Ingester
Posted by GitBox <gi...@apache.org>.
eamonford commented on a change in pull request #15:
URL: https://github.com/apache/incubator-sdap-ingester/pull/15#discussion_r466073626
##########
File path: granule_ingester/granule_ingester/pipeline/Pipeline.py
##########
@@ -91,43 +101,61 @@ def __init__(self,
self._metadata_store_factory = metadata_store_factory
self._max_concurrency = max_concurrency
- @classmethod
- def from_string(cls, config_str: str, data_store_factory, metadata_store_factory, max_concurrency: int = 16):
- config = yaml.load(config_str, yaml.FullLoader)
- return cls._build_pipeline(config,
- data_store_factory,
- metadata_store_factory,
- processor_module_mappings,
- max_concurrency)
+ # Create a SyncManager so that we can to communicate exceptions from the
+ # worker processes back to the main process.
+ self._manager = Manager()
+
+ def __del__(self):
+ self._manager.shutdown()
@classmethod
- def from_file(cls, config_path: str, data_store_factory, metadata_store_factory, max_concurrency: int = 16):
- with open(config_path) as config_file:
- config = yaml.load(config_file, yaml.FullLoader)
+ def from_string(cls, config_str: str, data_store_factory, metadata_store_factory, max_concurrency: int = 16):
+ try:
+ config = yaml.load(config_str, yaml.FullLoader)
+ cls._validate_config(config)
return cls._build_pipeline(config,
data_store_factory,
metadata_store_factory,
processor_module_mappings,
max_concurrency)
+ except yaml.scanner.ScannerError:
+ raise PipelineBuildingError("Cannot build pipeline because of a syntax error in the YAML.")
+
+ # TODO: this method should validate the config against an actual schema definition
+ @staticmethod
+ def _validate_config(config: dict):
+ if type(config) is not dict:
+ raise PipelineBuildingError("Cannot build pipeline because the config is not valid YAML.")
Review comment:
Thanks, I will try to come up with a better message here
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-sdap-ingester] tloubrieu-jpl commented on a change in pull request #15: SDAP-277: Improved error handling in Granule Ingester
Posted by GitBox <gi...@apache.org>.
tloubrieu-jpl commented on a change in pull request #15:
URL: https://github.com/apache/incubator-sdap-ingester/pull/15#discussion_r466009301
##########
File path: granule_ingester/granule_ingester/pipeline/Pipeline.py
##########
@@ -91,43 +101,61 @@ def __init__(self,
self._metadata_store_factory = metadata_store_factory
self._max_concurrency = max_concurrency
- @classmethod
- def from_string(cls, config_str: str, data_store_factory, metadata_store_factory, max_concurrency: int = 16):
- config = yaml.load(config_str, yaml.FullLoader)
- return cls._build_pipeline(config,
- data_store_factory,
- metadata_store_factory,
- processor_module_mappings,
- max_concurrency)
+ # Create a SyncManager so that we can to communicate exceptions from the
+ # worker processes back to the main process.
+ self._manager = Manager()
+
+ def __del__(self):
+ self._manager.shutdown()
@classmethod
- def from_file(cls, config_path: str, data_store_factory, metadata_store_factory, max_concurrency: int = 16):
- with open(config_path) as config_file:
- config = yaml.load(config_file, yaml.FullLoader)
+ def from_string(cls, config_str: str, data_store_factory, metadata_store_factory, max_concurrency: int = 16):
+ try:
+ config = yaml.load(config_str, yaml.FullLoader)
+ cls._validate_config(config)
return cls._build_pipeline(config,
data_store_factory,
metadata_store_factory,
processor_module_mappings,
max_concurrency)
+ except yaml.scanner.ScannerError:
+ raise PipelineBuildingError("Cannot build pipeline because of a syntax error in the YAML.")
+
+ # TODO: this method should validate the config against an actual schema definition
+ @staticmethod
+ def _validate_config(config: dict):
+ if type(config) is not dict:
+ raise PipelineBuildingError("Cannot build pipeline because the config is not valid YAML.")
Review comment:
The exception message should be different here: "Cannot build pipeline because the config is not valid" (no better idea)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org