You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by oppo <op...@qq.com> on 2017/12/21 09:09:48 UTC
How to get controller service in Script Executor
Hi,
I try to use javascript in processor to do some enrichment and transformation, but i don't know how to concat the service identifier.
As it is ExecutorScript processor, I can't add a service reference in properties, I can only add service in current Group (I put logic in one Process Group), I have name and id of service, I enable the service.
See the following code:
var enrichmentService = context.getControllerServiceLookup().getControllerService("DBCPConnectionPool");
if(enrichmentService == null) {
log.warn("Can't find DBCPConnectionPool");
session.transfer(flowFile, REL_FAILURE);
}
It always jump to null condition.
Anyone can help?
Kui
回复： How to get controller service in Script Executor
Posted by oppo <op...@qq.com>.
Eventually I find a way to make it work. The "init" method is triggered when the service instance is created before the property is configured/ready. I can't do data loading there.
I need to prepare another API to lazily load data from database, before any call of "getXXX(String key)". As it is multi-thread env, I need to use synchronized key to make it thread safe. See the following method:
private synchronized void reloadOnce() {
if(this.dbcpService == null) {
this.dbcpService = getProperty(DB_CONNECTION).asControllerService(DBCPService.class);
this.loadTable();
}
}public String getMonitor(String realValue) throws ProcessException {
reloadOnce();
return this.mapMonitor.get(realValue);
}Not sure the performance is good in this way. I also found the enable/disable is not override-able as declared with "final". As a service, I really want to control its life-cycle behavior, at least doing something at initialize/enable/disable/destroy stage. But the current controller service is lack of such capability, or I miss anything? Thanks and Regards.Kui
------------------ 原始邮件 ------------------
发件人: "oppo";<op...@qq.com>;
发送时间: 2017年12月22日(星期五) 上午9:42
收件人: "users"<us...@nifi.apache.org>;
主题: 回复： How to get controller service in Script Executor
Thanks, Matt. It works perfect!
Another question, I try to put the logic into a new controller service, as I want to load data from database and cache in service. So I write java code with api and bundle project and build in mvn, similar with NiFi standard service. In this enrichment service, I need DBCP service as a property, and I get connection and then load some data from database, cache them and prepare some "getXXX(String key)" method for Script in ExecutorScript processor.
The following is the sample code for property definition and init method. I met the problem in init() when trying to get DBCP service and load table. Whatever I use "DBCPService.class" or "ControllerService.class", getControllerServiceIdentifiers() method throw exception. I assume "init()" is triggered when I configure enrichment service in one Processor Group and enable it, as I also configured DBCP service in enrichment service's property and also configure DBCP properties like JDBC URL, user/pwd, etc., I hope I can get the connection, but fail.
See the exception from the log. It sounds Controller Service has only one instance in NiFi JVM, and init() is triggered at startup so it doesn't know processor group information?
------------------------------- Code ------------------------------
public static final PropertyDescriptor DB_CONNECTION =
new PropertyDescriptor.Builder()
.name("data.source")
.displayName("Data Source")
.description("Database connection information")
.required(true)
.identifiesControllerService(DBCPService.class)
.expressionLanguageSupported(false)
.build();
@Override
protected void init(final ControllerServiceInitializationContext context) throws InitializationException {
final List<PropertyDescriptor> properties = new ArrayList<>();
properties.add(DB_CONNECTION);
this.properties = Collections.unmodifiableList(properties);
Set<String> identifiers = context.getControllerServiceLookup().getControllerServiceIdentifiers(DBCPService.class);
// doesn't work if I use "identifiers.toArray()[0]" because identifiers is of 0 length
this.dbcpService = (DBCPService)context.getControllerServiceLookup().getControllerService("785b224c-0160-1000-d75e-dcb255cedd6a");
this.loadTable();
}
------------------------------- log ------------------------------
Caused by: org.apache.nifi.controller.exception.ControllerServiceInstantiationException: java.lang.UnsupportedOperationException: Cannot obtain Controller Service Identifiers for service type interface org.apache.nifi.controller.ControllerService without providing a Process Group Identifier
at org.apache.nifi.controller.service.StandardControllerServiceProvider.createControllerService(StandardControllerServiceProvider.java:174)
at org.apache.nifi.controller.FlowController.createControllerService(FlowController.java:3313)
at org.apache.nifi.controller.service.ControllerServiceLoader.createControllerService(ControllerServiceLoader.java:198)
at org.apache.nifi.controller.service.ControllerServiceLoader.loadControllerServices(ControllerServiceLoader.java:114)
at org.apache.nifi.controller.StandardFlowSynchronizer.addProcessGroup(StandardFlowSynchronizer.java:1078)
at org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:322)
... 33 common frames omitted
Caused by: java.lang.UnsupportedOperationException: Cannot obtain Controller Service Identifiers for service type interface org.apache.nifi.controller.ControllerService without providing a Process Group Identifier
at org.apache.nifi.controller.service.StandardControllerServiceProvider.getControllerServiceIdentifiers(StandardControllerServiceProvider.java:678)
at org.apache.nifi.controller.service.StandardControllerServiceInitializationContext.getControllerServiceIdentifiers(StandardControllerServiceInitializationContext.java:60)
at org.apache.nifi.dbproxy.EnrichmentServiceImpl.init(EnrichmentServiceImpl.java:76)
at org.apache.nifi.controller.AbstractControllerService.initialize(AbstractControllerService.java:43)
at org.apache.nifi.controller.service.StandardControllerServiceProvider.createControllerService(StandardControllerServiceProvider.java:148)
... 38 common frames omitted
------------------ 原始邮件 ------------------
发件人: "Matt Burgess";<ma...@apache.org>;
发送时间: 2017年12月22日(星期五) 凌晨3:19
收件人: "users"<us...@nifi.apache.org>;
主题: Re: How to get controller service in Script Executor
Kui,
The getControllerService() method requires a controller service (CS)
identifier, not the name (because names are not necessarily unique).
To get the CS by name, you have to get the list of all CSs and match
on the name. I have an example in Groovy on my blog [1], but here is a
similar one ported to Nashorn/Javascript:
var ControllerService =
Java.type('org.apache.nifi.controller.ControllerService');
var lookup = context.getControllerServiceLookup();
var dbServiceName = databaseConnectionPoolName.getValue();
var dbcpServiceId = null;
for each(var cs in
lookup.getControllerServiceIdentifiers(ControllerService.class)) {
if(lookup.getControllerServiceName(cs) == dbServiceName) {
dbcpServiceId = cs;
break;
}
}
if(dbcpServiceId != null) {
var conn = lookup.getControllerService(dbcpServiceId).getConnection();
var stmt = conn.createStatement();
var resultSet = stmt.execute("SELECT * FROM users");
// do stuff here
log.info('***** do stuff here *****');
} else {
log.error("Couldn't find Controller Service: " + dbServiceName);
}
Regards,
Matt
[1] http://funnifi.blogspot.com/2016/04/sql-in-nifi-with-executescript.html
On Thu, Dec 21, 2017 at 4:09 AM, oppo <op...@qq.com> wrote:
> Hi,
>
> I try to use javascript in processor to do some enrichment and
> transformation, but i don't know how to concat the service identifier.
> As it is ExecutorScript processor, I can't add a service reference in
> properties, I can only add service in current Group (I put logic in one
> Process Group), I have name and id of service, I enable the service.
>
> See the following code:
>
> var enrichmentService =
> context.getControllerServiceLookup().getControllerService("DBCPConnectionPool");
> if(enrichmentService == null) {
> log.warn("Can't find DBCPConnectionPool");
> session.transfer(flowFile, REL_FAILURE);
> }
>
> It always jump to null condition.
>
> Anyone can help?
>
> Kui
回复： How to get controller service in Script Executor
Posted by oppo <op...@qq.com>.
Thanks, Matt. It works perfect!
Another question, I try to put the logic into a new controller service, as I want to load data from database and cache in service. So I write java code with api and bundle project and build in mvn, similar with NiFi standard service. In this enrichment service, I need DBCP service as a property, and I get connection and then load some data from database, cache them and prepare some "getXXX(String key)" method for Script in ExecutorScript processor.
The following is the sample code for property definition and init method. I met the problem in init() when trying to get DBCP service and load table. Whatever I use "DBCPService.class" or "ControllerService.class", getControllerServiceIdentifiers() method throw exception. I assume "init()" is triggered when I configure enrichment service in one Processor Group and enable it, as I also configured DBCP service in enrichment service's property and also configure DBCP properties like JDBC URL, user/pwd, etc., I hope I can get the connection, but fail.
See the exception from the log. It sounds Controller Service has only one instance in NiFi JVM, and init() is triggered at startup so it doesn't know processor group information?
------------------------------- Code ------------------------------
public static final PropertyDescriptor DB_CONNECTION =
new PropertyDescriptor.Builder()
.name("data.source")
.displayName("Data Source")
.description("Database connection information")
.required(true)
.identifiesControllerService(DBCPService.class)
.expressionLanguageSupported(false)
.build();
@Override
protected void init(final ControllerServiceInitializationContext context) throws InitializationException {
final List<PropertyDescriptor> properties = new ArrayList<>();
properties.add(DB_CONNECTION);
this.properties = Collections.unmodifiableList(properties);
Set<String> identifiers = context.getControllerServiceLookup().getControllerServiceIdentifiers(DBCPService.class);
// doesn't work if I use "identifiers.toArray()[0]" because identifiers is of 0 length
this.dbcpService = (DBCPService)context.getControllerServiceLookup().getControllerService("785b224c-0160-1000-d75e-dcb255cedd6a");
this.loadTable();
}
------------------------------- log ------------------------------
Caused by: org.apache.nifi.controller.exception.ControllerServiceInstantiationException: java.lang.UnsupportedOperationException: Cannot obtain Controller Service Identifiers for service type interface org.apache.nifi.controller.ControllerService without providing a Process Group Identifier
at org.apache.nifi.controller.service.StandardControllerServiceProvider.createControllerService(StandardControllerServiceProvider.java:174)
at org.apache.nifi.controller.FlowController.createControllerService(FlowController.java:3313)
at org.apache.nifi.controller.service.ControllerServiceLoader.createControllerService(ControllerServiceLoader.java:198)
at org.apache.nifi.controller.service.ControllerServiceLoader.loadControllerServices(ControllerServiceLoader.java:114)
at org.apache.nifi.controller.StandardFlowSynchronizer.addProcessGroup(StandardFlowSynchronizer.java:1078)
at org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:322)
... 33 common frames omitted
Caused by: java.lang.UnsupportedOperationException: Cannot obtain Controller Service Identifiers for service type interface org.apache.nifi.controller.ControllerService without providing a Process Group Identifier
at org.apache.nifi.controller.service.StandardControllerServiceProvider.getControllerServiceIdentifiers(StandardControllerServiceProvider.java:678)
at org.apache.nifi.controller.service.StandardControllerServiceInitializationContext.getControllerServiceIdentifiers(StandardControllerServiceInitializationContext.java:60)
at org.apache.nifi.dbproxy.EnrichmentServiceImpl.init(EnrichmentServiceImpl.java:76)
at org.apache.nifi.controller.AbstractControllerService.initialize(AbstractControllerService.java:43)
at org.apache.nifi.controller.service.StandardControllerServiceProvider.createControllerService(StandardControllerServiceProvider.java:148)
... 38 common frames omitted
------------------ 原始邮件 ------------------
发件人: "Matt Burgess";<ma...@apache.org>;
发送时间: 2017年12月22日(星期五) 凌晨3:19
收件人: "users"<us...@nifi.apache.org>;
主题: Re: How to get controller service in Script Executor
Kui,
The getControllerService() method requires a controller service (CS)
identifier, not the name (because names are not necessarily unique).
To get the CS by name, you have to get the list of all CSs and match
on the name. I have an example in Groovy on my blog [1], but here is a
similar one ported to Nashorn/Javascript:
var ControllerService =
Java.type('org.apache.nifi.controller.ControllerService');
var lookup = context.getControllerServiceLookup();
var dbServiceName = databaseConnectionPoolName.getValue();
var dbcpServiceId = null;
for each(var cs in
lookup.getControllerServiceIdentifiers(ControllerService.class)) {
if(lookup.getControllerServiceName(cs) == dbServiceName) {
dbcpServiceId = cs;
break;
}
}
if(dbcpServiceId != null) {
var conn = lookup.getControllerService(dbcpServiceId).getConnection();
var stmt = conn.createStatement();
var resultSet = stmt.execute("SELECT * FROM users");
// do stuff here
log.info('***** do stuff here *****');
} else {
log.error("Couldn't find Controller Service: " + dbServiceName);
}
Regards,
Matt
[1] http://funnifi.blogspot.com/2016/04/sql-in-nifi-with-executescript.html
On Thu, Dec 21, 2017 at 4:09 AM, oppo <op...@qq.com> wrote:
> Hi,
>
> I try to use javascript in processor to do some enrichment and
> transformation, but i don't know how to concat the service identifier.
> As it is ExecutorScript processor, I can't add a service reference in
> properties, I can only add service in current Group (I put logic in one
> Process Group), I have name and id of service, I enable the service.
>
> See the following code:
>
> var enrichmentService =
> context.getControllerServiceLookup().getControllerService("DBCPConnectionPool");
> if(enrichmentService == null) {
> log.warn("Can't find DBCPConnectionPool");
> session.transfer(flowFile, REL_FAILURE);
> }
>
> It always jump to null condition.
>
> Anyone can help?
>
> Kui
Re: How to get controller service in Script Executor
Posted by Matt Burgess <ma...@apache.org>.
Kui,
The getControllerService() method requires a controller service (CS)
identifier, not the name (because names are not necessarily unique).
To get the CS by name, you have to get the list of all CSs and match
on the name. I have an example in Groovy on my blog [1], but here is a
similar one ported to Nashorn/Javascript:
var ControllerService =
Java.type('org.apache.nifi.controller.ControllerService');
var lookup = context.getControllerServiceLookup();
var dbServiceName = databaseConnectionPoolName.getValue();
var dbcpServiceId = null;
for each(var cs in
lookup.getControllerServiceIdentifiers(ControllerService.class)) {
if(lookup.getControllerServiceName(cs) == dbServiceName) {
dbcpServiceId = cs;
break;
}
}
if(dbcpServiceId != null) {
var conn = lookup.getControllerService(dbcpServiceId).getConnection();
var stmt = conn.createStatement();
var resultSet = stmt.execute("SELECT * FROM users");
// do stuff here
log.info('***** do stuff here *****');
} else {
log.error("Couldn't find Controller Service: " + dbServiceName);
}
Regards,
Matt
[1] http://funnifi.blogspot.com/2016/04/sql-in-nifi-with-executescript.html
On Thu, Dec 21, 2017 at 4:09 AM, oppo <op...@qq.com> wrote:
> Hi,
>
> I try to use javascript in processor to do some enrichment and
> transformation, but i don't know how to concat the service identifier.
> As it is ExecutorScript processor, I can't add a service reference in
> properties, I can only add service in current Group (I put logic in one
> Process Group), I have name and id of service, I enable the service.
>
> See the following code:
>
> var enrichmentService =
> context.getControllerServiceLookup().getControllerService("DBCPConnectionPool");
> if(enrichmentService == null) {
> log.warn("Can't find DBCPConnectionPool");
> session.transfer(flowFile, REL_FAILURE);
> }
>
> It always jump to null condition.
>
> Anyone can help?
>
> Kui