Administering the ISS Watcher Service

Skip to end of metadata
Go to start of metadata

Administering the ISS Watcher Service

Topics:

ISS Watcher Service Overview

This feature was introduced in Indexing and Search Service 1 Update 4.

The watcher service provides local host monitoring of ISS services. When the watcher service detects a service outage, it generates log file warnings and email alerts to help you take recovery action. The watcher service does not automatically restart ISS services. You must manually restart ISS services when the watcher detects an outage. The watcher service's monitoring consists of connecting to services, authenticating if necessary, and checking for a configuration parameter or a response from the service beyond a simple TCP check.

See How the ISS Watcher Service Performs Monitoring for which parameters and services the watcher service can monitor and check.

To enable the watcher service on an ISS host, you configure the appropriate parameters in the jiss.conf file. See To Enable the ISS Watcher Service for more information.

How the ISS Watcher Service Performs Monitoring

The watcher service monitoring consists of checking the availability of jiss.conf parameters at the TCP level. The watcher service connects to a configured host name and port to check if a TCP connect string is issued.

The following table shows which ISS configuration parameters the watcher service monitors based on the ISS installation type.

jiss.conf Parameters Monitored by the Watcher Service

iss.cluster.install Value iss.cluster.type Value jiss.conf Parameters Monitored
standalone (all) mail.imq
imq.host
ldap.host
appserv.web.port/ localhost
multi-machine index mail.imq
multi-machine jmq imq.host
multi-machine ldap ldap.host
multi-machine web appserv.web.port / localhost
cluster web imq.host (localhost)
appserv.web.port/ localhost
cluster index imq.host (localhost)
ldap.host
mail.imq
clusterv2 web appserv.web.port / localhost
clusterv2 index ldap.host
imq.host
mail.imq

The following table shows which services the watcher service checks and how it performs those checks.

ISS Services Monitored by the Watcher Service

iss.cluster.install iss.cluster.type Description of What Is Checked
standalone (all) JNDI lookup for queue.connection.factory.name
JNDI lookup for topic.connection.factory.name
JNDI lookup for iss.searchsvc.dst.name
JNDI lookup for iss.indexsvc.dst.name
Checks that indexSvc (isAlive method) is running
Checks that searchSvc (isSearch method) is running
Checks that jmqconsumer is running
multi-machine jmq JNDI lookup for queue.connection.factory.name
JNDI lookup for topic.connection.factory.name
JNDI lookup for iss.searchsvc.dst.name
JNDI lookup for iss.indexsvc.dst.name
multi-machine index Checks that indexSvc (isAlive method) is running
Checks that searchSvc (isSearch method) is running
Checks that jmqconsumer is running
cluster web JNDI lookup for queue.connection.factory.name
JNDI lookup for topic.connection.factory.name
JNDI lookup for iss.searchsvc.dst.name
Checks that searchSvc (isSearch method) is running per instance of cluster search service
cluster index JNDI lookup for queue.connection.factory.name
JNDI lookup for topic.connection.factory.name
JNDI lookup for iss.indexsvc.dst.name
Checks that indexSvc (isAlive method) is running
Checks that jmqconsumer is running
clusterv2 index JNDI lookup for queue.connection.factory.name
JNDI lookup for topic.connection.factory.name
JNDI lookup for iss.searchsvc.dst.name
JNDI lookup for iss.indexsvc.dst.name
Checks that indexSvc (isAlive method) is running
Checks that searchSvc (isSearch method) is running per instance of cluster search service
clusterv2 csearch JNDI lookup for queue.connection.factory.name
JNDI lookup for topic.connection.factory.name
JNDI lookup for iss.searchsvc.dst.name
Checks that searchSvc (isSearch method) is running per instance of cluster search service

Configuring the Watch Service

Topics in this section:

To Enable the ISS Watcher Service

To enable the watcher service on an ISS node, configure the appropriate iss.cluster and iss.watcher parameters in the jiss.conf file as described in this procedure.

  1. Edit the iss-base/etc/jiss.conf file.
  2. Set iss.watcher.enabled = true to run the watcher service on this host.
    Alternately, you can run the watchermrg.sh command. See watchermgr.sh for more information.
  3. (Optional) Set other watcher runtime parameters, such as notification and alert settings.
    For more information on optional iss.watcher parameters, see Indexing and Search Service Configuration Parameters.
  4. Save your changes to the jiss.conf file.
  5. Run the watchermgr.sh command to enable the service.
  6. Verify that the watcher has been configured correctly. For example, you can simulate failures on the following components and check the log file for entries or confirm that the appropriate email is sent.
    • Simulate a GlassFish Server failure.
    • Simulate a broker failure.
    • Simulate an LDAP failure.
    • Simulate an indexing service failure.
      On Linux:

      On Solaris:

The watcher service generates an alert each time that a single test fails. If more than one test fails in a configured interval, the alert level is increased by one for each test that fails. An alert level of zero (0) means that all basic and functionality test have passed. Depending on the iss.cluster.install and iss.cluster.type configurations, the maximum alert levels are as follows.

Watcher Maximum Alert Level

iss.cluster.install iss.cluster.type Maximum alert level
standalone (all) 10
multi-machine index 3
multi-machine jmq 5
multi-machine ldap 1
multi-machine web 1
cluster web 5 + number of indexing nodes
cluster index 7
clusterv2 web 1
clusterv2 index 8 + number of indexing nodes - 1

Multi-machine installations can combine various services into a single instance and thus iss.cluster.type has multiple values defined. This increases the maximum alert level for that instance.

To Configure Email Notifications

  1. Edit the iss-base/etc/jiss.conf file.
  2. Set or change the following configuration parameters accordingly.
    • iss.watcher.notification.type: email.
    • iss.watcher.notification.destination: Email address to which to send watcher notifications
    • iss.watcher.notification.source: Source (from) email address for watcher notifications
    • iss.watcher.notification.mail.host: host:port of the messaging host sending the notifications
    • iss.watcher.notification.protocol: Protocol to use when sending email notifications, either ssl, tls, or plain
    • iss.watcher.notification.email.interval: Number of seconds between which the watcher service sends an email alert, once it first detects a failure
    • iss.watcher.email.user: User name, if required, to send email
    • iss.watcher.email.user.password: User name password, if required, to send email
      For more information, see Indexing and Search Service Configuration Parameters.
  3. Save your changes to the jiss.conf file.
  4. Refresh the ISS configuration for the change to take place.

How the Watcher Service Determines to Send Email Notifications

The following table defines how the watcher service sends email to the value of the iss.watcher.notification.destination parameter. If the value of the iss.watcher.email.interval parameter is less than or equal to the value of the iss.watcher.interval parameter, the watcher service sends an alert email every iss.watcher.interval seconds.

Error Action Table

Current Level New Level iss.watcher.email.interval Reached? Email Sent?
0 Greater than or equal to 1 Not checked Yes
Greater than or equal to 1 0 Not checked Yes
Greater than or equal to 1 Greater than or equal to 1 No No
Greater than or equal to 1 Greater than or equal to 1 Yes Yes

If an alert level changes but does not reach zero, no new email is sent if the iss.watcher.email.interval has not been reached.

Labels:
ha ha Delete
indexsearchservice indexsearchservice Delete
Enter labels to add to this page:
Please wait 
Looking for a label? Just start typing.

Sign up or Log in to add a comment or watch this page.


The individuals who post here are part of the extended Oracle community and they might not be employed or in any way formally affiliated with Oracle. The opinions expressed here are their own, are not necessarily reviewed in advance by anyone but the individual authors, and neither Oracle nor any other party necessarily agrees with them.