Overview of Indexing and Search Service Processes

Skip to end of metadata
Go to start of metadata

Overview of Indexing and Search Service for Oracle Communications Unified Communications Suite Processes

This information discusses the different processes (services) comprising Indexing and Search Service and the components that it relies on; the purpose of each service; the relationship between the services; and the services' log files location.

Topics:

Overview of Indexing and Search Service Software Architecture

The following figure shows a "tiered" view of the ISS software architecture.

Indexing and Search Service Logical Tiered View
This figure shows a tiered view of the ISS software architecture.

This figure shows that an ISS deployment uses a logical architecture consisting of the following three tiers:

  • Front-End: Provides the ISS RESTful search servlet and Convergence and Messaging Multiplexor (MMP) client email front-ends
  • Service: Provides the back-end IMAP service, the Java Message Queue (JMQ) broker component of the Java Message Service, and LDAP services
  • Indexing: Provides ISS store back-ends (the "indexed" data for searching)

The ISS components communicate by using topics and queues on Java Message Service (JMS), enabling you to distribute components across multiple physical hosts. During the ISS initial configuration, you decide which ISS components are configured on which host by running the ISS setup command. The setup command refers to these components as:

  • web: Configures an ISS search front end
  • jmq: Configures a JMS Java Message Queue broker
  • ldap: Configures a Directory Server for Java Naming and Directory Interface storage and lookup
  • index: Configures an indexing back end

By default, all ISS components are configured on the system. For more information on the setup command, see Running setup.

Overview of Indexing and Search Service and Dependent Services

Expanding on the previous figure, here is a view of how the ISS services and other component services are distributed:

Indexing and Search Service Services View
This figure shows a view of how the ISS services and other component services are distributed.

From a deployment perspective, the groupings in this figure are pieces that could be deployed on one or multiple hosts, depending on your site's architecture.

Note
This figure shows that two Java Message Service JMQ brokers are present: one for Messaging Server notifications and one for internal ISS communication. In a simple configuration, the JMQ broker for Messaging Server notifications is running on the Messaging Server system and the JMQ broker for inter-process ISS communication is running on the ISS system.

The following sections discuss each component's services.

ISS Web Services

ISS web services (rest, store and searchui) are deployed in a GlassFish web container. These services can be connected to directly by an end user or indirectly by services such as Convergence or Messaging Server's imapd. The user/group directory in Directory Server is used for authentication. These services communicate to the ISS back-end servers by using Java Message Service.

Convergence

ISS content is searched by clients such as Convergence. Convergence is deployed in a GlassFish web container. Convergence contains service proxies for mail and ISS and other services, which communicate by using various protocols to Communications Suite services. For more information, see Introduction to the Convergence Service.

Messaging Server

Messaging Server consists of a number of processes, but for ISS, the most important ones are:

  • imapd: Writes messages to and reads messages from the Messaging Server message store by using the IMAP protocol. Generates event notifications to ISS in response to user mailbox activity (for example, message submits, flag changes, expunged messages, new/renamed/deleted folders).
  • ims_master: One of many channel programs that make up the Messaging Server job controller. Delivers messages into the message store and generates event notifications for each new message delivery.

Directory Server

ISS requires Directory Server for LDAP information storage. ISS uses the JNDI API to enable communications between JMS and Directory Server. ISS uses the following properties to connect to Directory Server to know which name space to use:

java.naming.factory.initial
java.naming.provider.url
java.naming.security.authentication
java.naming.security.credentials
java.naming.security.principal

ISS uses JNDI to look up the values of the following objects, which are stored in LDAP:

com.sun.messaging.QueueConnectionFactory
com.sun.messaging.TopicConnectionFactory
com.sun.messaging.Queue
com.sun.messaging.Topic

Indexing and Search Service Store

ISS back-end services run as separate JVM instances. The core Indexing and Search Service services that are required for normal operation are:

  • indexSvc: Bootstraps new users and indexes incoming content.
  • searchSvc: Searches index and returns results.
  • jmqconsumer: Listens to the JMQ on the message store and notifies indexSvc if new content requires indexing.
  • utilSvc: Provides utility services to Index Service.

On an Oracle Solaris system, these services are managed through System Management Facility (SMF). The ISS command svc_control.sh starts and stops all services. The individual services can also be separately queried, enabled, disabled or restarted via SMF, for example:

Log File Locations

The following table provides the default locations for the ISS log files by service.

ISS Log File Locations

Service Name Log Name and Default Location
Index Service /var/opt/sun/jiss/logs/iss-indexsvc.log.0
JMQ Consumer /var/opt/sun/jiss/logs/iss-jmqconsumer.log.0
JMQ broker /var/imq/instances/imqbroker/log/log.txt
RESTful search servlet, StoreUI servlet, SearchUI servlet /opt/SUNWappserver/domains/domain1/logs/server.log
Search Service /var/opt/sun/jiss/logs/iss-searchsvc.log.0
Util Service /var/opt/sun/jiss/logs/iss-utilsvc.log.0

Service Interaction and Communication

From a high-level perspective, the ISS system operates as follows:

  1. Content is searched by clients.
    • Email clients such as Thunderbird or an IMAP-enabled mobile device get enhanced search through an IMAP SEARCH ISS gateway.
    • The Convergence web-based client has an ISS plugin, which generates searches directly to ISS for attachment search, and enables fast cross folder search.
  2. Messaging Server manages the content in the message store and provides event notifications of changes to the message store to ISS.
  3. GlassFish Server hosts Indexing and Search Service servlets.
    • These servlets consist of the RESTful search API (rest), thumbnail access (store), and a demo application (searchui)
  4. JMQ broker hosts the event notification destination to which Messaging Server is publishing as well as search and index destinations used between ISS components.
    • There may be multiple brokers in use: one for Messaging Server JMQ notifications and one for internal ISS communication.
  5. User authentication and JNDI lookups are performed on the Directory Server.
  6. Indexing and Search Service store instance stores the indexes.
    • It also manages and serves up images from the thumbnail store.
    • Parses email messages from the message store into indexing content.
    • Receives and processes search request messages.

The following section discusses the ISS components in more detail as well as how information flows through the system.

Indexing and Search Service Data Flow

Refer to the following diagram throughout this section. The numbered circles 1 through 4 correspond to the following logical architectural pieces in the following sections, and are indicated in the data flow descriptions like (1), (2), and so on:

  1. Client - Messaging Server
  2. Web Tier
  3. Service Tier
  4. Indexing Tier

ISS Data Flow
"This figure shows the flow of data through the ISS system."|width=300

Topics in this section:

Indexing and Search Service Search Request

The ISS search request works as follows:

  1. (1) IMAP clients request a search.
    For example, x SEARCH BODY "foo".
  2. (1) The ISS Gateway diverts the search request to ISS.
  3. (2) ISS RESTful web service processes search requests. For example, the following is a request received from Messaging Server:

    /rest/search?q=%2busername:user1%20%2bhostname:ms.example.com%20%2bfolder:%22INBOX%22%20%2bbody:foo&c=2147483647&contentformat=simpleuid&format=atom

  4. (3) ISS RESTful web service posts the search to the Search Service JMS Topic.
  5. (4) The searchSvc service performs the search and returns the result over the JMS topic back to the RESTful web service.
    If searchSvc is logged at the INFO level, a log entry similar to the following appears:

    query number: 3, 3 total matching documents to query: +username:user1 +hostname:ms.example.com +folder:"INBOX" +body:foo, search time was 46ms

  6. (2) ISS RESTful web service sends the result back to the Messaging Server ISS Gateway.
  7. (1) Messaging Server returns the result back to the IMAP client. For example, * SEARCH 7 53 214 indicates that three matching emails were found: UIDs 7, 53 and 214.

Other clients, such as Convergence, issue search requests directly to ISS, which enables them to provide fast cross-folder search and features such as attachment search. In this case the flow is as follows:

  1. (1) Client accesses the Attachments virtual folder inside Convergence.
  2. (2) ISS RESTful web service manages search requests.
    For example, here is an example request from Convergence, as logged in the GlassFish Server log file:
    {{/rest/search?q=%2Busername%3Auser1%40example.com%20%2Bhostname%3Ams.example.com%20%2Battachment-type%3A(atdoc%20atppt%20atxls%20atodf%20atpdf%20athtml%20atplain%20atrtf%20atxml%20atimage%20ataudio%20atvideo%20atcompress%20atvcf%20atother%20atsencr%20atssign)&c=160&s=0&thumbnail=s&format=json&contentFormat=attachmentOnly&token=rB8SHWjWMA}}
    
  3. (3) ISS RESTful web service posts the search to the Search Service JMS Topic.
  4. (4) The searchSvc service performs the search and returns the result (to RESTful search service and then to Convergence).
    If searchSvc is logged at the INFO level, a log entry similar to the following appears:

    query number: 5, 160 total matching documents to query: +username:user1 +hostname:ms.example.com +attachment-type:(atdoc atppt atxls atodf atpdf athtml atplain atrtf atxml atimage ataudio atvideo atcompress atvcf atother atsencr atssign), search time was 95ms

  5. (1) The search result is displayed to the end user.
    If an attachment search had been requested, the client can then issue ISS thumbnail requests to resolve the thumbnail URLs.

For more information, see How ISS Searches Messages.

Indexing and Search Service Thumbnail Request

The ISS thumbnail request works as follows:

  1. (1) Client receives the results of a search request.
  2. (2) Client uses the received thumbnail URL to request the thumbnail.
    Log messages such as the following appear in the GlassFish Server log file for each downloaded thumbnail:

    [#|2012-02-14T00:38:29.384+0000|INFO|sun-appserver2.1.1|com.sun.comms.iss.storeui.StoreUIServlet|_ThreadID=74;_ThreadName=httpSSLWorkerThread-8070-0;|Processing file: /var/iss/attach//01/26/h1/u120026/f10/1195248462/00/53/tbn_small_2_.jpg|#]
    [#|2012-02-14T00:38:29.384+0000|INFO|sun-appserver2.1.1|com.sun.comms.iss.storeui.StoreUIServlet|_ThreadID=74;_ThreadName=httpSSLWorkerThread-8070-0;|LOG: ip:10.79.212.11 inst:iss1 auth:FORM user:user1 mailhost:ms.example.com pathinfo:/01/26/h1/u120026/f10/1195248462/00/53/tbn_small_2_.jpg|#]

  3. (2) The ISS store servlet receives the request for the thumbnail and returns the result from disk or, in the case of a multi-machine deployment, issues a request to the Thumbnail Store Server.
  4. (4) If applicable, the Thumbnail Store Server returns the thumbnail graphic.

Initial Indexing

The initial indexing works as follows:

  1. (4) A user account is requested to be indexed via either the issadmin.sh --bootstrap command or the arrival of a message that indicates the account should be auto-provisioned (for example, the arrival of the INBOX Create event).
    In the case of an auto-provisioned account, these messages appear in JMQ consumer:

    logMessageHeaders INFO: Received event Create generated on Sat Feb 04 11:28:57 PST 2012 with sequence number 7224 URL is imap://user100@ms.example.com/INBOX;UIDVALIDITY=1328383737/;UID=null
    indexEvent WARNING: Account user100@ms.example.com autoprovisioning begun
    setAccountState INFO: Set account user100@ms.example.com state to I
    scheduleEvent INFO: Queuing event BootStrap for user100@ms.example.com generated on Sat Feb 04 11:28:57 PST 2012, with sequence number 7224

  2. (4) This command is delivered over the Index Server Queue to the IndexSvc service.
    This message is logged in JMQ consumer:

    run INFO: Account user100@ms.example.com is active on Sat Feb 04 11:28:58 PST 2012, processing event BootStrap generated on Sat Feb 04 11:28:57 PST 2012 with sequence number 7224, time between generate and submit to index svc is 1956ms.

  3. (4) A crawler within the Index Service connects to Messaging Server over the IMAP protocol and fetches all the folders and messages for the user.
  4. (4) Each message is broken down into body parts and attached files.
  5. (4) Each part is indexed into the Index Store.
  6. (4) Thumbnail images are generated and stored if supported.
  7. (4) The bootstrap is complete.
    This message is logged in JMQ consumer as follows:

    setAccountState INFO: Set account user100@ms.example.com state to A
    run INFO: Finished indexing event BootStrap for account user100@ms.example.com with sequence number 7224, time between submit to and return from index svc is 1440ms.

For more information, see How ISS Indexes Messages.

Real-time Indexing

The real-time indexing works as follows:

Each time a change is made to the Messaging Server message store, a JMQ notification message is posted by imapd or ims_master to the Event Notification Queue.

  • Changes include new mail (NewMsg, UpdateMsg), changed flags (ChangeFlag), folder operations (Rename, Create, Delete), expunged messages (ExpungeMsg), and so on.
  • The JMQconsumer service picks up these notifications and delivers them to the Index Service.
  • Each of these changes are applied to the Index and Thumbnail Store.

The following messages appear in the JMQ consumer log when a new message arrives.

logMessageHeaders INFO: Received event NewMsg generated on Mon Feb 13 00:01:00 GMT 2012 with sequence number 4154 (size h:760 e:771 t:771 - Using text) URL is imap://amam@ms.example.com/INBOX;UIDVALIDITY=1238618761/;UID=189538977
scheduleEvent INFO: Queuing event NewMsg for amam@ms.example.com generated on Mon Feb 13 00:01:00 GMT 2012, with sequence number 4154

This message appears in the JMQ consumer log when the event has been delivered to Index Service for indexing.

run INFO: Account amam@ms.example.com is active on Mon Feb 13 00:01:00 GMT 2012, processing event NewMsg generated on Mon Feb 13 00:01:00 GMT 2012 with sequence number 4154, time between generate and submit to index svc is 390ms.

This message appears in the JMQ consumer log when the message was successfully indexed.

run INFO: Finished indexing event NewMsg for account amam@ms.example.com with sequence number 4154, time between submit to and return from index svc is 238ms.

The indexing of this event can also be tracked in the Index Service log, which logs the following in response to this event:

logmsg INFO: Received event NewMsg on account: amam@ms.example.com with sequence number 4154, current backlog in queue: 0
updateCounts INFO: FolderCount.updateCounts: host=ms.example.com user=amam updating folder: INBOX to 1476798/1476798/0 old mvalue:1476797 lastUid value:null

Labels:
indexsearchservice indexsearchservice Delete
Enter labels to add to this page:
Please wait 
Looking for a label? Just start typing.

Sign up or Log in to add a comment or watch this page.


The individuals who post here are part of the extended Oracle community and they might not be employed or in any way formally affiliated with Oracle. The opinions expressed here are their own, are not necessarily reviewed in advance by anyone but the individual authors, and neither Oracle nor any other party necessarily agrees with them.