Creating New Indexing and Search Service Accounts

Skip to end of metadata
Go to start of metadata

Creating and Managing Indexing and Search Service Accounts

After installing ISS, you must create, or "bootstrap" each account into the ISS store. You can do this by either manually running the issadmin.sh --bootstrap command or by enabling account autoprovisioning. This information describes how to use each method, and the benefits of each.

Topics:

Manually Bootstrapping ISS Accounts

Manually creating, or "bootstrapping" accounts, gives you more control over how you allocate accounts. When you create an account by using the issadmin.sh command, you can select either the group or singleton status for an account, by using the --group and --singleton options. You can also use the --accountlist option to specify a file containing account information for one or more accounts used to repeat the command action over multiple accounts.

Topics in this section:

To Bootstrap an ISS Account (Default Allocation Policy)

  1. Log in as superuser (root) or the ISS user.
  2. Run the issadmin.sh --bootstrap script with the --user and --host options.
    For example:

By default, the script uses the system administration information configured in mail.imap.admin.username from the iss-base/etc/jiss.conf file and mail.imap.admin.password from the iss-base/etc/jiss_passwd.conf file to establish access to the account on the Messaging Server message store.

If you specify only --user and --host options, the ISS store assigns the account being bootstrapped to a group by using the default allocation policy. See ISS Store Instance Overview and Concepts for more information about the default allocation policy.

To Bootstrap an ISS Account (Allocating to a Specific Singleton Group)

The default allocation policy is convenient for a deployment that consists of small numbers of small accounts. For more control over account allocation, create accounts beforehand by using the issadmin.sh --createaccount option. Alternatively, you can use the issadmin.sh --bootstrap script with the --group option and the --singleton option to direct the bootstrap to use a specific group number and to require the account to be created as the only account in the group, respectively.

  1. Log in as superuser (root) or the ISS user.
  2. Run the issadmin.sh --bootstrap script with the --group and --singleton options.
    For example:

To Bootstrap an ISS Account by Using the --accountlist Option

The --accountlist FILE option enables you to specify a file containing account information for one or more accounts used to repeat the command action over multiple accounts. In this case, when used with the --bootstrap option, it executes the command over a list of accounts, all within one executable command.

  1. Log in as superuser (root) or the ISS user.
  2. Run the issadmin.sh --bootstrap script with the --accountlist FILE option.
    For example:

Autoprovisioning ISS Accounts

Autoprovisioning is available starting with Indexing and Search Service 1 Update 3 Patch 9.

Topics in this section:

Overview of ISS Autoprovisioning

Autoprovisioning enables you to add new accounts to the ISS store automatically. By using autoprovisioning, you make account creation less prone to error than when manually bootstrapping accounts. However, unlike the issadmin.sh command, which gives you control over the group or singleton status for an account, autoprovisioning simply creates the account in the next available index group in the ISS store by using the default allocation policy See ISS Store Instance Overview and Concepts for more information about the default allocation policy.

Caution
Before using autoprovisioning, be sure to understand how it works to avoid creating undesirable effects on the ISS store.

When autoprovisioning is enabled, the system creates a new ISS account as follows:

  1. An administrator creates a new email account in the Messaging Server message store.
  2. When activity first occurs for the email account, indicating that the account is active, a JMQ notification is created. For example, when mail first arrives for this email account, the standard email folders such as INBOX are created in the message store, which in turn creates a notification.
    Note
    Autoprovisioning only applies when ISS sees a create event on the INBOX. Autoprovisioning does not occur if you restore an account with the imsrestore command.
  3. The Indexing and Search Service jmqconsumer process, which is listening to Messaging Server Java Message Queue, receives the event notification and triggers autoprovisioning to occur for the account.
  4. The ISS account is created in the next available index group in the ISS store by using the default allocation policy.
Note
If you maintain a list of your ISS accounts, understand that this list can become out of sync due to how autoprovisioning works. When autoprovisioning is enabled, the system can create new accounts in the ISS store at any time. Thus, your list can become incomplete. Your list can also become incomplete whenever an account is explicitly deleted from the store. To make sure that your list is current, use the issadmin.sh --listaccountlistfile command to obtain all accounts currently in the ISS store. Formatting of this output is enhanced if you use the --accountlist option.

Enabling ISS Autoprovisioning

Autoprovisioning is controlled at the global level through the iss.autoprovision.enabled configuration parameter. Once the iss.autoprovision.enabled parameter has been set to true, when you add new email accounts and those accounts become active, ISS automatically provisions accounts for them in the ISS store.

To Enable ISS Autoprovisioning

  1. Edit the iss-base/etc/jiss.conf file and set the iss.autoprovision.enabled configuration parameter to true.
  2. Refresh the ISS configuration for the change to take place.

Selectively Choosing Which Accounts Are Autoprovisioned for ISS

When autoprovisioning is enabled, the default setting creates any account for which the trigger event is received. However, for a variety of reasons, you might want to restrict the set of accounts for which autoprovisioning creates an account. For example, in a multiple ISS host deployment, you might want to autoprovision accounts across the hosts for load balancing purposes. Also, you don't want to create duplicate accounts on each host.

To restrict how ISS automatically provisions accounts across multiple hosts, ISS provides the following four account selection configuration parameters to control this process:

iss.autoprovision.host.include.list
iss.autoprovision.user.include.list
iss.autoprovision.host.exclude.list
iss.autoprovision.user.exclude.list

Each iss.autoprovision parameter takes a value specified as a list of regular expressions. These parameters control which host names and user names are included and excluded from autoprovisioning on each ISS instance.

To Choose Which Accounts Are Autoprovisioned for ISS

  1. Edit the iss-base/etc/jiss.conf file and set the iss.autoprovision.* configuration parameters with the appropriate regular expression.
    ISS employs the following rules when evaluating these expressions:
    • Each expression is compared to the name of the account before it is autoprovisioned.
    • The host name of the account must match one of the "host.include" expressions, and none of the "host.exclude" expressions, to be autoprovisioned.
    • Similarly, the user name must match one of the "user.include" expressions, and none of the "user.exclude" expressions, to be autoprovisioned.
    • Each "include" expression is checked before any "exclude" expressions. Thus, if no match of any "include" expression occurs, the account is not autoprovisioned, and the "exclude" checks are unneeded.
    • The expression lists are applied left to right, so place expressions with more general patterns before expressions with more specific matching.
      See http://download.oracle.com/javase/6/docs/api/java/util/regex/Pattern.html for more information on writing regular expressions.
  2. Refresh the ISS configuration for the change to take place.

Using Account Selection Parameters

You can use the include and exclude configuration parameters to restrict the set of accounts to be autoprovisioned on the ISS instance. If, for example, you want to avoid autoprovisioning all accounts in a domain named abc.com, you could set the following configuration parameter in each ISS instance configuration file for each instance where the domain should be excluded:

iss.autoprovision.user.exclude.list = .+@abc.com

Many more include and select configurations are possible.

ISS Autoprovisioning Example

Consider a deployment of two Messaging Server hosts hostAM and hostNZ, and two ISS instances named issAM and issNZ. Assume all existing users whose names start with the letters "a" through "m" are hosted on hostAM and bootstrapped on issAM. All users whose names start with letters "n" through "z" are hosted on hostNZ and bootstrapped on issNZ. Further, assume that issAM is configured only to receive event notifications from hostAM and issNZ is configured to receive only event notifications from hostNZ.

To configure autoprovisioning for this example, you would set the iss.autoprovision.enabled parameter to true in the jiss.conf file on both issAM and issNZ then refresh the ISS configuration. (If this parameter were enabled on only one ISS instance, then only new accounts directed to that instance would be autoprovisioned.) The default values for the other autoprovisioming parameters allow for any account to be autoprovisioned, so no additional selection regular expressions need to be added.

Every new account added to Messaging Server hosts hostAM or hostNZ is autoprovisioned only in the corresponding ISS store because the event notifications sent for each account are received in only one ISS instance.

Complex Autoprovisioning Example

The previous example contains a fairly simple configuration. Now consider a more complicated approach in which both Messaging Servers hosts send event notifications to both ISS instances. In this example, the selection parameters are configured to distribute new accounts to either ISS instance for autoprovisioning.

Note
If an account is not created manually or autoprovisioned on an ISS instance, any events received for that account are ignored.

In this case, the selection criteria for which accounts to autoprovision on each ISS instance must partition the set of all accounts such that each account is indexed on exactly one of the two ISS instances. Following is one way to partition this example:

  • On host issAM:
    iss.autoprovision.user.include.list = [a-m].+
    
  • On host issNZ:
    iss.autoprovision.user.include.list = [n-z].+
    

This configuration allows only account names starting with letters "a" through "m" to be autoprovisioned on instance issAM, and only those starting with letters "n" through "z" to be autoprovisioned on instance issNZ. Every account matches on exactly one instance, and all possible accounts are included somewhere.

Now consider how, over time, instance issAM has many more accounts than issNZ. To help balance the load, you decide to autoprovision new accounts starting with the letters "d" and "j" on issNZ from now on. The configuration parameters can be changed as follows:

  • On host issAM:
    iss.autoprovision.user.include.list = [a-m].+
    iss.autoprovision.exclude.include.list = d.+, j.+
    
  • On host issNZ:
    iss.autoprovision.user.include.list = [n-z].+, d.+, j.+
    

Several other possibilities exist to achieve this change with regular expressions. This example shows the utility of the exclude parameter. This configuration change does not effect the location of previously indexed accounts, just new ones autoprovisioned after the configuration is refreshed. Both ISS instances eventually contain accounts starting with the letters "d" and "j" as new accounts are autoprovisioned.

How ISS Autoprovisioning Handles Errors

Topics in this section:

Errors in Regular Expressions

ISS checks the autoprovisioning parameters that specify regular expression lists for proper syntax when the configuration is refreshed. If any of the expressions are malformed, an error message is logged, and that element of the list is omitted, allowing the remaining expressions to be used to include or exclude accounts. This evaluation could result in unintentionally autoprovisioning or excluding some user or host names, but does allow the processing to continue.

However, while each of the regular expressions may be well formed, account names or hosts may not be matched as intended. No checking is performed so that all accounts either match any include or exclude criteria. Thus, you need to ensure that all account names and hosts for which autoprovisioning is intended match some include criteria, and only accounts to be excluded match the exclude criteria. If a trigger event for an account arrives, and the account fails to match any include criteria or matches any exclude criteria, then the account is not autoprovisioned, and a WARNING level message is logged. If you find that accounts are not being autoprovisioned as expected, these message can help you understand whether the selection parameters are wrong.

Time Out Errors

Because autoprovisioning requires creating then bootstrapping an account, this processing is expensive relative to most other events, such as changing flags, and so on, and requires more synchronization than usual. Normally this completes in a few seconds. However, when the rate of event processing is high, an attempt to autoprovision a new account might time out, and not complete, to avoid delaying other event processing.

This time out failure might leave the account in an unfinished state, preventing new events processing for the account. Starting with version 1.0.5.19.0, ISS attempts to retry the autoprovisioning that timed out automatically in a separate thread, to limit interference with other processing. This retry might be repeated a few times if the load is high. In general, this retry enables the autoprovisioning to complete. A retry needs some added minutes to complete after the original time out occurred. A WARNING message is logged when a time out occurs, and additional log messages show the progress of any retry.

ISS Autoprovisioning Accounts Best Practices

The previous examples illustrate the use of the selection configuration parameters. In practice, your deployment will likely have much more than two Messaging Server hosts and ISS instances, so your autoprovisioning configuration will be much more complex.

Keep the following items in mind when using autoprovisioning:

  • As a general rule, keep the selection criteria as simple as practical to ensure that the full range of account names and hosts is covered.
  • If you disable autoprovisioning on any specific instance, check that any accounts it had been autoprovisioning are reflected in the configuration of other instances.
  • Because the jiss.conf configuration file for each ISS instance is local to the instance, keep a copy of the autoprovisioning parameters for all instances in a global file so that the selection criteria for new accounts can be easily understood and changes can be tracked.

Administering Periodic Automatic Synchronization for Indexing and Search Service Accounts

This option is available starting with Indexing and Search Service 1 Update 4.

Topics in this section:

Overview of Periodic Automatic Synchronization

As the Messaging Server message store changes, it sends event notifications to Indexing and Search Service (ISS) so that the index information for each account can be updated. At times, changes to the message store might not get reflected in the ISS store. For example, a folder in an account might be reconstructed in the message store, changing the assignment of UID numbers to emails, without notification to ISS. While ISS now can detect the UID validity mismatch during subsequent event processing, and re-bootstraps the folder to get it back in sync, a long time can pass before an event triggers the correction. Indeed, there is no guarantee any event involving the reconstructed folder is ever received, so the mismatch remains uncorrected, and searches on such a folder continually produce invalid results. Also, under very heavy load, notification messages can be delayed or lost, causing the index to become out of sync with the message store.

To correct the index after such unseen changes to the message store, ISS provides a periodic check of each account against the message store. If any account is found to be not in sync, ISS performs the necessary --sync or --bootstrap commands to correct the index automatically. ISS refers to this process as periodic autosync.

Enabling Periodic Autosync

Periodic autosync is controlled at the global level through the iss.indexsvc.periodic.autosync.enabled configuration parameter. Once the iss.indexsvc.periodic.autosync.enabled parameter has been set to true, ISS creates a list of all known active accounts in the index, and orders them in a list to be processed. During each time period, as specified by the configuration parameter iss.indexsvc.periodic.autosync.interval, a set of accounts is removed from the list and checked against the message store then synchronized if needed. The number of accounts in the set is determined by the value of the configuration parameter iss.indexsvc.periodic.autosync.count. After enough periods have completed to exhaust the list of accounts, the list is recreated and the periodic autosync continues again. In this way all accounts are eventually checked and synchronized over a period of time under control of the parameter choices.

To Enable Periodic Autosync
  1. Edit the jiss.conf file and set the iss.indexsvc.periodic.autosync.enabled configuration parameter to true.
  2. (Optional) Set the iss.indexsvc.periodic.autosync.interval configuration parameter to specify the number of seconds to wait from the start of one period until the start of the next autosync period.
    The default is 300 seconds.
  3. (Optional) Set the iss.indexsvc.periodic.autosync.count configuration parameter to specify the number of accounts to process from the list of accounts yet to be synced during each period.
    The default is 1000.
  4. Refresh the ISS configuration for the change to take place.
  5. For tuning suggestions, see Other Tuning Controls of Periodic Autosync.

Selecting Appropriate Periodic Autosync Configuration Values

The periodic autosync of accounts imposes overhead on both the ISS and Messaging Server hosts. Values for the configuration parameters described here must be selected to balance the tradeoff between rapid cycling through each account and the cost of load on the system, which can increase response time. The total time to check every active account in the ISS store can be estimated by dividing the number of such accounts by the value of the "count" parameter and multiplying by the "interval". (This assumes that "count" accounts can always be checked or synchronized in "interval" seconds.) For example, if there are 10000 active accounts, and "count" is 100 and "interval" is 600 seconds (10 minutes), all accounts are checked in approximately 10000/100 * 600 = 60,000 seconds, or 16.7 hours.

The amount of processing it takes to do autosync depends on many factors, including:

  • How many accounts are checked in a group
  • How many emails each contains
  • How many accounts are found to have problems that need to be synchronized (if any)

You cannot tell in advance what load this imposes on either the ISS or Messaging Server hosts. As the "count" value is increased and the "interval" value is reduced, the elapsed time to check all accounts decreases, but the average overhead imposed on the servers increases.

Therefore, set the initial values for "count" and "interval" so that the rate of processing is no faster than about 1 account every 5 seconds. (The rate in the preceding example shows 100/600 or about 1 every 6 seconds.) By running at such a conservative rate for a while, the overhead on the ISS and Messaging Server hosts can be monitored, and after some experience, faster rates can be attempted until the appropriate balance can be established.

The periodic autosync feature writes INFO messages at regular times to the log file. You can use this information to see how many accounts have been checked, how many were valid (that is, did not need to be synchronized), and how many accounts were synchronized because of a problem. From this information and the amount of idle time between periods (seen in the log as how long the autosync sleeps at the end of a period), you can adjust the "count" and "interval" to the desired frequency.

Periodic Autosync Configuration Examples

Topics in this section:

Large Interval, Small Count, Significant Idle Time

The previous example (10000 accounts, 100 checked per period, 600 seconds between periods) demonstrates a relatively slow rate of processing that should have low impact on both the ISS and message stores. If the time it takes to check the accounts in each period is short compared to the interval between periods, then the load from the autosync is "pulsed," that is, higher for a period of time (while the accounts are being checked) and lower when the period is done and is waiting for the next one to begin. If the load during the working phase of the period is within acceptable bounds for your deployment, the "interval" value can be reduced to shorten the wait. This could reduce the overall elapsed time significantly, and maintains a steadier load on the hosts.

Small Interval, Large Count, No Idle Time

A more aggressive configuration (10000 accounts, 500 checked per period, 300 seconds between periods) demonstrates how the load might increase. If all the "count" accounts cannot be checked withing the "interval," then the autosync runs continuously with no wait time between periods. This may be acceptable if your deployment has enough resources to handle the load. Under these conditions all accounts are checked in 10000/500 * 300 = 6000 seconds or 1.7 hours compared to 16.7 with the previous example if the 500 accounts can be checked within 300 seconds. This might be possible under some conditions, such as many accounts with relatively small numbers of emails, or during periods of relatively low activity so the contents of the accounts does not change much. With such a configuration, the load on the system is much more sensitive to arrival of new email or heavy user search activity.

Other Tuning Controls of Periodic Autosync

Topics in this section:

Controlling the Number of Threads to Use

The number of threads used to run the checks affects the overhead and speed of autosync processing. To set the number of threads used to run the checks, use the iss.indexsvc.periodic.autosync.thread.count configuration parameter. The default value is 10 threads. By decreasing the number of threads used, the elapsed time of processing within the period tends to increase, spreading the load on both the ISS and Messaging Server hosts over a longer period of time. Increasing the number of threads increases the amount of work done in parallel, which tends to increase the instantaneous load, but decreases the elapsed time to complete all the checks in the period.

Controlling the Level of Account Checking

The --checkaccount and --sync commands that implement autosync processing can be modified by the iss.indexsvc.periodic.autosync.flags.enabled parameter to perform more or less detailed checking of accounts. When this parameter has the value true, autosync behaves like the --detail FLAGS option has been specified for each account. This provides more detailed checking (that is, detects more out-of-sync problems) at the cost of additional overhead to both ISS and Messaging Server. If the overhead is excessive, set the value of this parameter to false to reduce the cost of using autosync. By default, the value for this parameter is true.

Monitoring Autosync Progress

The Index Service logs INFO level messages that describe the autosync progress. Whenever the cycle begins, the list of all accounts participating in the autosync is generated, and messages such as the following are logged:

Sat Jul 07 18:39:22 PDT 2012 com.sun.comms.iss.store.AutoSyncManager findAllCandidates INFO: findAllCandidates: time to create list of 53995 candidates : 339ms
Sat Jul 07 18:39:22 PDT 2012 com.sun.comms.iss.store.AutoSyncManager findAllCandidates INFO: findAllCandidates: current number of accounts found by state: total: 60000 active: 53995 inactive: 3 bootstrap: 2 unknown: 6000

The breakdown of accounts by account state can help identify problem accounts, such as those left in the bootstrap, inactive, or unknown state.

When a period of work begins, ISS logs how many accounts it is processing as a group. For example:

Sat Jul 07 23:14:53 PDT 2012 com.sun.comms.iss.indexapi.IndexService runAutoSyncLoop INFO: Periodic autosync summary: processing 250 accounts every 300 seconds will complete 46745 accounts currently queued in about 56100 seconds (15.5 hours)

If the time estimate for completing all the active accounts is too large, consider changing the configuration parameters iss.indexsvc.periodic.autosync.interval or iss.indexsvc.periodic.autosync.count to speed up the cycle.

As the next period of work begins processing, the names of all accounts to be checked in the period are logged. For example:

Sat Jul 07 23:14:53 PDT 2012 com.sun.comms.iss.indexapi.IndexService runAuto INFO: 250 autosync candidates found: all users share host sc11a2017.us.oracle.com unless explicitly present:
 candidate: 3010340@colorone.org, 3010346@colorone.org, 3010348@colorone.org, 3010313@colorone.org, 3010324@colorone.org, 3010350@colorone.org, 3010353@colorone.org, 3010344@colorone.org, 3010357@colorone.org, 3010322@colorone.org
 candidate: 3010351@colorone.org, 3010364@colorone.org, 3010363@colorone.org, 3010362@colorone.org, 3010367@colorone.org, 3010360@colorone.org, 3010342@colorone.org, 3010372@colorone.org, 3010347@colorone.org, 3010358@colorone.org
...
 candidate: 3010666@colorone.org, 3010679@colorone.org, 3010676@colorone.org, 3010661@colorone.org, 3010678@colorone.org, 3010674@colorone.org, 3010686@colorone.org, 3010685@colorone.org, 3010688@colorone.org, 3010691@colorone.org

Additional log messages appear as each individual account is checked or synced. Every 10 periods, a summary of progress since the beginning is logged. For example:

Sat Jul 07 22:43:49 PDT 2012 com.sun.comms.iss.store.AutoSyncManager logAutoSyncStats INFO: statistics: number of accounts checked: 6250 number of accounts OK: 5451 number of accounts synced: 799 number of accounts skipped because another command was running: 0 number of accounts skipped because folder not found: 0 number of accounts skipped because authentication failed: 0 number of accounts failed for other reasons: 0

These totals continue to accumulate over multiple cycles of active accounts. If the number of accounts synced is low relative to the number of accounts that are "OK," consider modifying the configuration parameters to lengthen the time to complete the full cycle of accounts to reduce the overhead due to autosync.

Automatically Bootstrapping Missing Accounts

Similar to the situation for active accounts described in Overview of Periodic Automatic Synchronization, accounts which have not been bootstrapped or autoprovisioned might cause event notifications of changes to such accounts to fail. To correct these accounts, you can use the autobootstrapping feature, which is useful if you want your deployment to index uninitialized accounts under such conditions.

Enabling Periodic Autobootstrap

Periodic autobootstrap is controlled at the global level through the iss.indexsvc.periodic.autobootstrap.enabled configuration parameter. When iss.indexsvc.periodic.autobootstrap.enabled is set to true, ISS tracks errors that occur in event notification events to create a list of accounts that should be bootstrapped. Errors are counted per account. Any event that fails because the account cannot be found is counted, and if enough such events occur, then the account is added to the list to be automatically bootstrapped. To avoid excessive load from the bootstrap process, accounts on this list are scheduled periodically to be bootstrapped at regular intervals, as specified by the {iss.indexsvc.periodic.autobootstrap.interval}} configuration parameter. The number of accounts on the list bootstrapped during each period is determined by the value of the iss.indexsvc.periodic.autobootstrap.count configuration parameter. Once an account has been bootstrapped it is removed from the list, and event errors caused because its index was missing should no longer occur.

To Enable Periodic Autobootstrap
  1. Edit the jiss.conf file and set the iss.indexsvc.periodic.autobootstrap.enabled configuration parameter to true.
  2. (Optional) Set the iss.indexsvc.periodic.autobootstrap.interval configuration parameter to specify the number of seconds to wait from the start of one period until the start of the next autobootstrap period.
    The default is 300 seconds.
  3. (Optional) Set the iss.indexsvc.periodic.autobootstrap.count configuration parameter to specify the number of accounts to process from the list of accounts yet to be bootstrapped during each period.
    The default is 500.
  4. Refresh the ISS configuration for the change to take place.

Selecting Conditions for Events to Track in Periodic Autobootstrap

You can choose which error conditions are used to select accounts for autobootstrapping, based on the kind and number of events that fail. To control what kind of events can trigger automatic bootstrapping of accounts, the configuration parameter iss.indexsvc.periodic.autobootstrap.triggerlist enables you to specify that all, none, or some specific kinds of event errors should be counted toward autobootstrapping. The default value of this parameter is ALL or empty, indicating all event errors are counted to trigger autobootstrapping. If the value is NONE, then no event errors are counted. You can also specify a comma-delimited list of specific event names, taken from the following fixed set:

NewMsg, UpdateMsg, ChangeFlag, Copy, Create, Delete, ExpungeMsg, Rename

Only event errors for any of the event kinds specified are counted. (All values specified in this parameter are treated as case insensitive.)

Once the kind of event errors used to trigger automatic bootstrapping has been established, the count of how many
events before the account is submitted for autobootrapping is determined by the value of the configuration parameter iss.indexsvc.periodic.autobootstrap.triggercount. The default value is 1.

Other Tuning Controls of Periodic Autobootstrap

Topic in this section:

Controlling the Number of Threads to Use for Autobootstrap

The number of threads used to run the bootstrap on multiple accounts affects the overhead and speed of autobootstrap processing. To set the number of threads used to run the bootstraps, use the iss.indexsvc.periodic.autobootstrap.thread.count configuration parameter. The default value is 10 threads. By decreasing the number of threads used, the elapsed time of processing within the period tends to increase, spreading the load on both the ISS and Messaging Server hosts over a longer period of time. Increasing the number of threads increases the amount of work done in parallel, which tends to increase the instantaneous load, but decreases the elapsed time to complete all the bootstraps in the period.

Managing the Autobootstrap Account List

Both the periodic autosync and autobootstrap features maintain a list of accounts to be processed. Unlike the autosync account list which is volatile, the autobootstrap account list is preserved across service restart so the accounts can continue to be bootstrapped when the server starts again. This list can also be updated by using the issadmin.sh commands to list, delete, and add accounts to the list to manage the autobootstrap processing dynamically as it progresses. See the issadmin.sh commands --autobootaccounts, --deleteautobootlist, --setautobootlist, and --unsetautobootlist for how to review and modify the list of accounts to be autobootstrapped.

Monitoring Autobootstrap Progress

The Index Service logs INFO level messages that describe the progress of the autobootstrap. Whenever a new work period begins, the list of all accounts participating in the autobootstrap then is generated, and messages are logged such as the following:

Thu Jul 05 11:51:27 PDT 2012 com.sun.comms.iss.store.AutoSyncManager selectPeriodicBootstrapCandidates INFO: selectPeriodicBootstrapCandidates: 134 accounts checked, 125 accounts selected
Thu Jul 05 11:51:27 PDT 2012 com.sun.comms.iss.indexapi.IndexService runAutoBootLoop INFO: 5500 accounts currently awaiting bootstrap

This example shows the number of accounts checked is greater than the number selected. This indicates that some accounts in the original list were found to exist already, and so were not bootstrapped, or that an account had been added to the list multiple times. This can happen as accounts can be created through autoprovisioning or manually while waiting to be autobootstrapped.

The specific accounts to be bootstrapped in this period are then logged as follows:

Thu Jul 05 11:51:27 PDT 2012 com.sun.comms.iss.indexapi.IndexService runAuto INFO: 125 autoboot candidates found: all users share host sc11a2017.us.oracle.com unless explicitly present:
 candidate: 3026464@colorone.org, 3016948@colorone.org, 3005936@colorone.org, 3026633@colorone.org, 3048656@colorone.org, 3026604@colorone.org, 3059277@colorone.org, 3016072@colorone.org, 3048919@colorone.org, 3016203@colorone.org
 candidate: 3048186@colorone.org, 3026696@colorone.org, 3026869@colorone.org, 3059255@colorone.org, 3005852@colorone.org, 3005295@colorone.org, 3005727@colorone.org, 3026976@colorone.org, 3059929@colorone.org, 3059432@colorone.org
...
 candidate: 3016352@colorone.org, 3048236@colorone.org, 3037843@colorone.org, 3005418@colorone.org, 3016549@colorone.org, 3048754@colorone.org, 3037213@colorone.org, 3005932@colorone.org, 3016985@colorone.org, 3059413@colorone.org
 candidate: 3059013@colorone.org, 3005853@colorone.org, 3016487@colorone.org, 3048117@colorone.org, 3059800@colorone.org

Additional log messages appear as each individual account is bootstrapped. When all the accounts in the group have been bootstrapped, another group is begun after the configured interval has elapsed, or started immediately if the time to complete the group exceeds the interval.

Automatically Removing Orphaned ISS Accounts

The features documented in this section were introduced in Indexing and Search Service 1.0.5.18.0.

Topics in this section:

Automatically Removing Orphaned ISS Accounts Overview

An ISS account that no longer appears to have a corresponding Messaging Server account is "orphaned." This situation can occur, for example, when you move Messaging Server user accounts between message stores, for load balancing purposes, without updating the ISS service. The corresponding ISS accounts are orphaned, because they are not automatically moved with the Messaging Server accounts. (Refer to the use of the issrehostuser.sh scripts to move the ISS account when the Messaging Server rehostuser command is used.) Thus, the ISS account remains in its original location, out of sync with the Messaging Server account that moved.

Prior to Indexing and Search Service 1.0.5.18.0, correcting an orphaned account requires manually updating the ISS store by moving the account (by using the --export and --import commands), or by deleting the old account then bootstrapping it on the new store. Indexing and Search Service 1.0.5.18.0 provides an automated, configurable mechanism to remove the orphaned account from the proper ISS store. Autoprovisioning could then be used to index the account on its new host (with either autobootstrap, as described in Enabling Periodic Autobootstrap or with autosync, as described in Enabling Periodic Autosync).

To remove orphaned accounts automatically, you must set both the global autosync parameter, iss.indexsvc.periodic.autosync.enabled, and the iss.indexsvc.periodic.autodelete.enabled parameter (available starting with Indexing and Search Service 1.0.5.18.0), to true. The iss.indexsvc.periodic.autosync.enabled parameter enables ISS to detect orphaned accounts. The ISS autosync process scans for accounts that are not in the Active state, as they are potentially orphaned. The accounts are placed on a list of candidates to check for autodelete. The iss.indexsvc.periodic.autodelete.enabled parameter enables ISS to remove candidate accounts from the store periodically.

This list of potentially orphaned accounts to be deleted is processed periodically, based on the value of the configuration parameter iss.indexsvc.periodic.autodelete.interval. During each autodelete period, the list of accounts to be deleted is scanned. If all conditions for autodelete are satisfied for an account, that account is deleted from the store. Conditions can be set for autodelete and delay time to allow for administrator intervention.

During an autodelete scan, each candidate account is checked to confirm that it cannot be found in the Messaging Server. If the account cannot be found, then the last transaction date of the account is compared to the value in the configuration parameter {iss.indexsvc.periodic.autodelete.purge.interval}} to determine if the account has been recently modified. If the account has been modified more recently than the purge interval, then the account is returned to the candidate list to be checked again during the next scan. If the account has not been modified, it is deleted from the ISS store.

To Configure Indexing and Search Service to Remove Orphaned Accounts Automatically

The configuration parameter iss.indexsvc.periodic.autosync.enabled controls whether ISS automatically deletes orphaned accounts. To enable the autodelete feature, the value for iss.indexsvc.periodic.autosync.enabled must be true. When the value is false, the autodelete feature is disabled.

  1. In the iss-base/etc/jiss.conf file, verify that the value of the iss.indexsvc.periodic.autosync.enabled parameter is set to true (default is false).
  2. In the iss-base/etc/jiss.conf file, verify that the value of the iss.indexsvc.periodic.autodelete.enabled parameter is set to true (the default).
  3. Use the iss.indexsvc.periodic.autodelete.interval parameter to specify the interval in seconds between autodelete scans. During each autodelete scan, ISS reviews the list of accounts to be deleted. If all conditions for autodelete are met, ISS deletes an account from the store. By adjusting the conditions for autodelete and the time between scans, you can allow extra time to make additional changes to the accounts that are candidates for autodelete.
  4. Use the iss.indexsvc.periodic.autodelete.purge.interval to specify the purge interval in seconds. This interval is the time that must have elapsed since the last modification to an account.
  5. Refresh the ISS configuration for your changes to take effect:

To Disable Automatic Removal of Orphaned Accounts

If you set either the iss.indexsvc.periodic.autosync.enabled parameter or iss.indexsvc.periodic.autodelete.enabled parameter to false, then the autodelete feature is disabled. Setting iss.indexsvc.periodic.autodelete.enabled to false disables only the autodelete processing, not the candidate detection. Setting iss.indexsvc.periodic.autosync.enabled to false disables all autosync processing, including the candidate detection, so no more accounts are autodeleted.

  1. In the iss-base/etc/jiss.conf file, verify that the value of the iss.indexsvc.periodic.autodelete.enabled parameter or the iss.indexsvc.periodic.autosync.enabled parameter is set to false, depending on how you want to disable autodelete processing.
  2. Refresh the ISS configuration for your changes to take effect:

Managing Out-of-Sync State Information

The features documented in this section were introduced in Indexing and Search Service 1.0.5.18.0.

Topics in this section:

Integrating JMQConsumer Information with autosync

The autosync feature integrates information from JMQConsumer to decide how accounts are to be checked and sync'd.

Two changes reduce the time required to complete the sync process for accounts known to be out of sync:

  • An event for an account may be dropped by JMQConsumer when the size of its events backlog reaches the maximum allowed, or when other internal data limits are reached. When an event is dropped for an account, autosync is notified and the account is identified to be out of sync. This means that the account can be sync'd without the initial --checkaccount step.
  • The --checkaccount command used by autosync halts when any folder is first detected to be out of sync. This avoids the overhead involved in checking the rest of the account before starting the sync process.

An event may be skipped in JMQConsumer when the processing of the account detects that the event has already been incorporated, for example, by a recent --sync or --bootstrap command. Skipping events that are already incorporated avoids unnecessary overhead and helps reduce the event backlog quickly.

The autosync feature sends information about the sync of an account to JMQConsumer, including the time of the sync and the NEXTUID value for each folder, which indicates when and how the folder last changed. This allows JMQConsumer to skip any events in the account backlog that occurred before the sync, since these have already been incorporated into the index for the account.

Output from the periodic autosync in the Index service log reflects the current rate of progress more accurately. The estimated time for completing the accounts remaining on the autosync candidates list is based on the actual time required for the previous period instead of the configured interval (which may be much less than the actual time).

Statistics for periodic autosync have been added to the Index server statistics file. Values tracked include the number of candidates added (from JMQConsumer dropping events), the number of accounts checked and sync'd, the number of accounts that were checked and were already in sync (so no sync was needed), the number of candidate accounts that were skipped instead of being checked/sync'd, and various types of failures in the sync process. These statistics are updated periodically with the other Index server statistics to provide additional insight into how often accounts need to be sync'd.

Accessing Account Event Backlog Information

Information about the backlog of unprocessed events held in the JMQConsumer service includes more details per account, a more detailed periodic summary, and a command-line query feature for quickly selecting details about specific subsets of the backlog information.

Logging Periodic Account Event Backlog Information

The periodic backlog queue checking in JMQConsumer now runs in an independent thread that loops at regular intervals instead of running in the same thread that reads new messages from the broker. The time interval between sampling is more regular, overhead is reduced, and the reading thread is not blocked while it processes the event backlog.

The output from the periodic checking of the event queue backlog in the JMQConsumer service is now redirected into separate statistics log files, similar to how statistics are generated in the Index and Search Server logs. The information recorded includes more details about the backlog. The accounts listed are now grouped by state, so that all accounts in the Active state are listed together, as are those in the Inactive, Bootstrap, and Unknown states. For individual accounts, counts are kept of the number of events dropped and skipped.

The summary of the backlog now includes the number of dropped events, with ChangeFlag events and others counted separately. Totals for the number of events received, finished, failed, and skipped, as well as the number of events received for which there was no account found, are logged periodically, including the instantaneous rate per second for each statistic.

Using issadmin.sh to Query Account Event Backlog Information

The periodic logging of the backlog displays the state of all accounts for which there are events waiting to be processed at regular intervals.

Alternatively, you can use issadmin.sh --listbacklog to dynamically query specific accounts that require follow-up. With no other arguments, the issadmin.sh command displays information for the entire backlog that is similar to what is periodically logged. To sort specific data from the large backlog output, you can use the following selection mechanisms:

  • To select by account name: Use the --user and --host options to select a single account, or the --accountlist option with a file containing multiple account names. (Although the --threads and --continueonerror options can be specified with the --accountlist option, these options have no effect when used with the --listbacklog command.)
  • To select by state of the account: Use the --selectstate option to specify a comma-delimited list of states from the set A, B, I, and X, representing Active, Bootstrap, Inactive, and Unknown respectively. The output shows the counts of events that were skipped and dropped for each account.
  • To select by age of events: Use the --selecttime H[:M] option to specify the number of hours and minutes that an event must have been in the backlog before it is considered a match. Use this option to find accounts with older events while ignoring accounts whose events have been queued recently. To specify a time less than one hour, use zero for the number of hours.

Use these selection mechanisms together to return results that contain only accounts with events in their backlog that satisfy all the selection criteria.

Note
The --listbacklog command can only be used when the services are running.

Rehosting and Deleting ISS Accounts

When rehosting accounts between mail servers, the source account is deleted from one Messaging Server message store and created in another. The issrehost* scripts in the ISS examples directory are provided to automate the corresponding process of rehosting the ISS index between ISS stores. The issrehost process includes deleting the old account in the index and creating the new one. These scripts properly sequence the steps to ensure that the ISS account information is created.

Neither the rehostuser nor imsrestore Messaging Server commands cause an event notification that triggers autoprovisioning. When you use these commands, take care to explicitly create any new ISS accounts on the destination ISS store by using the steps in the issrehost* scripts, or by using the issadmin.sh --createaccount and --bootstrap commands to index such accounts.

Whenever an account is removed from the Messaging Server message store (by using the mboxutil -d command), the data in all folders is deleted, and event notifications so generated cause the index data in ISS also to be deleted. However, the account itself is not deleted from ISS, just all the data. Use the issadmin.sh --deleteaccount command to remove the account from the ISS store afterward.

Sometimes, you might need to delete an ISS account during maintenance (such as to rebootstrap an account that has become corrupted or that cannot be synced by using the --sync command). If the account is also being deleted or restored on the Messaging Server message store during this period, the autoprovisioning feature might be triggered if the INBOX gets recreated. To avoid unwanted interference in such cases, create the ISS account explicitly again after you delete it by using the issadmin.sh --createaccount command before you attempt other maintenance on the account.

Labels:
indexsearchservice indexsearchservice Delete
Enter labels to add to this page:
Please wait 
Looking for a label? Just start typing.
  1. Apr 07, 2013

    For a small installation (i.e. a PoC/demo) with OUCS 7u2 bundle this script helps to automatically provision/destroy indexes. The limitation is that it can run on a host that is both the messaging server (has mboxutil/configutil) and an ISS server (has issadmin.sh). A smaller limitation (not substantial to small installations) is that the script does no queuing and will process all mailboxes/indexes without batching, but sequentially. Enjoy, though YMMV ;)

    Also, not tested yet with hosted domains (IMAP usernames would be user@domain, but I don't know yet what JISS usernames would be).

    jissinitnew.sh

    If there are no differences (nor errors), the script returns no text and a 0 errorcode, keeping your crontab diags clean. Many debugging options available (see the last case above).

    Hope this helps someone else,
    //Jim Klimov

    1. Mar 01, 2013

      Thanks for sharing this script Jim. I let the ISS developers know that it is here.

      Regards,

      Joe

Sign up or Log in to add a comment or watch this page.


The individuals who post here are part of the extended Oracle community and they might not be employed or in any way formally affiliated with Oracle. The opinions expressed here are their own, are not necessarily reviewed in advance by anyone but the individual authors, and neither Oracle nor any other party necessarily agrees with them.