Administering Indexing and Search Service

Skip to end of metadata
Go to start of metadata

Administering Indexing and Search Service

This information describes how to administer Indexing and Search Service, including the ISS Store Instance accounts.

Topics:

Administering the ISS Store Instance

The ISS Store contains all the index information used to support search services through the RESTful web interface.

Topics in this section:

ISS Store Instance Overview and Concepts

The information in an ISS Store Instance is organized into accounts. An account is uniquely identified by two strings, the user name and host (or domain) name. The account is the unit of security for content. Each account must have a password to control access to its data.

In the email environment, each account in the ISS store uses the same user name, host name, and password as the corresponding account in the Messaging Server message store from which its content is derived. Also, the ISS indexed data mirrors the folder directory structure of the email account.

Each account is assigned to one and only one group in the ISS store. Groups organize the underlying data files in the index. The data for all accounts assigned to a single group is indexed and stored together. This way of indexing and storing data reduces the overhead of managing multiple index directories for a large number of accounts.

You can restrict a group to contain only one account. A group that can contain only one account is known as a singleton group. After you create a singleton group with an account, you cannot assign another account to that group. If the account in a singleton group is deleted from the ISS store, that group is no longer a singleton, until explicitly assigned again.

When you create accounts, you can use the default allocation policy or you can explicitly specify a particular group or singleton group to contain the account. The default allocation policy places accounts in the lowest numbered, previously defined group that does not already contain the maximum allowed number of accounts. The first group by default is 101. Each group can contain up to iss.store.account.pergroup accounts (default: 1). If all previously defined groups are full, the number of the next group created is the successor to the lowest numbered group after the first group that has no successor group, thus filling in gaps in the sequence of group numbers. (Groups with numbers less than the first group are not affected by the default allocation policy.)

The initial ISS product installation does not define any accounts in the ISS store instance. You create accounts during the indexing process called bootstrapping. The bootstrapping process indexes the Messaging Server message store data and creates each account. For accounts that consist of very large numbers of emails, the initial indexing process can take up to several hours to complete. Currently, expect a bootstrapping rate of approximately 2000 email messages per minute.

It is difficult to estimate how much time the bootstrapping of any given account will take, so consider bootstrapping each account into a singleton group. After creating the account, check its size to see if it should be moved into another group with other accounts. See the issadmin.sh --moveaccount option for information about moving an account to another group. Bootstrapping accounts into singleton groups has the additional advantage of reducing the overhead on other accounts in the group that might slow the search process.

The indexing process sets the state of an ISS account based on its corresponding Messaging Server account. The following table describes ISS account states.

ISS Account States

Account Flag Account State Description
A Active The account is searchable. Account state must be active to be searchable.
B Bootstrapping The account is being bootstrapped.
I Inactive The account is in maintenance mode and is not active.
O Optimized Used to force optimization of the group containing the account.
X Unknown The account state is unknown. A new account is in this state after the --createaccount command.

You can manually change an account's state with the issadmin.sh tool, for example, from Active to Inactive. An account being bootstrapped will remain in the B (Bootstrapping) state. When bootstrapping has completed, the status changes to A (Active). After it is active, an account is available for searching, and real-time indexing events are enabled.

The issadmin.sh tool provides commands that enable you, for example, to inspect and modify accounts, list account, group, and folder information, modify the account state, create and delete accounts and folders, and verify the index against the Messaging Server store. For more information, see issadmin.sh.

Creating New ISS Accounts

After installing ISS, you must "bootstrap" each account into the ISS store by running the iss-base/bin/issadmin.sh --bootstrap script. For more information, see Creating New Indexing and Search Service Accounts and issadmin.sh. (Use the --help output to see a list of available options.)

Monitoring ISS Accounts

You monitor the status of the ISS store by using the iss-base/store/scripts/isssadmin.sh commands, such as --listaccounts, --listfolders, and --checkaccounts.

To List Information for All Accounts in the ISS Store

  • issadmin.sh --listaccounts

This command lists the information of all accounts in the ISS store, including any accounts currently being bootstrapped.

To Remove an Account From Active State

  • issadmin.sh --setstate I

This command removes an account from the Active state for maintenance (such as deleting or modifying it). The account state is set to I for Inactive and is disabled from searching and real-time updates.

To Verify Account Bootstrapping

  1. View the files in the log directory (defined by iss.log.dir in iss-base/etc/jiss.conf) to check for any problems while bootstrapping the account.
  2. Run either issadmin.sh --checkaccount (or issadmin.sh --checkfolder) to check the account for problems in specific emails that might have occurred during bootstrapping.
    These commands compare the data in the index against the current state of the Messaging Server store for that account and list any problems found in the index. For serious problems, you can delete the account (or just a single folder) and bootstrap it again to correct inconsistencies. Sometimes the bootstrapping does not process attachment data completely, causing incomplete search results, but the rest of the account data can be searched.

Periodic Statistics Logging

Log files contain error and informational messages about the operation of the various ISS processes. The IndexServer, SearchServer, and JMQConsumer processes also log statistics about their progress since the server was last restarted in separately configured log files. These logs provide insight into the services' overall status. They can be helpful in detecting use patterns and perfomance issues more easily than analyzing the individual informational messages logged.

To Generate a Statistic Snapshot

  • For the IndexServer and JMQConsumer processes, use the issadmin.sh command with the --liststats and --listbacklog commands, respectively.

IndexServer Statistics: Output of liststats Option

The following example output for Indexing and Search Service 1.0.5.19.0 shows the kinds of information that the --liststats option can provide on a running IndexService service. The output shows the general format of statistics that are periodically recorded in a log file. The first line indicates that the frequency is about every 10 minutes (600 seconds, the default). You can vary this frequency to between 15 and 60000 seconds by using the iss.indexsvc.statisticsinterval configuration parameter. The second line describes the various columns that appear in each line of the data that follows.

For details about specific lines of information, see the notes that follow the output.

Following the "autosync" lines are summary statistics for the Buffer Manager used within Lucene to reuse various sizes of data buffers to reduce memory pressure. The two sets of buffers are configured and behave independently of each other. If either the "too big buffer created" or "too big ByteB created" fields are not zero, additional information is printed indicating what size buffer was requested that could not be cached. Such buffers are created and released to the heap (not cached/reused), and, if they occur frequently, might be a source of large memory overhead (inducing garbage collection).

Fri Jul 06 09:06:59 GMT+00:00 2012 common.LogUtils.indexStats runLoop INFO:     607290  ms since prior sample, server has been running for 1.1 hours)
    count  description                 prior count     delta  percentage  rate/sec
  2314883  basic docs created             2259890     54993         2        91        (Note  1)
       14  document types created              14 
 table of doctypes:
   950418 xemailxmetax                     935619     14799         1        24        (Note  2)
   941960 xemailxcontentx                  928396     13564         1        22        (Note  3)
    93150 xemailxattachx                    91877      1273         1         2        (Note  4)
   167272 affpd                            157615      9657         6        16        (Note  5)
    92335 afd                               83389      8946        10        14        (Note  6)
    22761 amd                               20086      2675        13         4        (Note  7)
    31430 atdd                              28139      3291        11         5        (Note  8)
    10390 asd                                9731       659         6         1        (Note  9)
        0 admd 
        1 sdc                                   1                                      (Note 10)
        2 sda                                   2                                      (Note 10)
     3442 sdb                                3356        86         2         0        (Note 10)
        1 dd                                    1                                      (Note 11)
     1721 agd                                1678        43         2         0        (Note 12)

   186300  attachment type fields create   183754      2546         1         4        (Note 13)
       21  attachment types created            21
 table of attachment types:
    57924 pdf                               57402       522         0         0
     2474 plain                              1270      1204        94         1
       14 audio                                 2        12       600         0
      342 odf                                 206       136        66         0
     1872 ssign                              1038       834        80         1
     2726 image                              1390      1336        96         2
      232 html                                178        54        30         0
      314 compress                            108       206       190         0
     1800 vcf                                1270       530        41         0
   112376 jpeg                             110352      2024         1         3
       40 xml                                  30        10        33         0
      898 other                               712       186        26         0
       32 ppt                                  18        14        77         0
       68 doc                                  36        32        88         0
     4944 pgpsign                            3848      1096        28         1
       30 rtf                                  26         4        15         0
       48 video                                22        26       118         0
       28 xls                                  16        12        75         0
       10 applefile                             2         8       400         0
       24 iwork                                 6        18       300         0
       52 vsd                                   4        48      1200         0

   651138  single doc searches             627338     23800         3        39        (Note 14)
   158960  single doc search found none    150370      8590         5        14        (Note 15)
    22943  single map doc searches          22753       190         0         0        (Note 16)
   221513  account docs from cache         207014     14499         7        24        (Note 17)
   141091  folder flag docs from cache     127189     13902        10        23        (Note 18)
  1641865  total docs cached              1608713     33152         2        55        (Note 19) 
   140844   account                        138145      2699         1         4 
   864760   folder                         855814      8946         1        14 
        0   folder flags (obsolete)     
   636261   folder flags partial           614754     21507         3        35 
        0   account map  
   441613  total docs removed from cache   441560        53         0         0        (Note 20) 
      485   account                           459        26         5         0 
      485   folder                            459        26         5         0 
        0   folder flags (obsolete)     
   440643   folder flags partial           440642         1         0         0 
        0   account map                 
   238638  size of account doc cache       238556        82         0         0 
  1560726  size of folder doc cache       1560686        40         0         0 
   108530  size of flags partial cache      93945     14585        15        24 
    46266  # folder flag sets cached        40330      5936        14         9 
        0  size of account map doc cache
   125886  calls to closeAccess            118205      7681         6        12        (Note 21)
    35323  calls to delayCloseAccess        31590      3733        11         6        (Note 22)
     9716  closeAccess skipped               9161       555         6         0        (Note 23)
   158610  closeWriters done               148836      9774         6        16        (Note 24)
   119321  groupNumberCache size           119278        43         0         0        (Note 25)
   119246  groupAccountGroupCache size     119207        39         0         0        (Note 26)
        0  group Cache remote updates                                                  (Note 27)
        0  group Cache entries cleared                                                 (Note 28)
   379680  # IndexReaders opened           354571     25109         7        41        (Note 29) 
       11    # opens took over  1 sec          11 
        4    # opens took over 10 secs          4 
   148684  # dIndex IndexReaders opened    140264      8420         6        14        (Note 30) 
    56756    # opens took over  1 sec       51016      5740        11         9 
     6214    # opens took over 10 secs       5772       442         7         0
 table of event types:
   231875  Total events since index init   223109      8766         3        14        (Note 31)
    36294   NewMsg events                   32894      3400        10         5 
    20033   UpdateMsg events                17938      2095        11         3 
        0   Rename events               
        0   Delete events               
        0   Create events               
     8788   Copy events                      7486      1302        17         2 
    13206   ExpungeMsg events               11283      1923        17         3 
    31579   ChangeFlag events               31579 
     1969   Bootstrap events                 1923        46         2         0 
        0   SetAcl events               
        0   Login events                
        0   Logout events               
        0   unknown events   
 autosync: 
    19437  candidates added                 16739      2698        16         4        (Note 32) 
    10000  accounts checked                  9400       600         6         0 
     8690  accounts found OK                 8687         3         0         0 
     1310  accounts needed sync               713       597        83         0 
      993  account checking skipped           396       597       150         0 
        2  sync fail: other command             2 
        0  sync fail: missing folder    
        0  sync fail: authentication    
        0  sync fail: other reasons 
 simple buffer cache limits:     tiny  4096  small 16384  medium  6144  large  64  xlarge    16  
 simple buffer cache sizes now:  tiny   106  small  4661  medium  5511  large  33  xlarge    11  
 simple buffer sizes (in kb):    tiny   128b small     2  medium    16  large 128  xlarge  1024  
    4894m  reused buffers                   4610m 283790246         6    472924 
    2517m    tiny buffer reused             2375m 141769185         5    236252 
    2343m    small buffer reused            2203m 139914905         6    233162 
 25007652    medium buffer reused        23564990   1442662         6      2404 
  8265133    large buffer reused          7602221    662912         8      1104 
    17208    xlarge buffer reused           16626       582         3         0 
    32186  created buffers                  30709      1477         4         2 
      156    tiny buffer created              156 
     5009    small buffer created            5009 
    26977    medium buffer created          25500      1477         5         2 
       33    large buffer created              33 
       11    xlarge buffer created             11 
        0  too big buffer created       
    4894m  requested buffers                4610m 283791724         6    472927 
    2517m    tiny buffer requests           2375m 141769186         5    236252 
    2343m    small buffer requests          2203m 139914905         6    233162 
 25034437    medium buffer requests      23590298   1444139         6      2406 
  8265158    large buffer requests        7602246    662912         8      1104 
    17215    xlarge buffer requests         16633       582         3         0 
    4894m  cached buffers                   4610m 283792417         6    472928 
    2517m    tiny buffer cached             2375m 141769186         5    236252 
    2343m    small buffer cached            2203m 139915043         6    233162 
 25012971    medium buffer cached        23568277   1444694         6      2407 
  8265158    large buffer cached          7602246    662912         8      1104 
    17215    xlarge buffer cached           16633       582         3         0 
    20473  not cached buffers               18857      1616         8         2 
        0    tiny buffer not cached     
        0    small buffer not cached    
    20473    medium buffer not cached       18857      1616         8         2 
        0    large buffer not cached    
        0    xlarge buffer not cached   
        0  unknown size buffer not cache
 ByteBuffer cache limits:     tiny  4096  small 16384  medium  6144  large    64 xlarge    32  
 ByteBuffer cache sizes now:  tiny   128  small  4639  medium  5476  large    32 xlarge    20  
 ByteBuffer sizes (in kb):    tiny   128b small     2  medium    16  large   128 xlarge  1024  
 41008324  reused ByteB                  38397306   2611018         6      4351 
  1271295    tiny ByteB reused            1179606     91689         7       152 
 16413900    small ByteB reused          15250954   1162946         7      1938 
 23130463    medium ByteB reused         21785860   1344603         6      2240 
   176798    large ByteB reused            165554     11244         6        18 
    15868    xlarge ByteB reused            15332       536         3         0 
    33957  created ByteB                    32525      1432         4         2 
      128    tiny ByteB created               128 
     4987    small ByteB created             4987 
    28790    medium ByteB created           27358      1432         5         2 
       32    large ByteB created               32 
       20    xlarge ByteB created              20 
        0  too big ByteB created        
 41041241  requested ByteB               38428791   2612450         6      4353 
  1271295    tiny ByteB requests          1179606     91689         7       152 
 16418375    small ByteB requests        15255429   1162946         7      1938 
 23158869    medium ByteB requests       21812834   1346035         6      2243 
   176822    large ByteB requests          165578     11244         6        18 
    15880    xlarge ByteB requests          15344       536         3         0 
 41017579  cached ByteB                  38404592   2612987         6      4354 
  1271295    tiny ByteB cached            1179606     91689         7       152 
 16418027    small ByteB cached          15254979   1163048         7      1938 
 23135555    medium ByteB cached         21789085   1346470         6      2243 
   176822    large ByteB cached            165578     11244         6        18 
    15880    xlarge ByteB cached            15344       536         3         0 
    22355  not cached ByteB                 20723      1632         7         2 
        0    tiny ByteB not cached      
        0    small ByteB not cached     
    22355    medium ByteB not cached        20723      1632         7         2 
        0    large ByteB not cached     
        0    xlarge ByteB not cached    
        0  unknown size ByteB not cached
Tue Oct 15 13:44:21 PDT 2013 common.LogUtils.indexStats runLoop INFO: net change
 to AddDocsThreadsList size: 2 had been 0
Tue Oct 15 13:44:21 PDT 2013 common.LogUtils.indexStats runLoop INFO: net change
 to SingleFolderList size: 2 had been 0
Tue Oct 15 13:44:21 PDT 2013 common.LogUtils.indexStats runLoop INFO: net change
 to FolderThreadsList size: 4 had been 1
Tue Oct 15 13:44:21 PDT 2013 common.LogUtils.indexStats runLoop INFO: statistics
 sleeping for 600 seconds


Notes:

  1. Number of Lucene documents created since the server started. Following this line is the number of different type of documents, which are then formatted in a table.
  2. Number of meta data documents created. Each email must have one meta document. Therefore, this represents the number of emails processed since the server started.
  3. Number of content documents created. The body of an email is recorded here. This number is usually less than or equal to the number of meta documents.
  4. Number of attachment documents created. Each email attachment causes one such document to be created. Usually this count is much smaller than the previous two counts.
  5. Number of account folder flag partial documents created. Flag information for each email in each folder of an account is grouped in these documents. Updated whenever email flag values change.
  6. Number of account folder documents created. Each folder in each account requires one such document to record details such as name, current number of emails, and so on.
  7. Number of account map documents created. This is the basic account information record, containing identification data (host, user) plus account structure (number of folders, and so on).
  8. Number of account transition date documents created. Used to track the time of major account state transitions (such as creation of new emails in account, and so on).
  9. Number of account state documents created. Used to track each account state transition.
  10. Number of store summary documents created. Three separate document types are created (a,b,c) to represent the data because different parts are usually updated at different frequencies depending on what processing occurs. Data includes global counts, fixed configuration parameters, and so on.
  11. Dummy used for initializing empty index directories; always exactly one.
  12. Number of account group documents created. Updated when accounts are assigned, removed, or reassigned to an index group.
  13. Number of attachment type fields created. Content and attachment documents uses these fields to identify the kind of attachments in the email. The table shows how many attachments of each kind were processed. Usually it is exactly twice the number of xemailxattachx documents created.
  14. Number of times a particular document was searched for (such as affpd and amd doctypes).
  15. Number of times a particular document was searched for and not found. This is many times expected, such as when creating a new account, where the account record must be checked not already to exist, and so is expected not to be found before it is created.
  16. Number of times a single document was searched for (usually afd and amd). It uses a different interface, so a restricted subset of the counts in the previous two lines.
  17. Number of times an account document was found in the memory cache when looked up, thus avoiding the disk IO overhead.
  18. Number of times a folder document was found in the memory cache when looked up, thus avoiding the disk IO overhead.
  19. Number of documents added to the in-memory cache. Various kinds of documents are cached to avoid IO overhead because they are frequently accessed and infrequently changed. Account, folder, and flag documents are counted separately.
  20. Number of documents removed from in-memory cache. Cached documents are removed when data is stale. This might also be necessary to update the index on disk.
  21. Number of times access to an account was released. This ensures that the account is no longer modified, so any index writers can be closed.
  22. Number of times release of access to an account was delayed, causing a delay in the close of any open index writers. This is useful when several real-time events for an account occur in quick succession, since the rapid close and reopen of the index is unnecessary overhead and a potentially large throughput bottleneck.
  23. Number of times a delayedClose was skipped (for example, no close done). This occurs when real-time events occur so rapidly that a new event requiring the same index writer arrives before the previous delayed close has completed. Thus this measures how often the close and reopen was avoided between subsequent events, which can speed up processing at the cost of delaying the disk index update.
  24. Number of times any index writer was actually closed. Counts the close of the meta and content index writer for every account; indicates how frequently index directories on disk are being updated.
  25. Number of group number to account associations cached. Assignment of any account to a group occurs when it is created or moved, and does not change often so this information is cached in memory to speed up lookup (avoids disk IO). This line is primarily interesting only when creating, moving, importing, or deleting accounts to indicate progress.
  26. Number of group documents currently cached. Group information is cached in memory because it changes infrequently after accounts have been bootstrapped and assigned into groups. This line is primarily interesting only when creating, moving, importing, or deleting accounts to indicate progress.
  27. Number of times a group notification was sent to remote servers (such as index service to search service). This happens when major changes to the in memory group cache occurs to signal remote servers to update their in memory group caches. (This feature is disabled by default. See the iss.store.groupcache.enabled parameter for more information.)
  28. Number of group cache entries sent in group notifications to remote servers. This is a cumulative total of all group entries that are removed from the group cache. This feature is disabled by default. See the iss.store.groupcache.enabled parameter for more information.
  29. Number of times any IndexReader was opened.
  30. Number of times an IndexReader was opened for the dIndex directory. Included in the total for any IndexReader opened.
  31. Total number of events processed by the index service since the store instance was created. Below this total is the break down of the events by type processed since the service was last started. These numbers will restart at zero if the service is restarted, but the total will accumulate across server restart.
  32. Total number of accounts added to the autosync candidates list after the initial full candidate list was created. These are accounts where events were dropped or lost which caused the system to add the account to the front of the list to be corrected soon.

Search Service Statistics

The following example output for Indexing and Search Service 1.0.5.19.0 shows the kind of information that is logged for the SearchServer service. The example below shows the general format of statistics that are periodically recorded in a log file. These numbers are totals accumulated since the service was started. The first line indicates that the frequency is about every 30 minutes (1800 seconds, the default). You can vary this frequency to between 15 and 180000 seconds by using the iss.searchsvc.statisticsinterval configuration parameter. The second line describes the various columns that appear in each line of the data that follows.

For details about specific lines of information, see the notes that follow the output.

Following the "search time average" lines are summary statistics for the Buffer Manager used within Lucene to reuse various sizes of data buffers to reduce memory pressure. The two sets of buffers are configured and behave independently of each other. If either the "too big buffer created" or "too big ByteB created" fields are not zero, additional information is printed indicating what size buffer was requested that could not be cached. Such buffers are created and released to the heap (not cached/reused), and, if they occur frequently, might be a source of large memory overhead (inducing garbage collection).

Tue Oct 15 13:44:11 PDT 2013 common.LogUtils.searchStats runLoop INFO:    1800298  ms since prior sample, server has been running for 2.0 hours
    count  description                 prior count     delta  percentage  rate/sec
    54430  single doc searches              35200     19230        54        10        (Note  1)
        0  single doc search found none 
    54430  single map doc searches          35200     19230        54        10        (Note  2) 
        0  account docs from cache      
        0  folder flag docs from cache  
    54395  calls to closeAccess             35174     19221        54        10        (Note  3) 
        0  calls to delayCloseAccess                                                   (Note  4)    
        0  closeAccess skipped                                                         (Note  5)          
        0  closeWriters done                                                           (Note  6)            
    17730  groupNumberCache size            13219      4511        34         2        (Note  7)  
    17730  groupAccountGroupCache size      13219      4511        34         2        (Note  8)  
        0  group Cache remote updates                                                  (Note  9)   
        0  group Cache entries cleared                                                 (Note 10)  
   158711  # IndexReaders opened           102557     56154        54        31        (Note 11) 
        5    # opens took over  1 sec           2         3       150         0 
        0    # opens took over 10 secs  
    54451  # dIndex IndexReaders opened     35212     19239        54        10        (Note 12) 
     1689    # opens took over  1 sec         368      1321       358         0 
        0    # opens took over 10 secs  
 search statistics:
    54451  total queries                    35211     19240        54        10        (Note 13) 
        0   query failed                
        0   query parse failed          
        0   query parse token failed    
        0   query rewritten             

    35327  total number of folder terms     21267     14060        66         7        (Note 14) 
 table of search folder values:                                                        (Note 15) 
    35327 "INBOX"                           21267     14060        66         7 

 table of search field values:                                                         (Note 16) 
    19124 attachment-type                   13944      5180        37         2
    35327 folder                            21267     14060        66         7 
    24668 subject                           17570      7098        40         3 
    10659 body                               3697      6962       188         3 
 search result counts:                                                                 (Note 17)
    20098  zero results                     12303      7795        63         4 
    17897  1-10 results                     11777      6120        51         3 
    11127  11-100 results                    7535      3592        47         1 
     4760  101-1000 results                  3234      1526        47         0 
      515  more than 1000 results             326       189        57         0 
 search time counts:                                                                   (Note 18) 
    47516 less than 1 second                32447     15069        46         8 
     6830 1 less than 5 seconds              2725      4105       150         2 
       48 5 less than 10 seconds                2        46      2300         0 
        3 10 less than 20 seconds               1         2       200         0 
        0 20 less than 30 seconds       
        0 30 seconds or over            
 search time totals (ms):                                                              (Note 19)
 20830416 less than 1 second             13658003   7172413        52      3984 
 11131942 1 less than 5 seconds           3751215   7380727       196      4099 
   290501 5 less than 10 seconds            12692    277809      2188       154 
    39714 10 less than 20 seconds           11836     27878       235        15 
        0 20 less than 30 seconds       
        0 30 seconds or over            
 search time average (ms):                                                             (Note 20)
      438 less than 1 second                  420        18         4         0 
     1629 1 less than 5 seconds              1376       253        18         0 
     6052 5 less than 10 seconds             6346 
    13238 10 less than 20 seconds           11836      1402        11         0 
        0 20 less than 30 seconds       
        0 30 seconds or over     
 simple buffer cache limits:     tiny  4096  small 16384  medium  6144  large   64  xlarge    16  
 simple buffer cache sizes now:  tiny    64  small  1090  medium  3295  large    8  xlarge     4  
 simple buffer sizes (in kb):    tiny   128b small     2  medium    16  large  128  xlarge  1024  
    1199m  reused buffers               756545319 442563958        58    245828 
579180081    tiny buffer reused         369498075 209682006        56    116470 
593141149    small buffer reused        371354650 221786499        59    123194 
 24863511    medium buffer reused        14524620  10338891        71      5742 
  1924536    large buffer reused          1167974    756562        64       420 
        0    xlarge buffer reused       
     4586  created buffers                   2320      2266        97         1 
       64    tiny buffer created               64 
     1117    small buffer created            1117 
     3393    medium buffer created           1127      2266       201         1 
        8    large buffer created               8 
        4    xlarge buffer created              4 
        0  too big buffer created       
    1199m  requested buffers            756547115 442566224        58    245829 
579180081    tiny buffer requests       369498075 209682006        56    116470 
593142010    small buffer requests      371355511 221786499        59    123194 
 24866712    medium buffer requests      14525555  10341157        71      5744 
  1924536    large buffer requests        1167974    756562        64       420 
        0    xlarge buffer requests     
    1199m  cached buffers               756546029 442567185        58    245829 
579180081    tiny buffer cached         369498075 209682006        56    116470 
593141983    small buffer cached        371355268 221786715        59    123194 
 24866614    medium buffer cached        14524712  10341902        71      5744 
  1924536    large buffer cached          1167974    756562        64       420 
        0    xlarge buffer cached       
        0  not cached buffers           
        0    tiny buffer not cached     
        0    small buffer not cached    
        0    medium buffer not cached   
        0    large buffer not cached    
        0    xlarge buffer not cached   
        0  unknown size buffer not cache
 ByteBuffer cache limits:     tiny  4096  small 16384  medium  6144  large    64  xlarge    32  
 ByteBuffer cache sizes now:  tiny   126  small  1087  medium  3295  large     8  xlarge     8  
 ByteBuffer sizes (in kb):    tiny   128b small     2  medium    16  large   128  xlarge  1024  
 27492648  reused ByteB                  16168752  11323896        70      6290 
   198471    tiny ByteB reused             123644     74827        60        41 
  2503419    small ByteB reused           1591446    911973        57       506 
 24746464    medium ByteB reused         14426276  10320188        71      5732 
    44294    large ByteB reused             27386     16908        61         9 
        0    xlarge ByteB reused        
     4653  created ByteB                     2386      2267        95         1 
      128    tiny ByteB created               128 
     1115    small ByteB created             1115 
     3394    medium ByteB created            1127      2267       201         1 
        8    large ByteB created                8 
        8    xlarge ByteB created               8 
        0  too big ByteB created        
 27496261  requested ByteB               16170098  11326163        70      6291 
   198471    tiny ByteB requests           123644     74827        60        41 
  2504022    small ByteB requests         1592049    911973        57       506 
 24749474    medium ByteB requests       14427019  10322455        71      5733 
    44294    large ByteB requests           27386     16908        61         9 
        0    xlarge ByteB requests      
 27496132  cached ByteB                  16169016  11327116        70      6291 
   198469    tiny ByteB cached             123644     74825        60        41 
  2503994    small ByteB cached           1591810    912184        57       506 
 24749375    medium ByteB cached         14426176  10323199        71      5734 
    44294    large ByteB cached             27386     16908        61         9 
        0    xlarge ByteB cached        
        0  not cached ByteB             
        0    tiny ByteB not cached      
        0    small ByteB not cached     
        0    medium ByteB not cached    
        0    large ByteB not cached     
        0    xlarge ByteB not cached    
        0  unknown size ByteB not cached
Tue Oct 15 13:44:11 PDT 2013 common.LogUtils.searchStats runLoop INFO: statistics sleeping for 1800 seconds


Notes:

  1. Number of times a particular document was searched for (such as affpd and amd doctypes). Refer to corresponding IndexServer statistics for more detail.
  2. Number of times a single document was searched for (usually afd and amd). It uses a different interface, so a restricted subset of the counts in the previous two lines.
  3. Number of times access to an account was released. This ensures that the account is no longer modified, so any index writers can be closed. Search does not write to any accounts.
  4. Number of times release of access to an account was delayed, causing a delay in the close of any open index writers. Since search never writes to any accounts, this value should always be zero.
  5. Number of times a delayedClose was skipped (for example, no close done). Since search never writes to any accounts, this value should always be zero.
  6. Number of times any index writer was actually closed. Since search never writes to any accounts, this value should always be zero.
  7. Number of group number to account associations cached. Assignment of any account to a group occurs when it is created or moved, and does not change often so this information is cached in memory to speed up lookup (avoids disk IO). This line is primarily interesting only when creating, moving, importing, or deleting accounts to indicate progress.
  8. Number of group documents currently cached. Group information is cached in memory because it changes infrequently after accounts have been bootstrapped and assigned into groups. This line is primarily interesting only when creating, moving, importing, or deleting accounts to indicate progress.
  9. Number of times a group notification was sent to remote servers (such as index service to search service). Refer to corresponding IndexServer statistic for details.
  10. Number of group cache entries sent in group notifications to remote servers. Refer to corresponding IndexServer statistic for details.
  11. Number of times any IndexReader was opened.
  12. Number of times an IndexReader was opened for the dIndex directory. Included in the total for any IndexReader opened.
  13. Number of search queries received. The lines following this show the counts of failures by various types, and the number of times a search query was rewritten into a different form for easier processing. (The rewritten count is of interest primarily to developers.) Note the number of successful searches is found by subtracting the failure counts from the total received.
  14. Number of search queries which contained any folder term. Most searches specify a folder to search; those that do not search the entire account, across all folders.
  15. Table of counts of searches with folder terms, grouped by name of folder. This table will have a line for each unique folder name searched. In this example only the INBOX folder was ever searched.
  16. Table of counts of searches with each kind of term. The hostname and username terms are not counted since every search query must contain these terms.
  17. Table of counts of searches based on how many results were returned.
  18. Table of counts of searches based on how long each search took.
  19. Table of total times taken by searches based on how long each took. Times are in milliseconds. This table shows how long the search server spends processing these types of search query.
  20. Table of average times taken by searches based on how long each took. Times are in milliseconds. This table shows how long a typical search in each category took.
JMQConsumer Statistics: Output of listbacklog Option

The following example output for Indexing and Search Service 1.0.5.19.0 shows the kind of information that the --listbacklog command option can provide from a running JMQConsumer service. The example below shows the general format of statistics that are periodically recorded in a log file. These numbers are totals accumulated since the service was started.

The first two lines report how often the service runs (about every 30 minutes (1800 seconds)), and how long the server has been running.

The next set of lines list the counts of backlogged events for each account with more than zero events queued, grouped by account state. The number of lines in this set varies depending on how many accounts have events backlogged. Many of the lines indicating "Active" and "Inactive" accounts have been omitted in this example because they are so numerous. The summary of this information follows, starting with the totals for all events backlogged and for all users, with these totals then divided by account state.

After this summary, a table of counts is displayed, listing the number of events received, finished, failed, skipped, or dropped since the service started. Finally, the number of accounts whose state changed since the previous statistics were generated is logged, with the time required to generate these statistics.

For details about specific lines of information, see the notes that follow the output.

Tue Oct 15 13:14:13 PDT 2013 common.LogUtils.jmqStats run INFO: queue check sleeping for 1800 seconds
Tue Oct 15 13:44:13 PDT 2013 common.LogUtils.jmqStats runQueueCheck INFO: runQueueCheck: started, server has been running for 2.5 hours
Tue Oct 15 13:44:13 PDT 2013 common.LogUtils.jmqStats printQueues INFO: All users share host sc11b0217.us.oracle.com unless explicitly present in following messages:
Tue Oct 15 13:44:13 PDT 2013 common.LogUtils.jmqStats printQueues INFO: Active: 3025193@colorone.org:2 3099348@colorone.org:1 3076509@colorone.org:1 3114789@colorone.org:1 3014263@colorone.org:2 3063475@colorone.org:1
Tue Oct 15 13:44:13 PDT 2013 common.LogUtils.jmqStats printQueues INFO: Active:  3100179@colorone.org:2 3021434@colorone.org:1 3026333@colorone.org:1 3087690@colorone.org:1 3050238@colorone.org:2 3049006@colorone.org:1
Tue Oct 15 13:44:13 PDT 2013 common.LogUtils.jmqStats printQueues INFO: Active:  3025356@colorone.org:1 3012582@colorone.org:1 3034605@colorone.org:1 3062376@colorone.org:5 3071166@colorone.org:1 3075015@colorone.org:1
Tue Oct 15 13:44:13 PDT 2013 common.LogUtils.jmqStats printQueues INFO: Active:  3080474@colorone.org:2 3059811@colorone.org:1 3080156@colorone.org:1 3088724@colorone.org:3 3012326@colorone.org:1 3010701@colorone.org:1
Tue Oct 15 13:44:13 PDT 2013 common.LogUtils.jmqStats printQueues INFO: Active:  3029686@colorone.org:3 3036588@colorone.org:1 3118407@colorone.org:1 3090770@colorone.org:5 3112683@colorone.org:1 3112397@colorone.org:2
Tue Oct 15 13:44:13 PDT 2013 common.LogUtils.jmqStats printQueues INFO: Active:  3098379@colorone.org:1 3109379@colorone.org:1 3075342@colorone.org:1 3057362@colorone.org:1 3005288@colorone.org:1 3003412@colorone.org:1
...
Tue Oct 15 13:44:13 PDT 2013 common.LogUtils.jmqStats printQueues INFO: Inactive:  3091024@colorone.org:1 3013443@colorone.org:1 3066022@colorone.org:2 3118394@colorone.org:1 3039528@colorone.org:1 3107080@colorone.org:3
Tue Oct 15 13:44:13 PDT 2013 common.LogUtils.jmqStats printQueues INFO: Inactive:  3013876@colorone.org:3 3031089@colorone.org:2 3116432@colorone.org:3 3016384@colorone.org:2 3097988@colorone.org:1 3031007@colorone.org:1
Tue Oct 15 13:44:13 PDT 2013 common.LogUtils.jmqStats printQueues INFO: Inactive:  3056847@colorone.org:3 3065067@colorone.org:2 3116299@colorone.org:1 

Tue Oct 15 13:44:13 PDT 2013 common.LogUtils.jmqStats printQueues INFO: Bootstrap: 3099295@colorone.org:5 3107340@colorone.org:3 3030422@colorone.org:2 3060000@colorone.org:13 3116786@colorone.org:2 3045641@colorone.org:2 

Tue Oct 15 13:44:13 PDT 2013 common.LogUtils.jmqStats printQueues INFO:  Total: 28755 Users: 18556 Total active: 28577 Active Users: 18469 Total inactive: 151 Inactive Users: 81 Total bootstrap: 27 Bootstrap Users: 6 Total Unknown: 0 Unknown Users: 0
Tue Oct 15 13:44:13 PDT 2013 common.LogUtils.jmqStats printQueues INFO: 
    count  description                 prior count     delta  percentage  rate/sec
   203129 events Received                  132351     70778        53        39          (Note 1)
        0 events unsupported            
   109975 events Finished                   84252     25723        30        14          (Note 2)
        0 events failed/Not Finished    
      488 events with no account              407        81        19         0          (Note 3)
      285 events skipped from queues          227        58        25         0          (Note 4)
        0 events dropped: noops         
    63582 events dropped: changeflags       29130     34452       118        19 
        0 events dropped: others        
Tue Oct 15 13:44:13 PDT 2013 common.LogUtils.jmqStats call INFO: 0 account states changed, took 460ms 
Tue Oct 15 13:44:13 PDT 2013 common.LogUtils.jmqStats runQueueCheck INFO: runQueueCheck: done


Notes:

  1. Number of events read. This is the total of all events processed, dropped, skipped, or backlogged since the service started.
  2. Number of events successfully processed by the IndexService.
  3. Number of events read for which no account is defined in the IndexService. These events are ignored.
  4. Number of events skipped. These are events previously in the backlog that were not processed for a variety of reasons. (This number is not the same as the dropped events which were never backlogged.)

Starting and Stopping Services

The ISS-supplied commands start and stop ISS services.

To Stop ISS Services

  1. Log in as superuser (root).
  2. Run the svc_control.sh command.

To Start Services

  1. Log in as superuser (root).
  2. Run the svc_control.sh command, for example:

Using a Firewall Between ISS and Messaging Server

If you deploy a firewall between your Messaging Server and ISS Server, you need to open an additional port on the firewall for JMQ. That is, JMQ requires two ports. For more information, see Connecting Through a Firewall.

Using Solaris Management Facility with Indexing and Search Service

The Service Management Facility (SMF) enables you to manage the operating system and application services, such as Indexing and Search Service. SMF replaces the legacy init scripting start-up mechanism common to prior releases of Oracle Solaris and other UNIX operating systems. SMF improves the availability of a system by ensuring that essential system and application services run continuously even in the event of hardware or software failures.

The following instructions show how to configure the iss.user to be able to manage Indexing and Search Service through SMF. In this example, the iss.user is defined to be jiss.

  1. Log in as root.
  2. Set up Indexing and Search Service so that its services are managed by user jiss by adding the following line to the /etc/security/auth_attr file.
    solaris.smf.manage.iss:::Manage ISS Service States::
    
  3. Run the following command to add the user privilege.
  4. Run the svccfg command to configure the utilSvc service then refresh the configuration.
  5. Run the svccfg command to configure the indexSvc service then refresh the configuration.
  6. Run the svccfg command to configure the searchSvc service then refresh the configuration.
  7. Run the svccfg command to configure the jmqconsumer service then refresh the configuration.
  8. Verify the configuration.

Improving ISS Global Directory Index (dIndex) Performance

The features documented in this section were introduced in Indexing and Search Service 1.0.5.18.0.

Topics in this section:

Overview of ISS Global Directory Index

The dIndex is the global "directory index" that controls the Indexing and Search Service store. On average, the dIndex directory of a typical ISS store instance containing 150,000 accounts requires about 400 MB of disk space. Read, write, and especially segment merge operations on such large index directories are memory intensive and put a heavy workload on the CPU. Processing is also slowed due to how disk writes are serialized, even on updates to separate accounts. To reduce processing overhead as the dIndex grows over time, Indexing and Search Service 1.0.5.18.0 uses an alternate form of the dIndex, which distributes data for different accounts across multiple, smaller directories. This data distribution enables quicker access in parallel for independent account transactions. Using this dIndex alternative form, most dIndex data reads and updates require a smaller memory footprint, and fewer transactions need to be serialized.

The first implementation of the dIndex is known as "format 1." The form of the dIndex introduced in Indexing and Search Service 1.0.5.18.0 is known as "format 2." Both dIndex format 1 and 2 are fully functional, and either form can be used for any ISS instance independent of any other instance in a multiple ISS instance deployment. However, you can only use the format 2 dIndex with software installed from Indexing and Search Service 1.0.5.18.0 or greater.

Format 1 remains the default form of the dIndex. To use format 2, set the iss.store.partitions.count configuration parameter to a value greater than zero. You can only change this parameter when ISS services are stopped. Thus, this parameter is not refreshable. Once you have defined the value of this parameter for an instance, you can only change the format of the dIndex by using the issadmin.sh --converttoformat command.

You can set the partition count value in the jiss.conf file before creating a new ISS instance, and the ISS store can be populated by bootstrapping as before by using the value supplied. Once the dIndex has been created, usually the first time the Index Service has been started, the value is fixed in the dIndex, and can only be subsequently modified by using the --converttoformat command. You need to convert any ISS store instances created prior to ISS 1.0.5.18.0 to utilize this feature.

Choosing an Appropriate Partition Count

The value of the iss.store.partitions.count parameter defines the number of dIndex directories created for the store. If the count is zero, the format 1 dIndex only is created. For any value N more than zero, the format 2 dIndex is created. N additional directories are created in the same directory as the dIndex named dIndexXX, where XX takes the value 00 thru N-1. When the --converttoformat 2 command is used to convert an existing format 1 dIndex, account groups are distributed across the N dIndexXX directories more or less evenly to balance the load. Once assigned to a specific dIndexN directory, the group does not move, but any account may be moved to a different group (as with the format 1 implementation).

Information for all accounts in each group is distributed to the same dIndexNN directory. The amount of information for each account might vary greatly, due to the amount of content and structure of the account (for example, how many emails and how many folders the account contains). If this results in an unbalanced distribution of accounts, accounts can be moved between account groups after the --converttoformat 2 command completes using the --moveaccount command. Each dIndexNN directory is independently backed up in the same manner as the dIndex directory. (Thus if a backup is needed to recover from data corruption in the dIndex, only backup files for those partitions exhibiting problems need to be substituted individually, limiting potential data loss.)

The choice of value for the iss.store.partitions.count parameter depends on the number of accounts, account groups, and amount of data in the dIndex and account group directories. Theoretically, the larger the partition count, the more potential parallel activity between accounts is possible. The memory needed to process each individual account should typically be less because only a fraction of the dIndex must be referenced for each transaction. However, if more transactions run in parallel, the cumulative memory use at any given time could be greater than without partitioning. For a format 1 dIndex size up to about 0.5 GB, a partition count value of 5 to 10 typically reduces the average dIndexNN size enough to improve performance. For larger format 1 dIndex sizes up to 1 GB and more, higher partition count values might be needed.

Note
Limited tests have shown performance improvement plateaus rapidly after partition count 5 for a format 1 dIndex size of about 0.5 GB. Size is not the only factor, so it is not possible to predict accurately how your performance is affected.

The partition number for any group is displayed in the --accountinfo output (on the line containing the meta and content index sizes) whenever the partition count is not zero. The --converttoformat command takes up to about 10 minutes to run for most ISS store instances up to about 1 GB of data, depending on your specific configuration and hardware.

To Change the Partition Count

Changing the partition count for a format 2 dIndex involves converting back to format 1, updating the iss.store.partitions.count parameter, then converting back to format 2 using a different partition count. Also, you should check for any errors in the dIndex before converting. After changing the partition count, you then remove old backup files that are no longer valid for the new partition count.

  1. As superuser (root), stop ISS services.
  2. Back up the existing dIndex directories in case you have problems. For example:
  3. Run the checkIndex.sh command on each of the dIndex directories to be converted. For example:
  4. Resolve any problems detected by the checkIndex.sh command before continuing.
  5. Run the --checkstore command to check for inconsistencies in the dIndex. For example:
  6. Resolve any problem accounts before you convert the format. If necessary, you can delete accounts. Doing so avoids errors in the conversion. You can readd and bootstrap accounts that you delete after the format conversion as necessary.
  7. If necessary, convert the dIndex back to format 1 by editing the jiss.conf file and setting the iss.store.partitions.count parameter to 0.
  8. If necessary, run the --converttoformat 1 command.
  9. Convert the dIndex back to format 2 by editing the jiss.conf file and setting iss.store.partitions.count to a non-zero partition.
  10. Run the --converttoformat 2 command.
  11. To check for backup files that are no longer valid, change to the directory specified by the iss.store.dir configuration parameter.
  12. Remove unneeded backup files.

Administering the Indexing and Search Service Store

Topics:

Administering Account Placement in the ISS Store

Each account is uniquely assigned to a single ISS store instance, which contains all its index and data information. To manage the size of the store and locality of accounts, you need to be able to move an account from one store instance to another. This process consists of two steps:

  1. Exporting the information from its current store instance
  2. Importing the exported information into another store instance

The export step creates a snapshot of the account information in a file structure separate from the store instance. This structure can be used as an archive and as a backup. An exported account is therefore effectively a recovery from that backup.

Terminology and Conventions

The following terms describe the process of exporting and importing accounts:

  • Account -- Set of user information uniquely identified by "user" and "host" strings.
  • Account snapshot -- All information for an account captured at a given time by the --export command.
  • Snapshot repository -- Directory containing zero or more account snapshots. If you do not specify a snapshot repository by using the --eximpath option, the system uses the default snapshot repository to save snapshots.

Restrictions When Exporting and Importing Accounts

The following restrictions are applicable to exporting and importing ISS accounts:

  • When exporting and importing an account, ensure that the account state is set to Inactive.
    This setting ensures the validity of information while the export and import is occurring.
  • Only one account per command can be exported or imported.
    However, the --accountlist FILE option can be used to export or import multiple accounts in a single command if the values specified for all the other command line options used are the same for all accounts in the list (for example, options like --rehost and --eximpath).
  • During an export and import, you can move the account but not rename it.
    The user name must not change during the export and import process. To move the account to a different host, run the issadmin.sh command with the --rehost option. If the user name changes, the index records cannot be reused. The user name is treated as a new account, which you must then bootstrap from the content server (Messaging Server).
  • The administrator is responsible for managing the snapshot repository contents.
    ISS does not contain any automatic process for deleting old export snapshots. As you add new snapshots, disk space is used until you perform cleanup of old snapshots. The eximsummarylist summary file, located in the snapshot base directory, can also be cleaned up by simply editing it. The eximsummarylist file contains information for all the users in one snapshot directory, as well as a roll-up of all the export commands (as opposed to being overridden from a new issadmin.sh --export command). If you do edit eximsummarylist, be sure that you do not invalidate the basic structure, because locating the proper snapshot to --import depends on its integrity. This file also serves as a log record of each --export that occurred in the snapshot repository.

Exporting and Importing Accounts

In the following example, the mailbox is initially on mailhostA and the ISS account is on isshostA. You are moving the mailbox to mailhostB and the ISS account to isshostB.

  1. From isshostA, set the account to Inactive and export the account.
    (isshostA)#cd <iss-base-dir>/bin
    (isshostA)#./issadmin.sh --user userA --host mailhostA.example.com --setstate I
    (isshostA)#./issadmin.sh --export --eximpath /export/archive --user userA --host mailhostA.example.com
    
  2. Move the mailbox from mailhostA to mailhostB.
    (mailhostA)#cd <ms-base-dir>/bin
    (mailhostA)#./rehostuser -u userA -d mailhostB.example.com
    
  3. From isshostB, create the Indexing and Search Service account.
    (isshostB)#cd <iss-base-dir>/bin
    (isshostB)#./issadmin.sh --createaccount --userA --host mailhostB.example.com
    
  4. Rehost the account from isshostA to isshostB.
    (isshostA)#./issadmin.sh --import --eximpath /net/isshostA.example.com/export/archive --user userA --host mailhostB.example.com --rehost mailhostA.example.com
    
  5. From isshostB, check the account.
    (isshostB)#./issadmin.sh --checkaccount --user userA --host mailhostB.example.com
    
  6. If the checkaccount command did not return "in sync," perform a manual sync.
    (isshostB)#./issadmin.sh --checkaccount --sync --user userA --host mailhostB.example.com
    
  7. Set the account to Active.
    (isshostB)#./issadmin.sh --setstate A --user userA --host mailhostB.example.com
    
  8. (Optional) Delete the account on mailhostA.
    (isshostA)#./issadmin.sh --deleteaccount --user userA --host mailhostA.example.com
    

Each account snapshot consists of a directory in the snapshot repository. This directory contains directories for the meta index, content index, dIndex records (a modified subset for this account), data store files, and a plain-text summary file describing the snapshot. The snapshot also appears as part of the overall exisummarylist file. This summary identifies a specific account within the snapshot repository.

A snapshot repository can contain any number of such account snapshot directories. It also contains a plain-text summary file (eximsummarylist) that is used to navigate multiple snapshot directories in the repository.

The --eximpath option can specify any available directory. If you do not specify a directory, the --export and --import options use iss.store.dir/snapshots as the default. You can override the default setting by using the --setdefaulteximpath argument to the issadmin.sh command.

Automating User Rehosting

The issrehostuser.sh script is available starting in Indexing and Search Service 1.0.5.18.0

Starting with Indexing and Search Service 1.0.5.18.0, the process of rehosting a user from one Messaging Server instance to another is more simple and automated. The issrehostuser.sh script and the rehostuser command work together to move the account from its Indexing and Search Service instance to a new one while at the same time the account is moved to its new Messaging Server instance. See the Messaging Server 7 Update 5 Patch 30 documentation for more information on using the rehostuser command.

Labels:
indexsearchservice indexsearchservice Delete
guide guide Delete
administering administering Delete
Enter labels to add to this page:
Please wait 
Looking for a label? Just start typing.

Sign up or Log in to add a comment or watch this page.


The individuals who post here are part of the extended Oracle community and they might not be employed or in any way formally affiliated with Oracle. The opinions expressed here are their own, are not necessarily reviewed in advance by anyone but the individual authors, and neither Oracle nor any other party necessarily agrees with them.