Indexing and Search Service Performance Tuning

Skip to end of metadata
Go to start of metadata

Indexing and Search Service Performance Tuning

This information describes how to tune your Indexing and Search Service deployment.

Topics:

Bootstrapping Users

During the bootstrap process, frequent access to the dIndex might cause some performance issues, resulting in slow creation of accounts. To reduce some of the overhead on dIndex, increase the value of the following jiss.conf parameter:

# optimizeinterval: number of dIndex writes before optimize is done
iss.store.account.optimizeinterval=50000

Higher values cause less frequent optimization of the dIndex, reducing overhead while bootstrapping.

You can also increase the value of the following parameter to 5:

# optimizelevel: default level of optimize (number of segments left)
iss.store.account.optimizelevel=1

Check the iss-indexsvc.log to see how frequently the dIndex is being optimized, and how long it is taking. You should see a log entry similar to the following one:

INFO: SharedWriter.close: optimizing index:/dIndex Time: 123ms

RESTful Web Services Searches

Depending on the incoming search rate, you can modify the following parameter in the jiss.conf file to accommodate a higher search rate:

# Size of proxy connection pool in RESTful web services
iss.rest.proxypool.size=512

The default is currently set to 512. Because this value is larger than 100, make sure that the following parameter is added to the IMQ broker config.properties file:

imq.autocreate.destination.maxNumProducers=-1

Otherwise, RESTful Web Services fail with the following error message:

Could not create connection and producer com.sun.messaging.jms.JMSException: [ADD_PRODUCER_REPLY(19)] [C4036]: A broker error occurred. :[409] [B4183]: Producer can not be added to destination SearchTopic [Topic], limit of 100 producers would be exceeded

Also, when this value goes above 500, the following parameter also needs to increase to accommodate more than 2 * 500 threads in IMQ broker:

imq.jms.max_threads=2000

Setting Up Large Deployments

For large deployments, avoid bootstrapping users by using the default allocation policy or by randomly assigning accounts to groups. Instead, use the --accountlist file option, which enables you to provide a file containing the desired mapping of accounts to groups. Using this approach minimizes the contention between accounts in the dIndex. By using this method, you might reduce the bootstrap time of 20,000 accounts by over 15 percent.

Beginning with Indexing and Search Service 1 Update 2, the indexSvcFork.sh --file command is deprecated. Use issadmin.sh --bootstrap command with the --accountlist file option instead.

Indexing and Search Service 1 Update 2 (and greater) example:

Indexing and Search Service 1 Update 1 example:

Tuning RESTful Web Services

This section contains the following topics:

Note
This information was gathered during tests on CMT platforms.

Application Server domain.xml File

You can apply all configuration changes directly to the main Application Server configuration file, domain.xml. If necessary, use the Application Server administrative interface. This configuration file is located in the $INSTALL/domains/domain1/config directory.

Application Server modifications are based on the recommendations from the Sun GlassFish Enterprise Server 2.1 Performance Tuning Guide and Scott Oak's blog.

http-listener

You can use this the http-listener entry to accept search requests from the imapd process.

Set the acceptor thread value 64 to match the number of virtual processors available. Add the IP address to indicate which network interface is dedicated to ISS web service requests.

request-processing

This entry determines how HTTP requests are processed by the Application Server.

Modify the initial-thread-count, thread-count, and thread-increment settings to make more efficient use of the 64 virtual processors available on a Sun SPARC Enterprise T5220.

keep-alive Connections

Use the following keep-alive setting. Keep-alive connections enable connection re-use, saving resources that would be required to open and close a socket connection for every HTTP request that is received.

The thread count is modified to make more efficient use of the 64 virtual processors available on a Sun SPARC Enterprise T5220.

JVM Options

You can add the following Java Virtual Machine (JVM) options, based on advice given in the Scott Oaks blog entry for the Sun SPARC Enterprise T5220 platform:

Access Logging

You can disable access logging, which can be disk-intensive, by setting this entry:

JMQ Broker config.properties File

You can add non-default configuration settings to the bottom of the /var/imq/instances/imqbroker/props/config.properties file.

Java Message Service Threads

To enable more requests to be processed in any given time interval, you can increase the jms.max_threads used by the ISS jmqbroker. The default number of threads is 1000, but this is too small. The ISS jmqbroker handles both indexing and search requests, unlike the jmqbroker used by the Messaging Server's JMQ notification plugin, which handles only indexing requests.

Maximum Number of Producers

The default number of message producers that can be created by the jmqbroker is 100. This value is too low to handle the number of messages that are typically generated by an active Messaging Server deployment that supports more than 10,000 active users. You can disable the setting to enable an unlimited number of producers to be created to handle the load.

JVM Options

jmqbroker
ARGS=-javahome  /usr/jdk/latest -vmargs -d64 -vmargs -Xms2048m -vmargs -Xmx2048m -vmargs -Xmn1024m -vmargs -XX:PermSize=50m
-vmargs -XX:MaxPermSize=50m -vmargs -XX:+UseConcMarkSweepGC -vmargs -XX:+UseParNewGC -vmargs -XX:+CMSParallelRemarkEnabled
-vmargs -XX:+CMSScavengeBeforeRemark -vmargs -XX:ParallelGCThreads=8 -vmargs -XX:MaxTenuringThreshold=15 -vmargs
-XX:+DisableExplicitGC -vmargs  -Xloggc:/var/imq/instances/imqbroker/log/jmq-log.gclog -vmargs -XX:+PrintGCDetails
-vmargs -XX:+PrintGCDateStamps -vmargs  -XX:+PrintTenuringDistribution

The JVM option allows use of the 64-bit JVM and reserves 4 GBytes of memory for the jmqbroker. Scalability tests show that the Messaging Server and the ISS jmqbroker use approximately 1.5 GBytes of the JVM heap.

jmqconsumer

Here are the JVM options used for the jmqconsumer:

java.args.jmqconsumer = -Xms6144m -Xmx6144m -Xmn3072m -XX:PermSize=100m -XX:MaxPermSize=100m -XX:+CMSClassUnloadingEnabled
-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSParallelRemarkEnabled -XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSInitiatingOccupancyFraction=50 -XX:+CMSScavengeBeforeRemark -XX:ParallelGCThreads=8 -XX:SurvivorRatio=4
-XX:MaxTenuringThreshold=15 -Xloggc:/var/opt/sun/comms/iss/logs/iss-jmqconsumer-$$.gclog -XX:+PrintGCDetails
-XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution

Tuning the Configuration

Set the size of heap size of Index Service as large as is practical. Garbage collection activity should be monitored to determine whether the size is sufficient. To adjust the heap size, set java.args.indexsvc as follows. Replace YY with a value between one-fourth and one-half the RAM on the system. Replace ZZ with one-half the size of YY.

java.args.indexsvc = -XmsYYg -XmxYYg -XmnZZg -XX:PermSize=100m -XX:MaxPermSize=100m -XX:+CMSClassUnloadingEnabled
-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSParallelRemarkEnabled -XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSInitiatingOccupancyFraction=50 -XX:+CMSScavengeBeforeRemark -XX:ParallelGCThreads=8 -XX:SurvivorRatio=6
-XX:MaxTenuringThreshold=15 -Xloggc:/var/opt/sun/comms/iss/logs/iss-indexsvc-gclog -XX:+PrintGCDetails
-XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution

For example, consider a Sun Fire x4540 system with 56 Gbytes of memory, allowing for memory allocated as follows:

  • ramdisk used for zpool log device, 8 Gbytes
  • searchsvc, 4 Gbytes
  • GlassFish Server, 2 Gbytes
  • jmqbroker, 2 Gbytes
  • jmqconsumer, 6 Gbytes

The following java.args.indexsvc settings would be appropriate:

-Xms16g
-Xmx16g
-Xmn8g

Tips:

  • When debugging garbage collection issues, it might also be useful to add -XX:+PrintHeapAtGC.
  • When debugging services, increase the log level on the services from WARNING to INFO.
  • To ensure enough log data is retained, increase the size of the iss.service.logfile.limit parameter, or increase the iss.service.logfile.count parameter, or do both.
Labels:
indexsearchservice indexsearchservice Delete
tuning tuning Delete
Enter labels to add to this page:
Please wait 
Looking for a label? Just start typing.

Sign up or Log in to add a comment or watch this page.


The individuals who post here are part of the extended Oracle community and they might not be employed or in any way formally affiliated with Oracle. The opinions expressed here are their own, are not necessarily reviewed in advance by anyone but the individual authors, and neither Oracle nor any other party necessarily agrees with them.