Setting Up and Managing Messaging Server Security

Skip to end of metadata
Go to start of metadata
Starting with Oracle Communications Unified Communications Suite 7.0.6, and Delegated Administrator and Calendar Server 7.0.5, the documentation for Connector for Microsoft Outlook 8.0.2, Contacts Server 8.0, Instant Messaging Server 9.0.2, Delegated Administrator, and Calendar Server 7.0.5 is available on the Oracle Documentation site at

Setting Up and Managing Messaging Server Security

This information provides an overview about security for the Messaging Server product. It also provides links to security topics that provide more in-depth information for configuring and administering Messaging securely.


Overview of Messaging Server

For an overview of the product, see Introduction to Messaging Server Software. For information on general security principles, such as security methods, common security threats, and analyzing your security needs, see Designing for Security. For an overview of operating system security, see Oracle Solaris Security for System Administrators.

To see Messaging Server's high-level architecture, see Developing a Messaging Server Architecture. In addition, the following illustration describes the architecture of the Message Transfer Agent (MTA), Messaging Server's message router: MTA Architecture.

Secure Installation and Configuration

This section describes the installation considerations and recommendations for securing your Messaging Server deployment:

Installation Overview

Understanding Your Environment

To better understand your security needs, ask yourself the following questions:

  1. Which resources am I protecting?
    In a Messaging Server production environment, consider which of the following resources you want to protect and what level of security you must provide:
    • Messaging Server front end servers
    • Messaging Server back end servers
    • Dependent resources
  2. From whom am I protecting the resources?
    In general, resources must be protected from everyone on the Internet. But should the Messaging Server deployment be protected from employees on the intranet in your enterprise? Should your employees have access to all resources within the environment? Should the system administrators have access to all resources? Should the system administrators be able to access all data? You might consider giving access to highly confidential data or strategic resources to only a few well trusted system administrators. On the other hand, perhaps it would be best to allow no system administrators access to the data or resources.
  3. What will happen if the protections on strategic resources fail?
    In some cases, a fault in your security scheme is easily detected and considered nothing more than an inconvenience. In other cases, a fault might cause great damage to companies or individual clients that use Messaging Server. Understanding the security ramifications of each resource help you protect it properly.

Deployment Topologies

You can deploy Messaging Server on a single host or on multiple hosts, splitting up the components into multiple front-end Messaging Server hosts and multiple back-end hosts.

The general architectural recommendation is to use the well-known and generally accepted Internet-Firewall-DMZ-Firewall-Intranet architecture. For more information on addressing network infrastructure concerns, see Determining Your Communications Suite Network Infrastructure Needs.

The following guidelines provide specific recommendations for Messaging Server:

Securing Your Firewall/DMZ architecture

Secure your Messaging Server infrastructure by determining your firewall/DMZ architecture. The following topics cover securing your Messaging Server infrastructure:

Your firewall/DMZ architecture solution might depend on your anti-spam solution and client capabilities. How you handle firewall and DMZ architecture depends on requirements for a geographically dispersed deployment and whether your deployment is targeted at individual end users or enterprises.
Using a Firewall to Allow Connections

Because the Webmail server (mshttpd) supports both unencrypted and encrypted (SSL) communication with mail clients, you might use a firewall between your Messaging Store and your mail clients for added security.

Some guidelines to consider when using a firewall:

  • If using firewall only allow Convergence server to connect to mshttpd (8990, 8991).
  • If using firewall (preferably whitelist-based for Messaging Servers), verify internal service protocols are blocked (watcher 49994, job_controller 27442, ENS 7997, third-party authentication server, msadmind 7633, LMTP, Metermaid, and JMS).
Planning Secure High Availability and Load Balancing for your Deployment

The following topics describe how to set up a secure high availability and load balancing Messaging Server deployment:

Minimizing Operating System Security Risks

See: Operating System Security

In particular, pay attention to:

  • OS hardening, turning off unused OS services (especially in Linux)
  • OS minimization, using minimal OS packages

Installing Infrastructure Components

The following infrastructure components should be installed and secured prior to secure Messaging Server installation. You need to understand how all components in the infrastructure interconnect so that you can apply appropriate security measures to every interconnect.

  • Directory Server: Messaging Server connects to the Oracle Directory Server Enterprise Edition, an LDAP-based directory server for user and group information and for provisioning. See Installation Scenario - Directory Server and Oracle Directory Server Enterprise Edition Enhanced Security.
  • Communications Suite Directory Server Setup Script: the script to prepare the Directory Server for Communications Suite installation.
  • GlassFish Message Queue: is automatically installed with the other shared components. For the list of shared components that can be installed by the Communications Suite installer, run the commpkg info --listPackages command. See commpkg info Usage for additional information. To see the output for each supported operating system for more information, see Shared Components. Additionally, see: Configuring and Managing Message Queue Security Services.
  • Messaging Server Sun Cluster HA Agent 7.0: High Availability can be optionally installed.
  • DNS Server: You must ensure that Domain Name System (DNS) is running and configured properly. For details, see DNS configuration.
  • File System: Recommended file systems for the message store are listed in Message Store File Systems.

In addition to dependent products, it is equally important to secure the other components within Unified Communications Suite for secure Messaging Server deployment.

Review the following guidelines for components within the Unified Communications Suite that impact Messaging Server security:

Performing a Messaging Server Initial Configuration

See Messaging Server Initial Configuration.

When you perform a Messaging Server initial configuration by running the configure command, you are prompted for authentication credentials for the following:

  • User Name and Group Name for Server Processes
  • Directory Server manager (bind DN and password)
  • Password for server administration

Post Installation Configuration

The high-level post-installation steps to configuring Messaging Server for a secure deployment include:

  1. Installing Messaging Server Provisioning Tools
  2. Enabling SMTP Relay Blocking
  3. Enabling Startup After a Reboot
  4. Configuring JMQ Notification
  5. Configuring Certificate Based Authentication

For instructions, see [ Messaging Server Initial Configuration].

Once installation is complete, Oracle recommends encrypting and moving the initial state files and configure.ldif file, if generated.

Security Features

Messaging Server supports security features that enable you to keep messages from being intercepted, prevent intruders from impersonating your users or administrators, and permit only specific people access to specific parts of your messaging system. The Messaging Server security architecture is part of the security architecture of Oracle servers as a whole. It is built on industry standards and public protocols for maximum interoperability and consistency. For a general overview of Messaging Server security strategies, see: Planning Messaging Server Security.

This section describes additional features, considerations, and recommendations for securing your Messaging Server deployment:

Messaging Server Security Strategy for your Deployment

Creating a security strategy is one of the most important steps in planning your deployment. Your strategy should meet your organization's security needs and provide a secure messaging environment without being overbearing to your users. For more information, see Creating a Security Strategy.

How you set up the following topics impacts your security strategy guidelines:

Identifying Password Policy Requirements

The user password login process is described in User Password Login for Messaging Server.

Review the following additional password policy recommendations:

  1. Select a password policy system that meets your requirements and any additional requirements you might add at a later time.
  2. Require your users to create high quality passwords on your site identity system's password change web page. Do not require your users to change passwords too frequently as it cause users to write their passwords on paper.
  3. Directory Server has password policy capabilities, but if you enable password expiration, be sure the administrator and service accounts used by Messaging Server are exempt from expiration.
  4. Keep a strong administration password.
  5. Maintain administrative access policies for Messaging Server hosts.
  6. Create Delegated Administrator access policies for domains.
  7. If needed by Oracle Support, plan how to gather configuration files, excluding passwords.
    If, starting with Messaging Server 7 Update 5, you use Unified Configuration, this task is made much easier.

Verifying File Ownership for Configuration Files

Related topics include:

Securely Monitoring and Auditing Your Messaging Server Deployment

Monitoring your server is an important part of your security strategy. To identify attacks on your system, monitor message queue size, CPU utilization, disk availability, and network utilization. Unusual growth in the message queue size or reduced server response time may identify some of these attacks on MTA relays. Also, investigate unusual system load patterns and unusual connections. Review logs on a daily basis for any unusual activity.

Refer to Monitoring Messaging Server, which includes topics on:

  • Automatic Monitoring and Restart
  • Daily Monitoring Tasks
  • Utilities and Tools for Monitoring
  • Monitoring User Access to the Message Store
  • Monitoring System Performance
  • Monitoring Disk Space
  • Monitoring the MTA
  • Monitoring LDAP Directory Server
  • Monitoring the Message Store
  • Messaging Server Monitoring Framework Support
  • SNMP Support
  • How to Monitor MeterMaid

Additional guidelines for secure monitoring:

  • Ensure you have the right monitoring and auditing tools for your specific deployment and that you have contingency plans in place.
  • Enable MTA logging.

Tracking Security Patches

Be sure to install the most recent operating environment security patches and set up procedures to update the patches once every few months and in response to security alerts from the vendor. Be sure to pay close attention to NSS patches.

Identifying Legal-intercept Requirements

The following topics provide an overview for message archiving for legal and compliance purposes:

Determine which Messaging Server capture mechanism is best to meet those requirements in your jurisdiction prior to responding to a compliance request.

Securing Your Archiving Needs

Once you have satisfied legal requirements, use your third-party archiving system in your jurisdiction so that it can be configured to delete messages from the archive (or make them unreadable by discarding encryption keys). Refer to Identifying Legal-intercept Requirements for message archiving options.

Disabling Users in Response to Abuse/Appeal Process

The following topics describe how to disable accounts, block mail to an account, and enable and disable services at different levels:

Utilizing a Disk Consumption Growth Plan

Unusual disk consumption may identify some attacks on MTA relays.

See the following topics:

Preventing Unrelated Usage of Messaging Server Hosts and Virtual Machines

Oracle recommends that you do not use Messaging Server hosts or virtual machines for unrelated tasks. Single purpose hosts and virtual machines are better for securing your deployment. Be sure to turn off any unused Messaging Server services.

Determining Security Capabilities of Your Supported Mail Clients

For information on security and access control for mail clients and mail client infrastructure, refer to Security and Access Control where the following topics are covered:

  • Authentication Mechanisms
  • User Password Login
  • Encryption and Certificate-based Authentication
  • Mail Filtering and Access Control
  • Client Access to POP, IMAP, and HTTP Services

Consider these questions when designing your Messaging Server security strategy:

  • Do your mail clients support SMTP Authentication?
  • Do your mail clients support standard SSL (STARTTLS on port 143 for IMAP, STLS on port 110 for POP, STARTTLS on port 587 for SMTP Submission)?
    • If not, do they support legacy non-standard SSL (IMAPS on port 993, POPS on port 995, SSL SMTP Submission on non-standard port 465)?
  • Do you have a plan in place to handle accidental/inappropriate blacklisting of your site by reputation services?

MTA Security Guidelines

Following secure guidelines protect your MTAs from unauthorized users, large quantities of spam, reduced response time, and used up disk space and resources. Protecting MTAs outlines general guidelines to protect your MTAs. This section provides additional details in the following topics:

Securing Internal MTA Information

By default, Messaging Server enables four private commands called XADR, which returns information about how an address is routed internally by the MTA as well as general channel information, XCIR, which returns MTA circuit check information, XGEN, which returns status information about whether a compiled configuration and compiled character set are in use, and XSTA, which returns status information about the number of messages processed and currently in the MTA channel queues. Releasing such information may constitute a breach of security for some sites.

Sites might want to disable these commands for the tcp_local channel. For example, the commands to do so for Unified Configuration are as follows:

Options in the .options scope under channel are free form. As such, they are neither type-checked nor validated.

Identifying, Integrating, and Configuring Anti-spam and Anti-virus Solutions with Messaging Server

Refer to the following topics on anti-spam and anti-virus solutions:

In addition, consider these guidelines:

  • Make sure your domain's MX records point directly to Messaging Server's MTA and have the Messaging Server call out to anti-spam/anti-virus systems preferably through the spam plug-in or Milter mechanism.
  • Filter both inbound and outbound mail.
  • Consider restricting outbound port 25 to outbound MTAs only.

Creating a Narrow Scope of MTA Relay Blocking in INTERNAL_IP Mapping Table

To use the INTERNAL_IP mapping table for MTA Relay Blocking, refer to the following topics:

Starting with Messaging Server 7 Update 5, you can accomplish STMP relay blocking during initial configuration. The configure command prompts for a list of IP addresses of internal systems that are allowed to relay. If the list of such systems is empty, then the default settings from the configure command are used.

Using LMTP to Connect to Inbound MTAs and in Multi-tier Deployments

LMTP can provide an extra layer of security between the relays and the back end message stores. See the following topics:


See: Protecting Against Spammers who Compromise Messaging Server User Accounts

Forbidding Emailing Executable Code

See: How Can I Remove Image or Executable Files From Email?

Throttling Incoming MTA Connections with MeterMaid

MeterMaid is a server that can provide centralized metering and management of connections and transactions, including through monitoring IP addresses SMTP envelope addresses. Functionally, MeterMaid can be used to limit how often a particular IP address can connect to the MTA. Limiting connections by particular IP addresses is useful for preventing excessive connections used in denial-of-service attacks.

See the following topics:

Setting MTA Recipient Limits?

See: Limiting Message Recipients

Using Sieve Securely

Review your MAX_* options settings relevant to Sieve filter limits, especially MAX_NOTIFYS.

Notify, Forward, and Redirect can potentially increase the load of generating new messages. You need to consider if abusers could exploit such features by generating message loops or exponential growth of messages.

For Sieve external lists, enable setup carefully only allowing specific criteria. Some Sieve filter user education/Sieve filter creation interface guidelines to consider:

  • Discourage users from attempting to personally block spam by using Sieve.
  • Check that the interface generates efficient Sieves (for example: lists, wildcard matches, and so on)


Using the MTA to Fix Messages from Bad Clients

If users use email clients that especially vulnerable to buffer overruns, malicious embedding in malformed header lines, and so on, consider configuring the MTA with maximal MTA MIME processing and fixing up messages passing through the MTA with the inner MTA channel option.

Configuring Secure ETRN Command Support

Consider explicitly configuring the ETRN commands that the MTA honors. See the ETRN_ACCESS mapping table, the *etrn channel options, and the ALLOW_ETRNS_PER_SESSION TCP/IP channel option.

Review to the following topics:

ENS Security Guidelines

Securing ENS Server (7997) with Firewall and/or TCP Access Control Filters

To deploy the Event Notification Service (ENS) with Messaging Server, see: Communications Suite Event Notification Service Guide.

The current implementation of ENS does not provide security on events that can be subscribed to. Thus, a user could register for all events, and portions of all other users' mail. Because of this it is strongly recommended that the ENS subscriber be on the "safe" side of the firewall at the very least.

A firewall system generally controls what TCP/IP communications are allowed between internal networks and the external world. Firewalls prevent packets considered to be unsafe from passing through.

Message Store Security Guidelines

The most important data in the Messaging Server is the data in the message store. Physical access and root access to the message store must be protected. Protecting Your Message Store outlines general guidelines to protect your message store. In addition, you should review Message Store Management. This section provides additional details in the following topics:

Securing Your Backup System

The process for backing up the Messaging Server is described in Message Store Backup and Restore.

Some security guidelines to consider for message store backup:

  • Be sure that such a system does not leave unneeded data.
  • Backup systems that encrypt data improve your security if you manage the encryption keys properly.

Using configutil Parameters for Securing Messaging Server

The,, and service.http.ipsecurity configutil parameters are used to secure Messaging Server. See:

Being Aware of IMAP ACLs

See the following topics: Message Store readership command-line utility and Managing Shared Folders.

Disabling IMAP Shared Folders if Not Needed

Disable shared folders if not in use. See: Disabling IMAP Shared Folders

MMP Security Guidelines

The MMP serves as a proxy for the message store, therefore, it needs to protect access to end user data and guard against unauthorized access. Protecting MMPs outlines general guidelines.

You can use the server machine on which the multiplexor is installed as a firewall machine. By routing all client connections through this machine, you can restrict access to the internal message store machines by outside computers. The multiplexors support both unencrypted and encrypted communications with clients.

For information on MMP badguy throttling and MMP connection limits, see: MMP Reference

User Authentication Guidelines

User authentication allows end users to securely log in through their mail clients to retrieve their mail messages. Planning Messaging User Authentication outlines general guidelines. This section adds the following topics:

Acquiring SSL Server Certificates for the Server Domains

Refer to Certificate Based Authentication for Messaging Server on certificate based authentication.

Note the following recommendations:

  • Acquire SSL server certificates for server domains to which your users will connect from a third-party CA. If you also wish to secure inter-deployment connections (recommended for a geographically distributed deployment as well as helpful to meet legal requirements in some jurisdictions), also get certificates for your Directory Servers and back-end IMAP/POP storage servers.
  • Purchasing Certificate Authority (CA) service or software for your enterprise may be cost effective if you have many hosts in your deployment. Be sure to use at least 2048-bit RSA with SHA 256 signatures per current guidelines unless your jurisdiction does not permit that or some of your mail clients do not support that.
  • The certutil provided with Solaris and Communications Suite Installer too can be used with the -g 2048 and -Z sha256 switches). Once enabled, you can configure SSL.

See: Configuring Messaging Server and UltraSPARC T1 and T2 Hardware SSL Acceleration.

Some guidelines for SSL include:

  • Having a plan for SSL certificate or CA expiration.
  • Turning on SSL where required (external services, possible internal services)
  • Requiring SSL where possible (RestrictPlainPasswords, plaintextmincipher)

Requiring SMTP Authentication for Mail Submission

SMTP Authentication, or SMTP Auth (RFC 2554) is the preferred method of providing SMTP relay server security. SMTP Auth allows only authenticated users to send mail through the MTA.

Message Encryption Guidelines

Planning Message Encryption Strategies covers S/MIME and Encryption with SSL for encryption and privacy solutions. Review the following guidelines and recommendations:

Determining SSL Cipher Suites

See: Configuring Encryption and Certificate-Based Authentication

Review your SSL Cipher framework to determine the following:

  • Do your mail clients work if the mail server only supports modern AES cipher suites, modern AES and slow-but-secure 3DES? Or, are legacy RC4 cipher suites required?
  • Is SSL3 required, or can you restrict to TLS?

Using Solaris Crypto Framework in Place of NSS Default Software Token

If you use SSL for encryption, you can improve server performance by installing a hardware encryption accelerator.

See: Configuring Messaging Server and UltraSPARC T1 and T2 Hardware SSL Acceleration

Security Considerations for Developers

For secure programming best practices, refer to the MTA Developer's Reference.

messagingserver messagingserver Delete
security security Delete
Enter labels to add to this page:
Please wait 
Looking for a label? Just start typing.

Sign up or Log in to add a comment or watch this page.

The individuals who post here are part of the extended Oracle community and they might not be employed or in any way formally affiliated with Oracle. The opinions expressed here are their own, are not necessarily reviewed in advance by anyone but the individual authors, and neither Oracle nor any other party necessarily agrees with them.