||Upgrading to Lotus Notes and Domino 7|
||Tim Speed, Dick McCarrick, Tara Hall, Barry Heinz, Matthew Henry, Wendi Pohs|
||Packt Publishing Ltd.|
This chapter, along with the ones that follow, discusses the many new features found in the Domino 7 server. Here is a list of the features:
- Domino Domain Monitoring (DDM)
- DB2 support and administration
- Autonomic data collection
- Policy improvements, including new management features and mail policies
- AdminP enhancements
- Rename reversion
- SMTP improvements
- Client lock down
- Smart Upgrade enhancements
- Linux/Mozilla Web Administration client
- New ID and password recovery features
- CA process improvements
- Support for additional standards, including IPV6, CIDR, and IOCP
- Improvements and additions to rules, configuration, backup and restore, and server administration
One of the most significant new Domino 7 features—one that's gotten a lot of attention throughout the early beta programs—is Domino Domain Monitoring (DDM). This feature allows you to monitor the status of multiple servers in one or more domains, all from a single location.
DDM uses a set of preconfigured probes to gather status and process information about the servers being monitored. These probes collect data relating to applications, databases, directories, messaging, the operating system, replication, security, the server, and the Web. Special filters allow you to select the type and level of data recorded by the probes. After this data has been collected, it is consolidated, organized, and processed into easy-to-read summary reports. The data is then entered automatically into the Event Resolution Center (ERC). Each event that is processed and placed into the ERC database has a document link back to the specific monitor that generated the event. The ERC is updated with a status document each time a probe detects an error, or a particular threshold is exceeded. By viewing DDM events recorded in the ERC, you can identify (and in some cases even predict) systemic Domino events. The ERC is automatically created when you start the first server. The ERC database is based on the new template ddm.ntf; by default its file name is ddm.nsf.
By default, results generated by DDM probes are placed in the ERC on each server that runs the probes. You can create a DDM server collection hierarchy to aggregate data from several servers to a single server. By using this collection hierarchy, you can designate that a single server can collect all DDM-based event data (and thus use this single server to monitor multiple servers in your domain). Alternately, your can set up several servers to collect data across a domain.
The following figure shows how you can set up servers to collect DDM data in a worldwide domain. In this case, we use a multi-tiered collection model. The top server in our example is Server A in Los Angeles. This server collects data from itself and three other collection servers located in the USA. Collection Server B, located in a data center in Asia, collects data for itself and two other servers in Hong Kong and Tokyo. Collection Server C collects data from itself and two servers located in London and Paris.
Reported data is generated in each ERC based on its location in the collection architecture. To review data about the London server, an administrator can open the ERC on Server C, where data for the London and Paris servers is stored. It's also possible to view London data by opening the ERC on Server A, which contains all DDM data for all servers in the hierarchy.
This collection hierarchy is possible because each Domino 7 server writes its own probe results into a local ERC replica on each collection server. As a result, the ERC maintains data about its own probes as well as the probe data from every server that is monitored by this server. As you can see from the preceding figure, data is rolled up and pushed into the collection server that represents the next higher level in the tree. This process is managed by Lotus Notes replication. Selective replication formulas are automatically created when you create the DDM server hierarchy. Using this simple technique, you see the rolled up data where you want to see it—for instance, in the figure, data from Hong Kong exists on Server B (the Asia server) and Server A (the top-level server), but not on Server C (the Europe server).
Probes are the internal engines that make DDM work. There are nine types of probes available in Domino 7:
- Application code monitors an agent's schedule and resource (CPU and memory) usage.
- Database monitors database status and various activities.
- Directory monitors various directory functions.
- Messaging monitors the Domino-based messaging infrastructure.
- Operating System monitors operating system statistics and events.
- Replication monitors various replication activities. Replication probes can be configured to monitor all database replication, or specific databases.
- Security monitors the overall security of servers and databases in the domain.
- Server monitors the administration process for errors and reports them back to the ERC database.
- Web monitors web field settings and HTTP configuration fields. Each of these probes is described in more detail later in this section.
You can select which probes you want to run, and what data these probes collect, through Probe documents. These documents reside in the Events database (events4.nsf). Through Probe documents, you can specify when the probe runs. Many probes can be configured to run on a schedule, on an event, or real-time. The function of the probe will dictate what type of schedule can be executed. For example, if you select the Schedule option, you can choose to run the probe:
- Multiple times per day (including the time between each probe execution)
- Daily (including the days of the weeks and the time when the probe will execute)
- Weekly (including the day and time for the probe to run)
- Monthly (including the calendar day number that the probe will execute;
For example, if you want the probe to run on the fifteenth day of each month, enter 15.
In some cases (for example, the Security probe), you can enable the probe to run when a particular event occurs, such as when a Person, Server, and/or a Configuration document has changed. This can provide a very quick alert back to an administrator. You can also determine how missed probes are handled—you can ignore the missed probe, run the missed probe on startup, or run it at the next time range.
One very convenient feature is the ability to assign probe events based on server type. For many probes, you can select an option called Special Target Servers, which offers a set of server types, including:
- The Administration server of the Domino Directory
- LDAP server
- POP3 server
- IMAP server
- SMTP server
- Mail server
- Scheduled directory catalog aggregation servers
For instance, if you select the type as mail server, the probe will run on all mail servers in your domain.
You can create DDM filters to control the event type and event severity of events generated inside and outside of DDM. These filters determine what data is included in the DDM log file. You can specifically include all DDM events or include/exclude specific type of events. The DDM filter is created in the events4.nsf database. After the filter document has been created, you can determine the following for each filter document:
- Description: Provides explanatory information about this filter.
- Event Filter Type selection: Offers two choices: apply filter to DDM and non-DDM events, and apply filter only to non-DDM events.
- Event Types and Severities to Log: Determines which event types are recorded in the ERC. You can choose to log all event types, which would record all the types of events and all severity levels shown in the following figure. Or you can choose to log selected event types. If you choose this option, you can then select the types of events and their levels of severity.
- Servers on which the filter will be applied: Identifies the servers on which this filter will apply. You can choose all servers in the domain, or select the option Special Target Servers to specify the type of server for this filter (as described in the preceding section). You can also identify individual servers by name.
The Event Resolution Center (ERC) Database
The Event Resolution Center (ERC) ("See ERC, event resolution center" database (ddm.nsf)) contains the data generated by active DDM probes. When a probe runs, it records all the relevant data that it finds (if any) to a report that is placed in the ERC. This report contains the results of the specific probe, the probable cause that generated the result, suggested solutions for each event, and a link to the probe that was used to generate this event.
The ERC includes seven navigator views and a link to events4.nsf. Each view can help you find and/or diagnose a particular problem. The views are as follows:
- By Severity shows a list of probe results documents organized by severity level (Fatal, Failure, Warning (high), Warning (low), and Normal).
- By Type shows the probe results by the probe type (Application Code, Messaging, and so on).
- By Server displays results based first on the domain names, and then by a list of servers that the probes reported on.
- By Date shows all probe events in chronological order.
- By Assignment provides you with the ability to assign events to people and/or groups.
- My Events shows the events that are assigned to your username. This is a formula-based view (@Name([Abbreviate];@UserName).
- Open Monitoring Configuration provides a link to events4.nsf.
Types of Probes
As we mentioned earlier, there are nine types of probes. You select the type of probe you want to create in the Probe document:
This probe monitors an agent's schedule and resource usage. It also checks for agent-related conditions and events such as agents that are disabled by the design server task, agent security errors, and agent full-text index errors resulting from search operations.
The Database probe monitors database status and activities such as database compacting errors and design errors, as well as status on the ability to actually open this database. This probe has four different probe types that you can enable:
- Compact reports errors about the status of many server-based database compaction activities.
- Design reports any errors that occur during the design process.
- Error Monitoring is a very powerful database probe type. This monitors a number of database activities, including the internals of the Notes Storage Facility (NSF) and the Notes Indexer Function (NIF). The following screen shows the configuration document for this probe option:
Note the option to remove error codes from the list of errors that are to be recorded. By default, a number of error codes are automatically ignored, such as "Document has been deleted", "Entry not found in index", "File does not exist", and so on.
- Scheduled Database Checks 'pings' each database to check whether the selected database can be opened. Additional options for this include the ability to check for unused space in the targeted database and for any database inactivity.
The Directory probe is one of our favorites. This powerful probe monitors many different directory functions:
- Directory Availability monitors the availability of all directories being hosted and then reports any identified errors. The directories that are monitored can include the primary Domino Directory (names.nsf), Domino configuration directories, directory catalogs, directories enabled via directory assistance, and LDAP directories.
- Directory Catalog Aggregation Schedule monitors the scheduling of the Directory Catalog. The Directory Catalog is maintained by the DirCat Domino server tasks. This option monitors the schedule status of the task looking for scheduling elements, including missed directory aggregation and any aggregations that are taking too long to process.
- Directory Catalog Creation monitors the server-based DirCat Directory Catalog task process that creates the directory catalogs. This helps you take quick action to get this task back online.
- Directory Indexer Process State monitors the running status of the Directory Indexer.
- LDAP Process State monitors the status of the LDAP Domino server task.
- LDAP Search Response monitors the server's average search response time for LDAP searches. This can be configured via a set of thresholds. The following figure shows that there are four different events that can be generated based on each threshold, Warning Low, Warning High, Failure, and Fatal:
- LDAP TCP Port Health monitors the TCP port response for both the standard LDAP TCP port (389) and the LDAP-SSL port (636).
- LDAP View Update Algorithm monitors the algorithms that are used to update the LDAP server directory views. This algorithm can be tuned by using the LDAPBatchAdds NOTES.INI setting.
- NAMELookup Search Response monitors the average search response time of directory NAMELookups performed on the Domino server.
- Secondary LDAP Search Response monitors the average search response time of searches of secondary LDAP servers that are performed on the probed server.
The Messaging probe monitors the Domino-based messaging infrastructure. Features include the ability to monitor SMTP activity, Notes (NRPC) mail routing, and various mail-routing statistics. There are currently ten options that you can choose from:
- Mail DSN tests the SMTP mail flow using a Delivery Status Notification (DSN) technique. This can help you determine whether a particular site is online. This can be effective if the target domain supports DSN extensions.
- Mail Flow Statistic Check uses a metric known as the 'slack percentage' to monitor a series of messaging-based statistics, including
Mail.Waiting. This lets you monitor the quantity of mail moving through the Domino server. The slack percentage is a representative indication of how the router is processing mail.
- Mail Reflector provides a mechanism to test mail flow to a variety of mail systems. You specify a mail recipient as part of this configuration, but you also will need to configure the recipient to send the message back to the originating server. One method is to enable auto-forward messages from the mail recipient to an ISpy mail-in database on the server that is executing the probe.
- Message Retrieval Process State verifies that the IMAP and POP3 server tasks are executing properly on the server being probed.
- Message Retrieval TCP Port Health monitors the Domino Internet Message Access Protocol (IMAP) and Post Office Protocol (POP3) messaging protocols, and reports service status on each process. Additional options include the ability to monitor POP3SSL and IMAPSSL.
- NRPC Routing Status tests the status of the Notes NRPC mail router by placing a message in the mailbox. This message will then route to a mail-in database. This can report status based on a set of thresholds.
- Router Process State monitors the status of the Domino router server task.
- SMTP Process State monitors the status of the Domino SMTP server task.
- SMTP TCP Port Health verifies that the SMTP mail routing services are working correctly.
- Transfer Queue Check tests SMTP- and/or NRPC-based mail to individual destinations. This can be configured via a set of thresholds and can report when messages are not being delivered.
POP3 is defined in RFC-1725, and IMAP4 is defined in RFC-1730.
One of the many challenges that an administrator must deal with is the status of system resources. Operating System probes provide a mechanism to alert you about potential problems at the OS level. There are four types of Operating System probes that you can enable: CPU, Disk, Memory, and Network.
The Replication probe lets you monitor various replication activities within your Domino domain. Replication probes can be configured to monitor all database replication, or replication on specific databases. There are two options available with this probe: Errors and Replication Check.
You can create a Security probe to assess the overall security of servers and databases in your domain. A Security probe can identify a security-related server configuration problem and/or security issues with specific databases.
One significant variable with security probes is how the event severity is assigned. Severity is assigned during runtime and is calculated based on the number of various potential issues found. This severity is a percentage-level score that is generated for each Server Configuration document analyzed. (Consult the product documentation for details about how these percentages are calculated.) The basic percentages are shown in the following table:
|< = 50%
There are five Security probe types: Best Practices, Configuration, Database ACL, Database Review, and Review.
- Best Practices compares a set of baseline security configuration settings to the existing configuration in a Notes domain. You can modify the default values assigned to the security configuration. The following options are available for the Best Practices probe type:
- Compare Notes public key against those stored in directory
- Check password
- Allow anonymous Notes connections
- Required change interval
- Check passwords on Notes IDs
- Check for existence of ID file in the person document
- Internet authentication
- Check the security of SSL settings
- Check the security of Web settings
- Check the security of Domino Directory settings
- Check the security of Mail settings
- Check the security of DIIOP settings
- Check the security of the Remote Debug Manager
- Use more secure internet passwords
- Security settings in my Configuration document
- Internet password
- Verify all Server Document Security Tab sections. (This includes the Admins, Program, Web, Security Settings, Server Access, and Passthru Use sections.)
- Configuration compares a known 'good' Domino server document and a target server document, and then reports any differences or discrepancies. This type of security probe also has a Specifics section that you can configure. This allows you to compare a known good server configuration to the server being probed. Options include:
- Which server should be used as the guideline server?
- Which server settings should be compared to the guideline server's settings? You have several options here: Directory Profile Note, Security settings in the Server Configuration document, Server document (all sections or individual sections such as Admins, Program, Web, and so on).
- Database ACL monitors the Access Control List (ACL) of individuals and groups in various databases. You can set this up to monitor specific databases and list access levels in the probe configuration document. You can also check the access level status of any particular group. For instance, suppose you have a group known as 'External Contractors'. This group needs access to the 'Bass Fishing' Database with read-only access. You can configure a probe to monitor this critical database and report whether this group has been given an access level greater than Reader. This particular probe has the ability to monitor all basic ACL access levels, including Designer, Editor, Author, Depositor, Reader, and Default.
- Database Review is the 'inverse' of Database ACL. This monitors changes in access levels for all ACL members against a specific ACL level. You can create a probe and then select a database for it to monitor. You can then select a base level to monitor, for example review all ACL members whose privileges are equal or greater than Editor. You can also select one of the following parameters:
- Review the following database properties: enforcing consistent ACLs across replicas, enabling of extended ACLs, encryption settings, and the Administration server of the database.
- Review agents defined as restricted or unrestricted.
- Review creates a report on the security settings specified in the Specifics tab of the Security Probe document. You can select the same settings available for the Configuration type of security probe.
The Server probe monitors the administration process for errors, and reports the errors back to the ERC database. The following administration requests can be monitored:
- Change HTTP password in Domino Directory
- Change user password in Domino Directory
- Initiate rename in Domino Directory
- Initiate web user rename in Domino Directory
- Recertify Certificate Authority in Domino Directory
- Recertify Cross Certificate in Domino Directory
- Rename in person documents
- Rename person in calendar entries and profiles in mail file
- Rename person in Domino Directory
- Rename web user in Access Control List
- Set Password Information
The Web probe monitors Web-related statistics and events. You can select from two probe types, Web Best Practices and Web Configuration:
- Web Best Practices monitors HTTP configuration fields in the domain by comparing these fields to recommended base 'best practices' values. Fields that don't match these values are recorded in a report in the ERC database. This allows you to:
- Verify that server is using the most current web server configurations
- Verify basic web server configuration settings
- Verify web server performance settings
- Verify web server debug-log settings
- Verify web server security settings
- Web Configuration monitors web field settings in relation to a base configuration document. As with Web Best Practices, settings that do not match the base configuration are reported to the ERC. The settings you can monitor are similar to Web Best Practices settings.
After you open the configuration setting for a Web probe, you will notice that you cannot assign severity levels. The severity of an event generated by Web probes is determined using a percentage formula. This score is based on the number of potential problems that are found. After this calculation is complete, a 'severity percentage score' is calculated and logged to the ERC.
Event Notification Using an Agent
Domino has the ability to monitor and execute specific actions based on a large variety of events. These events can be the results of normal processing activities or can be error-type events. Event examples include:
- Directory (LDAP)
- News (NNTP)
- Web (HTTP/HTTPS)
With Notes/Domino 7, the Domino administrator now has several new options that can be triggered by an event. When an event takes place, an administrator now has the option to run an agent, run a program (with new parameters), send a console command to the server, or send a Java controller command.
Domino 7 includes processes called 'event generators'. These generators gather information by monitoring Domino tasks and statistics. Also, there is a built-in probing system that can be enabled and linked into the event generators. Specific conditions and/or thresholds can initialize event generators. Once an event generator has been started, it can pass data into the event monitor task. The event monitor task can be loaded manually at the Domino server console, or can be set to load each time the server is started via the NOTES.INI file, in the servertasks line. Once the event has been passed into the event monitor task, it will be processed against event handler configuration documents in the events4.nsf database. If there are no configuration documents defined, then no actions are executed when an event is passed into the event handler.
The Domino Administrator includes a set of default event generators; these are shown in the Event Generators view of the Monitoring Configuration database (events4.nsf). The following table lists the types of event generators that can be created:
|Database event generator
||Monitors database activity and free space. Monitors frequency and success of database replication. Reports on ACL changes, including those made by replication or an API program.|
|Domino server response event generator
||Checks connectivity and port status of designated servers in a network.|
|Mail routing event generator
||Sends a mail-trace message to a particular user's mail server and gathers statistics indicating the amount of time, in seconds, it takes to deliver the message.|
|Statistic event generator
||Monitors a specific Domino or platform statistic.|
|Task status event generator
||Monitors the status of Domino server and add-in tasks.|
|TCP server event generator
||Verifies the availability of Internet ports (TCP-based services) on servers and generates a statistic indicating the amount of time, in milliseconds, it takes to verify that the server is responding on the specified port.|
Each event document can have a severity assigned. The severity levels can be assigned to each event as needed. Examples of these are shown below:
||Imminent system crash|
||Severe failure that does not cause a system crash|
||Loss of function, requiring intervention|
To create a new event, just open the events4.nsf and select New Event Handler. The following screenshot shows how this looks:
There are three tabs: Basics, Event, and Action. The Basics tab provides two basic sections: Server(s) to monitor and Notification trigger. The first section is where the servers to monitor are set. The second section has three choices:
- Any event that matches a criteria.
- A built in or add in task event.
- A custom event generator.
Each choice will display additional choices on the Event tab. The following illustration shows these choices:
The administrator can create an event handler document to specify how to log the event to a specified destination. Also, administrators can prevent events from being logged or handled in any circumstance.
At this point, you might be asking, "What is this scripting stuff that I keep hearing about?" Let's start with an example and a goal. The goal is to send a notification and track ACL notifications in a log database. Use the following steps:
- Create a tracking database.
- Create a simple agent, view, and form in the tracking database.
- Create a database event generator document in events4.nsf.
- Create an event handler (run an agent).
- Enable the event handler and the event generator.
Create a Tracking Database
Overall, almost any database format will work. Once the database has been created, you will need to create a form, agent, and a view.
Create a Simple Agent, View, and Form in the Tracking Database
The form can have the following fields defined:
||Field Data Type
|Text of event|
|Target server for this event|
|Time and date stamp of event|
|Type of event|
|Severity of event|
|Text parameters in event|
|Event type error code|
|Server that originated the event|
|Textual representation to Severity|
|Textual representation to Type|
Here's a screenshot example of a form:
Create a default view as well. Any view will do; you can customize it as you like.
Next up is the agent. Below is a sample agent that you can base your agents on. Name the agent
EventAgent. This agent uses the
DocumentContext method. This method is a Read-only property. Basically, an in-memory document is created when an agent starts. This method is defined in the
NotesSession class, and uses
NotesDocument as its data type.
The basic syntax for the
Set notesDocument = notesSession.DocumentContext
Dim session As New NotesSession
Dim doc As NotesDocument
Set doc = session.DocumentContext
Print "Event Text = " & doc.Eventtext(0)
Print "Event Error Code = " & doc.errorcode(0)
Set db = session.CurrentDatabase
Set tardoc = db.CreateDocument
tardoc.form = "EventForm"
tardoc.Subject = "Event Information"
tardoc.EventText = doc.Eventtext(0)
tardoc.EventPrams = doc.EventPrams(0)
tardoc.ErrorCode = doc.Errorcode(0)
tardoc.EventSeverity = doc.EventSeverity(0)
tardoc.EventSeverityText = doc.EventSeverityText(0)
tardoc.EventTime = doc.EventTime(0)
tardoc.EventText = doc.Eventtext(0)
tardoc.EventTypeText = doc.EventTypeText(0)
tardoc.OriginatingServer = doc.OriginatingServer(0)
tardoc.EventErrorcode = doc.errorcode(0)
Call tardoc.Save( True, True )
Print "Agent Complete ! ! ! "
Don't forget to:
- Sign the agent
- Add the correct permission into the security document in order for the agent to run
- Make sure the events process is loaded on the server
Create a Database Event Generator Document in events4.nsf
Open the events4.nsf database on the server and select New Database Event Generator. Enable the event generated to monitor ACL changes for names.nsf. Here's an example screenshot:
Notice the TSPD-6FR4CG string in the screenshot. You will see a similar string, and it can be used as a reference point for the event handler.
Create an Event Handler (Run an Agent)
In events4.nsf, you can also create the event handler document. Using the flowchart as a reference, select New Event Handler. We'll now take a look at the changes/settings to be made.
Select and complete the following fields (refer to the first screenshot under the Event Notification Using an Agent section):
- Notify of the event only on the following servers
- A custom event generator
Select the event using the reference string from the event generator. Once this is selected, you will be able to see details on that document.
Select Run an Agent. Complete the following fields:
- Agent Database (use the name of your new database)
- Agent Name:
EventAgent (from the preceding example)
- Agent Parameters (any specific parameters and/or field names from your form)
Enable the Event Handler and the Event Generator
On the Action tab, enable the event handler and set a schedule if needed.
It is now time to test your event handler, agent, and event generator. Open the names.nsf database on the server where you have enabled each of the event handlers and the generator. Make a simple change to the ACL of the names.nsf database. As soon as this happens, you should see a display on the server console and a document created in your database. An example is shown in the following screenshot:
In this chapter, we have taken a detailed look at Domino Domain Monitoring (DDM), one of the major new administration features introduced in Domino 7. We reviewed the set of pre-configured probes that gather status and process information (applications, databases, directories, messaging, the operating system, and so on) about the servers being monitored. We discussed the Event Resolution Center (ERC), where processed events are placed. We also talked about event monitoring, Domino 7's ability to monitor and execute specific actions based on a large variety of events. These events can be the results of normal processing activities and/or error-type events. The next chapter discusses the Administration process.