InterMapper Database Overview
InterMapper Database is a part of InterMapper DataCenter. It acts as a central repository for data collected from one or more InterMapper servers. This data includes historical readings from customer charts, events that are written to a log file (up/down/warn/alarm/critical events, as well as notifications and acknowledgements), and a snapshot of the current state of each server's managed elements (maps, devices, interfaces, etc).
InterMapper Database gathers this data efficiently, organizes it in a clear and logical schema, and provides a means to respond to queries using industry standard SQL statements. InterMapper Database can be operated on its own or could be part of an organization's Configuration Management Database initiative (CMDB).
More detailed information on the database schema can be found at:
http://download.dartware.com/schema/imdatabaseschema.sql
http://download.dartware.com/schema/imdatabaseschema.png
http://download.dartware.com/Dartware Database Schema.pdf
Basic Operation
Customers install InterMapper Database on a machine on their network: it need not be the same computer as an InterMapper Server. They then configure each InterMapper Server on their network to register itself with the InterMapper Database server.
Once an InterMapper Server registers, InterMapper Database begins to query each InterMapper Server to retrieve three general types of data:
- snapshot data that describes the current state of the server, and what maps, devices, interfaces, etc. it is monitoring
- events data (up/down/warn/alarm/critical events, as well as notifications and acknowledgements)
- continuous data, which can be measurements of traffic, error rates, temperatures, CPU utilization, etc..
When InterMapper Database retrieves snapshot data, it updates the SQL database to match that snapshot. Devices that have been removed persist in InterMapper Database to preserve relational integrity, but are specially marked as not being part of the current snapshot. Continuous data is simply accumulated in the SQL database and can be compressed, reduced according to retention policy, or moved offline.
Installing InterMapper Database
InterMapper Database is installed as part of the InterMapper DataCenter download. To install InterMapper Database, follow the instructions for installing InterMapper DataCenter, then follow the instructions for configuring InterMapper Database.
Registering an InterMapper Server
Once InterMapper Database is installed, you will need to configure your InterMapper Server(s) to register with the database, so that it knows to retrieve data from those servers. To do so, follow the instructions for registering an InterMapper Server with InterMapper Database.
Getting Data into the Database
Every minute, InterMapper Database requests all the current continuous data from its InterMapper server(s). In addition, if there is historical data (from previous days, months, or years of operation by earlier InterMapper servers) InterMapper Database will progressively request that older data. It works from the current time backwards until it has all been retrieved.
The Collection Profile in the InterMapper Server Settings/Database Server pane controls how aggressively to collect the old data. “Now” retrieves the data as fast as possible, but may use 100% of the computing resources on the InterMapper Database machine. “Gradually” requests historical data between every other one-minute query. This will impose a lower processor load, but gets the data more slowly. The “Nightly” and “Weekend” profiles use the Now policy, but only during specified hours of the day/weekend.
InterMapper Database records several kinds of datasets:
- All probes record the response time (in msec) and the long-term packet loss (%) for devices, and the input and output byte rates for all visible interfaces.
- any datasets that were created by a chart.
- any datasets that a probe writer has specified as an <autorecord> variable in a custom probe.
- For built-in probes, certain data are marked as autorecord and are automatically recorded.
Data Reduction Policy
InterMapper Database collects raw data at the full resolution from the InterMapper Server. If the polling rate is fast — 30 seconds or even faster — InterMapper Database will collect and preserve all the information.
InterMapper Database can summarize all the raw data samples, and average over a longer (e.g., five, ten or 15 minute) time periods. This process, called data reduction dramatically reduces the amount of data stored.
For example, it might make sense to keep raw 30-second data samples for a week, but then reduce those data values to five-minute averages for six months, and retain only hourly averages after a year. This combination of poll rates and intervals is called a data retention policy. It specifies the time periods and durations that the particular data values are retained.
The InterMapper Database web GUI allows you to create additional data retention policies. You can use this to select how long to retain any particular set of data.
By default, InterMapper Server gives all devices and interfaces the server-wide default data retention policy, which is called "24 hour". All data is kept for 24 hours, then discarded. Note: There is one exception to this; when you upgrade to InterMapper 5.0 or later from InterMapper 4.6.x or earlier, InterMapper gives existing devices or interfaces that contain charts the "Forever" policy, which means that all data is saved forever.
To set the server-wide default data retention policy, open the server settings and click on the Database Server pane. To set the map-wide default data retention policy, open the map settings and click on the Retention Policy pane. To select a different retention policy for a device, use the Set Data Retention item in the Monitor menu or right-click on the device or interface and choose Set Info...->Set Data Retention.
Getting Data out of the Database
Until InterMapper Reports is available to complement InterMapper Database, you will need external tools to retrieve data from InterMapper Database via SQL. To get you started, we've written several short example reports in Crystal Reports and OpenRPT, as well as several example perl scripts. The perl scripts require DBI and DBD::pg.
These scripts have been packaged and zipped and placed on our downloads server, available at http://download.dartware.com/sql/sql_examples.tar.gz. Please feel free to share your own, as well.
Internal Operation
InterMapper Database can be divided into four major components:
- The Retriever maintains a list of InterMapper Servers, retrieves information on the designated schedule, and passes that data to the DBUpdater.
- The DBUpdater writes data into the tables of the Database. In addition, it performs data reduction according to the specified retention policies.
- The Database is an SQL database manager that holds the accumulated data and provides SQL access to it.
- The Configuration module is a web-based interface used to configure InterMapper Database.
Retriever Module
The Retriever is implemented as a Python application running under the InterMapper DataCenter (IMDC). It uses an encrypted HTTP-based protocol to retrieve data from an InterMapper server. We do not plan to open this protocol as an API, as it is primarily designed as a means of conveying the information from InterMapper into the SQL database. Customers should use SQL queries against the Database to retrieve data for reports.
DBUpdater Module
The DBUpdater is another Python application running under the IMDC. It performs these tasks:
Insert a brand-new snapshot of all the maps, devices, interfaces, notifications, etc. on a given server into the Database.
Update already-inserted devices in response to a snapshot.
Add new devices and remove deleted devices in response to add/remove events.
Insert blocks of continuous chart or event data, and link them to the objects that produced them.
Compress, reduce or discard data as dictated by user-specified data retention policies
A snapshot is received as a list of all entries in a certain category (maps, devices, interfaces, etc). Each element across all types is guaranteed to have a unique identifying code - an InterMapper ID (IMID).
An object's IMID contains information on its parents. If an interfaces report is requested, the IMID for each indicates the device, map and server that interface belongs to. This is used to set the server_id, map_id and device_id fields, as well as the interface's own interface_id field entry in the interface table.
Chart data is a series of data points, and can be inserted into the database sequentially. Each data point is associated with a data set.
Events may be split into a start and stop event. Start events have a half-empty entry created for them. When the event is over, the stop event completes the previous entry rather than getting a new one.
An InterMapper Server only keeps times up to the second. But there may be several events happening in one second, and they need to be distinguished. As an easy solution for now, InterMapper Database simply orders the events as they arrive, by adding an extra decimal value at the end of the time; i.e. .00000, .00001, .00002, etc.
Database Module
InterMapper Database initially uses a PostgreSQL database (either the default one shipped with IMDC, or an external one) for storage. On installation or first startup, it creates a new database with the name "InterMapper Database", and sets up any default values and lookup tables. Support for other database managers will be added in future releases.
Configuration Module
The first version of InterMapper Database has a web-based admin interface. It is not designed to construct queries or create reports; its only purpose is to set configuration options. When InterMapper Database is first started, a wizard walks the customer through setup steps to get it up and running.
Notes
If IMDatabase ever shuts down uncleanly, the built-in DB (if used) process must be killed manually before IMDC can start again
With the default configuration files on MacOS X, the built-in database may not start if you have multiple applications using postgresql. In this situation, the shared memory max must be increased manually. To increase the shared memory max, follow these instructions::
OSX 10.3.9 and later
--------------------------------------------
1. Open /etc/sysctl.conf, or create it if it doesn't exist.
2. Search through the file to find lines like the following:
kern.sysv.shmmax=4194304
kern.sysv.shmmin=1
kern.sysv.shmmni=32
kern.sysv.shmseg=8
kern.sysv.shmall=1024
If those lines don't exist, create them at the top of the file.
3. Change the values to these:
kern.sysv.shmmax=33554432
kern.sysv.shmmin=1
kern.sysv.shmmni=128
kern.sysv.shmseg=64
kern.sysv.shmall=8192
This increases the maximum shared memory to 32MB, from the default of 4MB.
It also increases the number of available shared memory handles and pages.
4. Save the file and reboot your machine.
Known bugs
- If the address of an InterMapper server reporting to IM Database changes, IMDatabase is not notified of the change and will continue to try to query the old address. Workaround: Stop and restart reporting in the Database Server pane in the server settings in InterMapper.