NXLog Documentation

NXLog Community Edition Reference Manual

Man Pages

nxlog(8) — The manual page for the main NXLog agent
nxlog-processor(8) — The manual page for the NXLog offline processor

Configuration

An NXLog configuration consists of global directives, module instances, and routes. The following sections list the core NXLog directives provided. Additional directives are provided at the module level. A valid configuration must contain at least one input module instance and at least one output module instance.

A module instance name may contain letters, digits, periods (.), and underscores (_). The first character in a module instance name must be a letter or an underscore. The corresponding regular expression is [a-zA-Z_][a-zA-Z0-9._]*.

A route instance name may contain letters, digits, periods (.), and underscores (_). The first character in a route instance name must be a letter, a digit, or an underscore. The corresponding regular expression is [a-zA-Z0-9_][a-zA-Z0-9._]*.

General Directives

The following directives can be used throughout the configuration file. These directives are handled by the configuration parser, and substitutions occur before the configuration check.

define

Use this directive to configure a constant or macro to be used later. Refer to a define by surrounding the name with percent signs (%). Enclose a group of statements with curly braces ({}).

Example 1. Using the define Directive

This configuration shows three example defines: BASEDIR is a constant, IMPORTANT is a statement, and WARN_DROP is a group of statements.

nxlog.conf

define BASEDIR /var/log
define IMPORTANT if $raw_event =~ /important/ \
                    $Message = 'IMPORTANT ' + $raw_event;
define WARN_DROP { log_warning("dropping message"); drop(); }

<Input messages>
    Module  im_file
    File    '%BASEDIR%/messages'
</Input>

<Input proftpd>
    Module  im_file
    File    '%BASEDIR%/proftpd.log'
    <Exec>
        %IMPORTANT%
        if $raw_event =~ /dropme/ %WARN_DROP%
    </Exec>
</Input>

include

This directive allows a specified file to be included in the current configuration file. Wildcarded filenames are supported.

The SpoolDir directive only takes effect after the configuration is parsed, so relative paths specified with the include directive must be relative to the working directory NXLog was started from.

Example 2. Using the include Directive

This example includes a file relative to the directory NXLog is started from:

nxlog.conf

include modules/module1.conf

This example includes all matching files and uses an absolute path:

nxlog.conf

include /etc/nxlog.d/*.conf

Global Directives

CacheDir: This directive specifies a directory where the cache file (configcache.dat) should be written. This directive has a compiled-in value which is used by default.

FlowControl: This optional boolean directive specifies whether all input and processor modules should use flow control. This defaults to TRUE. See the description of the module level FlowControl directive for more information.

Group: Similar to User, NXLog will set the group ID to run under. The group can be specified by name or numeric ID. This directive has no effect when running on the Windows platform or with [nxlog-processor-8].

IgnoreErrors: If set to FALSE, NXLog will stop when it encounters a problem with the configuration file (such as an invalid module directive) or if there is any other problem which would prevent all modules functioning correctly. If set to TRUE, NXLog will start after logging the problem. The default value is TRUE.

LogFile: NXLog will write its internal log to this file. If this directive is not specified, self logging is disabled. Note that the im_internal module can also be used to direct internal log messages to files or different output destinations, but this does not support log level below INFO. This LogFile directive is especially useful for debugging.

LogLevel: This directive has five possible values: CRITICAL, ERROR, WARNING, INFO, and DEBUG. It will set both the logging level used for LogFile and the standard output if NXLog is started in the foreground. The default LogLevel is INFO.

ModuleDir: By default the NXLog binaries have a compiled-in value for the directory to search for loadable modules. This can be overridden with this directive. The module directory contains sub-directories for each module type (extension, input, output, and processor), and the module binaries are located in those.

NoCache: Some modules save data to a cache file which is persisted across a shutdown/restart. Modules such as im_file will save the file position in order to continue reading from the same position after a restart as before. This caching mechanism can be explicitly turned off with this directive. This is mostly useful with [nxlog-processor-8] in offline mode. If this boolean directive is not specified, it defaults to FALSE (caching is enabled). Note that many input modules, such as im_file, provide a SavePos directive that can be used to disable the position cache for a specific module instance. SavePos has no effect if the cache is disabled globally with NoCache TRUE.

NoFreeOnExit: This directive is for debugging. When set to TRUE, NXLog will not free module resources on exit, allowing valgrind to show proper stack trace locations in module function calls. The default value is FALSE.

Panic: A panic condition is a critical state which usually indicates a bug. Assertions are used in NXLog code for checking conditions where the code will not work unless the asserted condition is satisfied, and for security. Failing assertions result in a panic and suggest a bug in the code. A typical case is checking for NULL pointers before pointer dereference. This directive can take three different values: HARD, SOFT, or OFF. HARD will cause an abort in case the assertion fails. This is how most C based programs work. SOFT will cause an exception to be thrown at the place of the panic/assertion. In case of NULL pointer checks this is identical to a NullPointerException in Java. It is possible that NXLog can recover from exceptions and can continue to process log messages, or at least the other modules can. In case of assertion failure the location and the condition is printed at CRITICAL log level in HARD mode and ERROR log level in SOFT mode. If Panic is set to OFF, the failing condition is printed in the logs but the execution will continue on the normal code path. Most of the time this will result in a segmentation fault or other undefined behavior, though in some cases turning off a buggy assertion or panic will solve the problems caused by it in HARD/SOFT mode. The default value for Panic is SOFT.

PidFile: Under Unix operating systems, NXLog writes a PID file as other system daemons do. The default PID file can be overridden with this directive in case multiple daemon instances need to be running. This directive has no effect when running on the Windows platform or with [nxlog-processor-8].

RootDir: NXLog will set its root directory to the value specified with this directive. If SpoolDir is also set, this will be relative to the value of RootDir (chroot() is called first). This directive has no effect when running on the Windows platform or with the [nxlog-processor-8].

SpoolDir: NXLog will change its working directory to the value specified with this directive. This is useful with files created through relative filenames (for example, with om_file) and in case of core dumps. This directive has no effect with the [nxlog-processor-8].

SuppressRepeatingLogs: Under some circumstances it is possible for NXLog to generate an extreme amount of internal logs consisting of the same message due to an incorrect configuration or a software bug. In this case, the LogFile can quickly consume the available disk space. With this directive, NXLog will write at most 2 lines per second if the same message is generated successively, by logging "last message repeated n times" messages. If this boolean directive is not specified, it defaults to TRUE (suppression of repeating messages is enabled).

Threads: This directive specifies the number of worker threads to use. The number of the worker threads is calculated and set to an optimal value if this directive is not defined. Do not set this unless you know what you are doing.

User: NXLog will drop to the user specified with this directive. This is useful if NXLog needs privileged access to some system resources (such as kernel messages or to bind a port below 1024). On Linux systems NXLog will use capabilities to access these resources. In this case NXLog must be started as root. The user can be specified by name or numeric ID. This directive has no effect when running on the Windows platform or with [nxlog-processor-8].

Common Module Directives

The following directives are common to all modules. The Module directive is mandatory.

Module: This mandatory directive specifies which binary should be loaded. The module binary has a .so extension on Unix and a .dll on Windows platforms and resides under the ModuleDir location. Each module binary name is prefixed with im_, pm_, om_, or xm_ (for input, processor, output, and extension, respectively). It is possible for multiple instances to use the same loadable binary. In this case the binary is only loaded once but instantiated multiple times. Different module instances may have different configurations.

FlowControl

This optional boolean directive specifies whether the module instance should use flow control. FlowControl is only valid for Input and Processor modules. By default, FlowControl is TRUE (enabled). This module-level directive can be used to override the global FlowControl directive.

When flow control is in effect, a module (Input or Processor) which tries to forward log data to the next module in the route will be suspended if the next module cannot accept more data. For example, if a network module (such as om_tcp) cannot forward logs because of a network error, the preceding module in the route will be paused. When flow control is disabled, the module will drop the log record if the queue of the next module in the route is full.

Disabling flow control can be useful when multiple output modules are configured to store or forward log data. When flow control is enabled, the output modules will only process log data if all outputs are functional. Consider the case where log data is stored in a file using om_file and also forwarded over the network using om_tcp. When flow control is enabled, a network disconnection will make the data flow stall and log data will not be written into the local file either. With flow control disabled, NXLog will write log data to the file and will drop messages that cannot be forwarded over the network.

Suspending an im_udp instance is ineffective, because UDP provides no receipt acknowledgement. Suspending an im_uds instance when collecting local Syslog messages from the /dev/log Unix domain socket will cause the syslog() system call to block in any programs trying to write to the system log. It is generally recommended to disable flow control in these cases.

InputType

This directive specifies the name of the registered input reader function to be used for parsing raw events from input data. Names are treated case insensitively. This directive is only available for stream oriented input modules: im_file, im_exec, im_ssl, im_tcp, im_udp, and im_uds. These modules work by filling an input buffer with data read from the source. If the read operation was successful (there was data coming from the source), the module calls the specified callback function. If this is not explicitly specified, the module default will be used. Note that im_udp may only work properly if log messages do not span multiple packets and are within the UDP message size limit. Otherwise the loss of a packet may lead to parsing errors.

Modules may provide custom input reader functions. Once these are registered into the NXLog core, the modules listed above will be capable of using these. This makes it easier to implement custom protocols because these can be developed without concern for the transport layer.

The following input reader functions are provided by the NXLog core:

Binary: The input is parsed in the NXLog binary format, which preserves the parsed fields of the event records. The LineBased reader will automatically detect event records in the binary NXLog format, so it is only recommended to configure InputType to Binary if compatibility with other logging software is not required.

Dgram: Once the buffer is filled with data, it is considered to be one event record. This is the default for the im_udp input module, since UDP Syslog messages arrive in separate packets.

LineBased: The input is assumed to contain event records separated by newlines. It can handle both CRLF (Windows) and LF (Unix) line-breaks. Thus if an LF (\n) or CRLF (\r\n) is found, the function assumes that it has reached the end of the event record.

Example 3. TCP Input Assuming NXLog Format

This configuration explicitly specifies the Binary InputType.

nxlog.conf

<Input tcp>
    Module      im_tcp
    Port        2345
    InputType   Binary
</Input>

OutputType

This directive specifies the name of the registered output writer function to be used for formatting raw events when storing or forwarding output. Names are treated case insensitively. This directive is only available for stream oriented output modules: om_file, om_exec, om_ssl, om_tcp, om_udp, and om_uds. These modules work by filling the output buffer with data to be written to the destination. The specified callback function is called before the write operation. If this is not explicitly specified, the module default will be used.

Modules may provide custom output formatter functions. Once these are registered into the NXLog core, the modules listed above will be capable of using these. This makes it easier to implement custom protocols because these can be developed without concern for the transport layer.

The following output writer functions are provided by the NXLog core:

Binary: The output is written in the NXLog binary format which preserves parsed fields of the event records.

Dgram: Once the buffer is filled with data, it is considered to be one event record. This is the default for the om_udp output module, since UDP Syslog messages are sent in separate packets.

LineBased: The output will contain event records separated by newlines. The record terminator is CRLF (\r\n).

Example 4. TCP Output Sending Messages in NXLog Format

This configuration explicitly specifies the Binary OutputType.

nxlog.conf

<Output tcp>
    Module      om_tcp
    Port        2345
    Host        localhost
    OutputType  Binary
</Output>

Exec

The Exec directive/block contains statements in the NXLog language which are executed when a module receives a log message. This directive is available in all input, processor, and output modules. It is not available in most extension modules because these do not handle log messages directly (the xm_multiline and xm_rewrite modules do provide Exec directives).

Example 5. Simple Exec Statement

This statement assigns a value to the $Hostname field in the event record.

nxlog.conf

Exec    $Hostname = 'myhost';

Each directive must be on one line unless it contains a trailing backslash (\) character.

Example 6. Exec Statement Spanning Multiple Lines

This if statement uses line continuation to span multiple lines.

nxlog.conf

Exec    if $Message =~ /something interesting/       \
            log_info("found something interesting"); \
        else                                         \
            log_debug("found nothing interesting");

More than one Exec directive or block may be specified. They are executed in the order of appearance. Each Exec directive must contain a full statement. Therefore it is not possible to split the lines in the previous example into multiple Exec directives. It is only possible to split the Exec directive if it contains multiple statements.

Example 7. Equivalent Use of Statements in Exec

This example shows two equivalent uses of the Exec directive.

nxlog.conf

Exec    log_info("first"); \
        log_info("second");

This produces identical behavior:

nxlog.conf

Exec    log_info("first");
Exec    log_info("second");

The Exec directive can also be used as a block. To use multiple statements spanning more than one line, it is recommended to use the <Exec> block instead. When using a block, it is not necessary to use the backslash (\) character for line continuation.

Example 8. Using the Exec Block

This example shows two equivalent uses of Exec, first as a directive, then as a block.

nxlog.conf

Exec    log_info("first"); \
        log_info("second");

The following Exec block is equivalent. Notice the backslash (\) is omitted.

nxlog.conf

<Exec>
    log_info("first");
    log_info("second");
</Exec>

Schedule

The Schedule block can be used to execute periodic jobs, such as log rotation or any other task. Scheduled jobs have the same priority as the module. The Schedule block has the following directives:

Every: In addition to the crontab format it is possible to schedule execution at periodic intervals. With the crontab format it is not possible to run a job every five days for example, but this directive enables it in a simple way. It takes an integer value with an optional unit. The unit can be one of the following: sec, min, hour, day, or week. If the unit is not specified, the value is assumed to be in seconds.

Exec: The mandatory Exec directive takes one or more NXLog statements. This is the code which is actually being scheduled. Multiple Exec directives can be specified within one Schedule block. See the module-level Exec directive, this behaves the same. Note that it is not possible to use fields in statements here because execution is not triggered by log messages.

First: This directive sets the first execution time. If the value is in the past, the next execution time is calculated as if NXLog has been running since and jobs will not be run to make up for missed events in the past. The directive takes a datetime literal value.

When

This directive takes a value similar to a crontab entry: five space-separated definitions for minute, hour, day, month, and weekday. See the crontab(5) manual for the field definitions. It supports lists as comma separated values and/or ranges. Step values are also supported with the slash. Month and week days are not supported, these must be defined with numeric values. The following extensions are also supported:

@yearly        Run once a year, "0 0 1 1 *".
@annually      (same as @yearly)
@monthly       Run once a month, "0 0 1 * *".
@weekly        Run once a week, "0 0 * * 0".
@daily         Run once a day, "0 0 * * *".
@midnight      (same as @daily)
@hourly        Run once an hour, "0 * * * *".

Example 9. Scheduled Exec Statements

This example shows two scheduled Exec statements in a im_tcp module instance. The first is executed every second, while the second uses a crontab(5) style value.

nxlog.conf

<Input in>
    Module  im_tcp
    Port    2345

    <Schedule>
        Every   1 sec
        First   2010-12-17 00:19:06
        Exec    log_info("scheduled execution at " + now());
    </Schedule>

    <Schedule>
        When    1 */2 2-4 * *
        Exec    log_info("scheduled execution at " + now());
    </Schedule>
</Input>

Route Directives

The following directives can be used in Route blocks. The Path directive is mandatory.

Path

The data flow is defined by the Path directive. First the instance names of Input modules are specified. If more than one Input reads log messages which feed data into the route, then these must be separated by commas. The list of Input modules is followed by an arrow (=>). Either processor modules or output modules follow. Processor modules must be separated by arrows, not commas, because they operate in series, unlike Input and Output modules which work in parallel. Output modules are separated by commas. The Path must specify at least an Input and an Output. The syntax is illustrated by the following:

Path INPUT1[, INPUT2...] => [PROCESSOR1 [=> PROCESSOR2...] =>] OUTPUT1[, OUTPUT2...]

Example 10. Specifying Routes

The following configuration shows modules being used in three different routes.

nxlog.conf

<Input in1>
    Module  im_null
</Input>

<Input in2>
    Module  im_null
</Input>

<Processor p1>
    Module  pm_null
</Processor>

<Processor p2>
    Module  pm_null
</Processor>

<Output out1>
    Module  om_null
</Output>

<Output out2>
    Module  om_null
</Output>

<Route 1>
    # Basic route
    Path    in1 => out1
</Route>

<Route 2>
    # Basic route with one processor module
    Path    in1 => p1 => out1
</Route>

<Route 3>
    # Complex route with multiple input/output/processor modules
    Path    in1, in2 => p1 => p2 => out1, out2
</Route>

Priority

This directive takes an integer value in the range of 1-100 as a parameter, and the default is 10. Log messages in routes with a lower Priority value will be processed before others. Internally, this value is assigned to each module part of the route. The events of the modules are processed in priority order by the NXLog engine. Modules of a route with a lower Priority value (higher priority) will process log messages first.

Example 11. Prioritized Processing

This configuration prioritizes the UDP route over the TCP route in order to minimize loss of UDP Syslog messages when the system is busy.

nxlog.conf

<Input tcpin>
    Module      im_tcp
    Host        localhost
    Port        514
</Input>

<Input udpin>
    Module      im_udp
    Host        localhost
    Port        514
</Input>

<Output tcpfile>
    Module      om_file
    File        "/var/log/tcp.log"
</Output>

<Output udpfile>
    Module      om_file
    File        "/var/log/udp.log"
</Output>

<Route udp>
    Priority    1
    Path        udpin => udpfile
</Route>

<Route tcp>
    Priority    2
    Path        tcpin => tcpfile
</Route>

Language

Types

The following types are provided by the NXLog language.

Unknown: This is a special type for values where the type cannot be determined at compile time and for uninitialized values. The undef literal and fields without a value also have an unknown type. The unknown type can also be thought of as "any" in case of function and procedure API declarations.

Boolean: A boolean value is TRUE, FALSE or undefined. Note that an undefined value is not the same as a FALSE value.

Integer: An integer can hold a signed 64 bit value in addition to the undefined value. Floating point values are not supported.

String: A string is an array of characters in any character set. The binary type should be used for values where the NUL byte can also occur. An undefined string is not the same as an empty string. Strings have a limited length to prevent resource exhaustion problems, this is a compile-time value currently set to 1M.

Datetime: A datetime holds a microsecond value of time elapsed since the Epoch. It is always stored in UTC/GMT.

IPv4 Address: An ip4addr type stores a dotted-quad IPv4 address in an internal format (integer).

IPv6 Address: An ip6addr type stores an IPv6 address in an internal format.

Regular expression: A regular expression type can only be used with the =~ or !~ operators.

Binary: This type can hold an array of bytes.

Variadic arguments: This is a special type only used in function and procedure API declarations to indicate variadic arguments.

Expressions

Literals

Undef

The undef literal has an unknown type. It can be also used in an assignment to unset the value of a field.

Example 12. Un-Setting the Value of a Field

This statement unsets the $ProcessID field.

$ProcessID = undef;

Boolean: A boolean literal is either TRUE or FALSE. It is case-insensitive, so True, False, true, and false are also valid.

Integer: An integer starts with a minus (-) sign if it is negative. A "0X" or "0x" prepended modifier indicates a hexadecimal notation. The "K", "M" and "G" modifiers are also supported; these mean Kilo (1024), Mega (1024^2), or Giga (1024^3) respectively when appended.

Example 13. Setting an Integer Value

This statement uses a modifier to set the $Limit field to 44040192 (42×1024^2).