Program filter syslog-ng

As a result, the queue file is not truncated automatically, but grows until it reaches the maximal configured size, after which the write head will wrap around, later followed by the read head.

In case of normal disk queue files, growth in size is not so apparent, as the disk-based queue file is only used if the in-memory overflow buffer fills up. Once the destination sends messages faster than the incoming message rate, the queue will start to empty, and when the read and write heads of the queue reach the same position, the queue files are finally truncated.

Note that if a queue file becomes corrupt, syslog-ng OSE starts a new one. This might lead to the queue files consuming more space in total than their maximal configured size and the number of configured queue files multiplied together.

Using filters describes how to configure and use filters. Combining filters with boolean operators shows how to create complex filters using boolean operators. Comparing macro values in filters explains how to evaluate macros in filters. Using wildcards, special characters, and regular expressions in filters provides tips on using regular expressions. Tagging messages explains how to tag messages and how to filter on the tags. Filter functions is a detailed description of the filter functions available in syslog-ng OSE.

Filters perform log routing within syslog-ng: a message passes the filter if the filter expression is true for the particular message. If a log statement includes filters, the messages are sent to the destinations only if they pass all filters of the log path. For example, a filter can select only the messages originating from a particular host. Complex filters can be created using filter functions and logical boolean expressions. To define a filter, add a filter statement to the syslog-ng configuration file using the following syntax:.

You can also define the filter inline. For details, see Defining configuration objects inline. The following filter statement selects the messages that contain the word deny and come from the host example. Feedback Terms of Use Privacy. Upgrade to IE 11 Click here. Upgrade to Chrome Click here. If you continue in IE8, 9, or 10 you will not be able to take full advantage of all our great self service features. Table of Contents. Preface Summary of contents Target audience and prerequisites Products covered in this guide Summary of changes Feedback Acknowledgments.

Use the not operator to invert filters, for example, to select the messages that were not sent by host example1 :. However, to select the messages that were not sent by host example1 or example2 , you have to use the and operator that's how boolean logic works :. For a complete description on filter functions, see Filter functions. For details on using the match filter function, see match. Filters are often used together with log path flags. For details, see Log path flags. Starting with syslog-ng OSE version 3.

String comparison is alphabetical: it determines if a string is alphabetically greater or equal to another string. Use the following syntax to compare macro values or templates. For details on macros and templates, see Customize message format using macros and templates.

The following expression selects log messages that do not contain a PID. Also, it uses a template as the left argument of the operator and compares the values as strings:. Using comparator operators can be equivalent to using filter functions, but is somewhat slower. You can use any macro in the expression, including user-defined macros from parsers and results of pattern database classifications.

The results of filter functions are boolean values, so they cannot be compared to other values. The host , match , and program filter functions accept regular expressions as parameters. The exact type of the regular expression to use can be specified with the type option. This expression matches for the following strings: ilter , filter , ffilter , and so on.

To achieve the wildcard functionality commonly represented by the asterisk character in other applications, use. Alternatively, if you do not need regular expressions, only wildcards, use type glob in your filter:.

The following filter matches on hostnames starting with the myhost string, for example, on myhost-1 , myhost-2 , and so on. For details on using regular expressions in syslog-ng OSE, see Using wildcards, special characters, and regular expressions in filters. In syslog-ng OSE 3. For example, to filter on carriage returns, use the following filter:. Feedback Terms of Use Privacy. Upgrade to IE 11 Click here. However, to select the messages that were not sent by host example1 or example2 , you have to use the and operator that's how boolean logic works :.

For a complete description on filter functions, see Filter functions. The following filter statement selects the messages that contain the word deny and come from the host example.

For details on using the match filter function, see match. Filters are often used together with log path flags. For details, see Log path flags. Starting with syslog-ng OSE version 3. String comparison is alphabetical: it determines if a string is alphabetically greater or equal to another string.

Use the following syntax to compare macro values or templates. For details on macros and templates, see Customize message format using macros and templates.

The following expression selects log messages that do not contain a PID. Also, it uses a template as the left argument of the operator and compares the values as strings:.

Using comparator operators can be equivalent to using filter functions, but is somewhat slower. You can use any macro in the expression, including user-defined macros from parsers and results of pattern database classifications. The results of filter functions are boolean values, so they cannot be compared to other values.

The host , match , and program filter functions accept regular expressions as parameters. The exact type of the regular expression to use can be specified with the type option.

This expression matches for the following strings: ilter , filter , ffilter , and so on. To achieve the wildcard functionality commonly represented by the asterisk character in other applications, use. Alternatively, if you do not need regular expressions, only wildcards, use type glob in your filter:. The following filter matches on hostnames starting with the myhost string, for example, on myhost-1 , myhost-2 , and so on.

For details on using regular expressions in syslog-ng OSE, see Using wildcards, special characters, and regular expressions in filters.

In syslog-ng OSE 3. For example, to filter on carriage returns, use the following filter:. You can label the messages with custom tags. Tags are simple labels, identified by their names, which must be unique. The syslog-ng OSE application waits for this number of lines to accumulate and sends them off in a single batch. Increasing this number increases throughput as more messages are sent in a single batch, but also increases message latency. The syslog-ng OSE application flushes the messages if it has sent flush-lines number of messages, or the queue became empty.

If you stop or reload syslog-ng OSE or in case of network sources, the connection with the client is closed, syslog-ng OSE automatically sends the unsent messages to the destination. For optimal performance when sending messages to an syslog-ng OSE server, make sure that the value of flush-lines is smaller than the window size set in the log-iw-size option in the source of your server.

Description: The syslog-ng application can store fractions of a second in the timestamps according to the ISO format. The frac-digits parameter specifies the number of digits stored. The digits storing the fractions are padded by zeros if the original timestamp of the message specifies only seconds.

Fractions can always be stored for the time the message was received. Note that syslog-ng can add the fractions to non-ISO timestamps as well. Description: This option makes it possible to execute external programs when the relevant driver is initialized or torn down. The hook-commands can be used with all source and destination drivers with the exception of the usertty and internal drivers.

NOTE: The syslog-ng OSE application must be able to start and restart the external program, and have the necessary permissions to do so. Description: Defines the external program that is executed as syslog-ng OSE starts. Description: Defines the external program that is executed as syslog-ng OSE stops.

Description: Defines an external program that is executed when the syslog-ng OSE configuration is initiated, for example, on startup or during a syslog-ng OSE reload. Description: Defines an external program that is executed when the syslog-ng OSE configuration is stopped or torn down, for example, on shutdown or during a syslog-ng OSE reload. Whenever the syslog-ng OSE created rule is there, packets can flow, otherwise the port is closed. Description: The number of messages that the output queue can store.

Description: By default, when program starts an external application or script, it inherits the entire environment of the parent process that is, syslog-ng OSE. Use inherit-environment no to prevent this. Description: Specifies whether the external program should be closed when syslog-ng OSE is reloaded. Description: An alias for the obsolete mark option, retained for compatibility with syslog-ng version 1. The number of seconds between two MARK messages.

MARK messages are generated when there was no message traffic to inform the receiver that the connection is still alive. If set to zero 0 , no MARK messages are sent. If mark-freq is not defined in the destination, then the mark-freq will be inherited from the global options.

If the destination uses internal mark-mode , then the global mark-freq will be valid does not matter what mark-freq set in the destination side. Description: The mark-mode option can be set for the following destination drivers: file , program , unix-dgram , unix-stream , network , pipe , syslog and in global option.

This mode only yields the mark messages from internal source. This is the mode as syslog-ng OSE 3. MARK signal from internal source will be dropped.

MARK signal can be sent by the following destination drivers: network , syslog , program , file , pipe , unix-stream , unix-dgram.

For example MARK is generated even if messages were received from tcp. If an explicit mark-mode is not given to the drivers where none is the default value, then none will be used. Note that setting the global mark-mode to global causes a syntax error in syslog-ng OSE. In case of dst-idle , host-idle and periodical , the MARK message will not be written in the destination, if it is not open yet.

Available in syslog-ng OSE 3. Note that in earlier versions of syslog-ng OSE, the default for the mark-mode of the program destination was none. Now it defaults to the global setting, so the program destination will emit a MARK message every mark-freq interval. To avoid such messages, set the mark-mode option of the destination to none.

Description: If several identical log messages would be sent to the destination without any other messages between the identical messages for example, an application repeated an error message ten times , syslog-ng can suppress the repeated messages and send the message only once, followed by the Last message repeated n times. The parameter of this option specifies the number of seconds syslog-ng waits for identical messages.

Description: Specifies a template defining the logformat to be used in the destination. Macros are described in Macros of syslog-ng OSE. Please note that for network destinations it might not be appropriate to change the template as it changes the on-wire format of the syslog protocol which might not be tolerated by stock syslog receivers like syslogd or syslog-ng itself.

For network destinations make sure the receiver can cope with the custom format defined. Description: Turns on escaping for the ' , " , and backspace characters in templated output files.