Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions _includes/head.html
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@
{% include globals.html %}
{% include skins.html %}

<link rel="stylesheet" href="{{ '/assets/css/log.css' | relative_url}}">
<link rel="preload" href="https://cdn.jsdelivr.net/npm/@fortawesome/fontawesome-free@5/css/all.min.css" as="style" onload="this.onload=null;this.rel='stylesheet'">
<noscript><link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/@fortawesome/fontawesome-free@5/css/all.min.css"></noscript>

Expand Down
25 changes: 25 additions & 0 deletions _plugins/log.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
require "cgi"

# Liquid block to render log excerpts
#
# {% log %}
# New lines are preserved.
# Long lines will be wrapped when displayed on screen.
# {% endlog %}
#
# Will be rendered in the browser to:
#
# |<--------- screen size --------->|
# | New lines are preserved. |
# | Long lines will be wrapped when |
# | displayed on screen. |
class LogBlock < Liquid::Block
def render(context)
output = '<blockquote class="log">'
output << CGI.escapeHTML(super.strip).gsub(/\n[[:blank:]]*/, "<br/>")
output << "</blockquote>"
output
end
end

Liquid::Template.register_tag("log", LogBlock)
12 changes: 12 additions & 0 deletions assets/css/log.scss
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
---
search: false
toc: false
---

@import "minimal-mistakes/variables";

blockquote.log {
text-indent: 2em hanging each-line;
font-family: $monospace;
white-space: pre-wrap;
}
Original file line number Diff line number Diff line change
Expand Up @@ -21,9 +21,9 @@ on the receiver side.

The following is a sample log message in EWMM format.

><13>1 2018-05-13T13:27:50.993+00:00 my-host @syslog-ng - - -
>{"MESSAGE":"<34>Oct 11 22:14:15 mymachine su: 'su root' failed for username on
>/dev/pts/8","HOST_FROM":"my-host","HOST":"my-host","FILE_NAME":"/tmp/in","._TAGS":".source.s_file"}
{% log %}
<13>1 2018-05-13T13:27:50.993+00:00 my-host @syslog-ng - - - {"MESSAGE":"<34>Oct 11 22:14:15 mymachine su: 'su root' failed for username on /dev/pts/8","HOST_FROM":"my-host","HOST":"my-host","FILE_NAME":"/tmp/in","._TAGS":".source.s_file"}
{% endlog %}

The message has the following parts:

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -71,29 +71,33 @@ To check the status of {{ site.product.short_name }} service

- **active (running)** - {{ site.product.short_name }} service is up and running

Example: {{ site.product.short_name }} service active

> syslog-ng.service - System Logger Daemon
> Loaded: loaded (/lib/systemd/system/syslog-ng.service; enabled; vendor preset: enabled)
> Active: active (running) since Tue 2019-06-25 08:58:09 CEST; 5s ago
> Main PID: 6575 (syslog-ng)
> Tasks: 3
> Memory: 13.3M
> CPU: 268ms
> CGroup: /system.slice/syslog-ng.service
> 6575 /opt/syslog-ng/libexec/syslog-ng -F --no-caps --enable-core
Example: {{ site.product.short_name }} service active

{% log %}
syslog-ng.service - System Logger Daemon
Loaded: loaded (/lib/systemd/system/syslog-ng.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2019-06-25 08:58:09 CEST; 5s ago
Main PID: 6575 (syslog-ng)
Tasks: 3
Memory: 13.3M
CPU: 268ms
CGroup: /system.slice/syslog-ng.service
6575 /opt/syslog-ng/libexec/syslog-ng -F --no-caps --enable-core
{% endlog %}

- **inactive (dead)** - syslog-ng service is stopped

Example: {{ site.product.short_name }} status inactive

> syslog-ng.service - System Logger Daemon
> Loaded: loaded (/lib/systemd/system/syslog-ng.service; enabled; vendor preset: enabled)
> Active: inactive (dead) since Tue 2019-06-25 09:14:16 CEST; 2min 18s ago
> Process: 6575 ExecStart=/opt/syslog-ng/sbin/syslog-ng -F --no-caps --enable-core $SYSLOGNG_OPTIONS(code=exited, status=0/SUCCESS)
> Main PID: 6575 (code=exited, status=0/SUCCESS)
> Status: "Shutting down... Tue Jun 25 09:14:16 2019"
> Jun 25 09:14:31 as-syslog-srv systemd: Stopped System Logger Daemon.
{% log %}
syslog-ng.service - System Logger Daemon
Loaded: loaded (/lib/systemd/system/syslog-ng.service; enabled; vendor preset: enabled)
Active: inactive (dead) since Tue 2019-06-25 09:14:16 CEST; 2min 18s ago
Process: 6575 ExecStart=/opt/syslog-ng/sbin/syslog-ng -F --no-caps --enable-core $SYSLOGNG_OPTIONS(code=exited, status=0/SUCCESS)
Main PID: 6575 (code=exited, status=0/SUCCESS)
Status: "Shutting down... Tue Jun 25 09:14:16 2019"
Jun 25 09:14:31 as-syslog-srv systemd: Stopped System Logger Daemon.
{% endlog %}

### Checking the process of {{ site.product.short_name }}

Expand All @@ -105,19 +109,20 @@ ps u `pidof syslog-ng`

Expected output example:

> USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
>
> syslogng 6709 0.0 0.6 308680 13432 ? Ss 09:17 0:00
> /opt/syslog-ng/libexec/syslog-ng -F --no-caps --enable-core
{% log %}
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
syslogng 6709 0.0 0.6 308680 13432 ? Ss 09:17 0:00 /opt/syslog-ng/libexec/syslog-ng -F --no-caps --enable-core
{% endlog %}

```bash
ps axu | grep syslog-ng | grep -v grep
```

Expected output example:

> syslogng 6709 0.0 0.6 308680 13432 ? Ss 09:17 0:00
> /opt/syslog-ng/libexec/syslog-ng -F --no-caps --enable-core
{% log %}
syslogng 6709 0.0 0.6 308680 13432 ? Ss 09:17 0:00 /opt/syslog-ng/libexec/syslog-ng -F --no-caps --enable-core
{% endlog %}

### Checking the internal logs of {{ site.product.short_name }}**

Expand Down Expand Up @@ -154,10 +159,11 @@ If the numbers are changing, {{ site.product.short_name }} is processing the mes

Example: output example

> Every 2.0s: /opt/syslog-ng/sbin/syslog-ng-ctl stats | grep
> ^center Tue Jun 25 10:33:25 2019
> center;;queued;a;processed;112
> center;;received;a;processed;28
{% log %}
Every 2.0s: /opt/syslog-ng/sbin/syslog-ng-ctl stats | grep ^center Tue Jun 25 10:33:25 2019
center;;queued;a;processed;112
center;;received;a;processed;28
{% endlog %}

### Source statistics

Expand All @@ -173,11 +179,12 @@ If the numbers are changing, {{ site.product.short_name }} is receiving messages

Example: output example

> Every 2.0s: /opt/syslog-ng/sbin/syslog-ng-ctl stats | grep
> ^source Tue Jun 25 10:40:50 2019
> source;s_null;;a;processed;0
> source;s_net;;a;processed;0
> source;s_local;;a;processed;90
{% log %}
Every 2.0s: /opt/syslog-ng/sbin/syslog-ng-ctl stats | grep ^source Tue Jun 25 10:40:50 2019
source;s_null;;a;processed;0
source;s_net;;a;processed;0
source;s_local;;a;processed;90
{% endlog %}

### Destination statistics

Expand All @@ -193,12 +200,13 @@ If the numbers are changing, {{ site.product.short_name }} is receiving messages

Example: output example

> Every 2.0s: /opt/syslog-ng/sbin/syslog-ng-ctl stats | grep
> ^destination Tue Jun 25 10:41:02 2019
> destination;d_logserver2;;a;processed;90
> destination;d_messages;;a;processed;180
> destination;d_logserver;;a;processed;90
> destination;d_null;;a;processed;0
{% log %}
Every 2.0s: /opt/syslog-ng/sbin/syslog-ng-ctl stats | grep ^destination Tue Jun 25 10:41:02 2019
destination;d_logserver2;;a;processed;90
destination;d_messages;;a;processed;180
destination;d_logserver;;a;processed;90
destination;d_null;;a;processed;0
{% endlog %}

**NOTE:** If you find error messages in the internal logs, messages are not
processed by {{ site.product.short_name }} or you encounter any issue, you have the
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,18 +21,19 @@ macros to every message the comes through the connection later on:

- PROXY_DSTPORT (the destination port of the proxy)

>**NOTE:** Consider the following about macros and headers:
>
>**NOTE:** Consider the following about macros and headers:
>
>- When the proxy protocol header is PROXY UNKNOWN, no additional macros are added.
>
>
>- When {{ site.product.short_name }} cannot parse a proxy protocol header, the connection is closed:
>
> [2020-11-20T17:33:22.189458] PROXY protocol header received; line='PROXYdsfj'
> [2020-11-20T17:33:22.189475] Error parsing PROXY protocol header;
> [2020-11-20T17:33:22.189517] Syslog connection closed; fd='13',
> client='AF_INET(127.0.0.1:51665)', local='AF_INET(0.0.0.0:6666)'
> [2020-11-20T17:33:22.189546] Freeing PROXY protocol source driver; driver='0x7fffcba5bcf0'
>
> {% log %}
> [2020-11-20T17:33:22.189458] PROXY protocol header received; line='PROXYdsfj'
> [2020-11-20T17:33:22.189475] Error parsing PROXY protocol header;
> [2020-11-20T17:33:22.189517] Syslog connection closed; fd='13', client='AF_INET(127.0.0.1:51665)', local='AF_INET(0.0.0.0:6666)'
> [2020-11-20T17:33:22.189546] Freeing PROXY protocol source driver; driver='0x7fffcba5bcf0'
> [2020-11-20T17:33:22.189600] Closing log transport fd; fd='13'
> {% endlog %}
{: .notice--info}

**NOTE:** Since the driver only implements version 1 of the protocol, it
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -76,9 +76,8 @@ the output.
With the PROXY TCP4 192.168.1.1 10.10.0.1 1111 2222 input header, the
output looks like this:

> {"SOURCE":"s_tcp_pp","PROXIED_SRCPORT":"1111","PROXIED_SRCIP":"192.168.1.1",
>"PROXIED_IP_VERSION":"4","PROXIED_DSTPORT":"2222","PROXIED_DSTIP":"10.10.0.1",
>"PROGRAM":"TestMsg","MESSAGE":"","LEGACY_MSGHDR":"TestMsg",
>"HOST_FROM":"localhost","HOST":"localhost"}
{% log %}
{"SOURCE":"s_tcp_pp","PROXIED_SRCPORT":"1111","PROXIED_SRCIP":"192.168.1.1", "PROXIED_IP_VERSION":"4","PROXIED_DSTPORT":"2222","PROXIED_DSTIP":"10.10.0.1", "PROGRAM":"TestMsg","MESSAGE":"","LEGACY_MSGHDR":"TestMsg", "HOST_FROM":"localhost","HOST":"localhost"}
{% endlog %}

Note that the macros that {{ site.product.short_name }} adds to the message appear in the output.
27 changes: 6 additions & 21 deletions doc/_admin-guide/060_Sources/102_osquery/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ The osquery() source of {{ site.product.short_name }} allows you read the result
periodical osquery queries (from the
/var/log/osquery/osqueryd.results.log file) and automatically parse the
messages (if you want to use {{ site.product.short_name }} to send log messages to
osquery, read this blogpost).
osquery, read this blogpost).

For example, you can:

Expand Down Expand Up @@ -100,16 +100,9 @@ the outgoing message will be a well-formed JSON message.

#### Input message

>{"name":"pack_osquery-monitoring_osquery_info","hostIdentifier":"testhost",
>"calendarTime":"Fri Jul 21 10:04:41 2017 >UTC","unixTime":"1500631481",
>"decorations":{"host_uuid":"4C4C4544-004D-3610-8043-C2C04F4D3332",
>"username":"myuser"},>"columns":{"build_distro":"xenial",
>"build_platform":"ubuntu",>"config_hash":"43cd1c6a7d0c283e21e026a53e619b2e582e94ee",
>"config_valid":"1","counter":"4","extensions":"active",
>"instance_id":"d0c3eb0d-f8e0-4bea-868b-18a2c61b438d","pid":"19764",
>"resident_size":"26416000",>"start_time":"1500629552","system_time":"223",
>"user_time":"476","uuid":"4C4C4544-004D-3610-8043-C2C04F4D3332",
>"version":"2.5.0","watcher":"19762"},"action":"added"}
{% log %}
{"name":"pack_osquery-monitoring_osquery_info","hostIdentifier":"testhost", "calendarTime":"Fri Jul 21 10:04:41 2017 >UTC","unixTime":"1500631481", "decorations":{"host_uuid":"4C4C4544-004D-3610-8043-C2C04F4D3332", "username":"myuser"},>"columns":{"build_distro":"xenial", "build_platform":"ubuntu",>"config_hash":"43cd1c6a7d0c283e21e026a53e619b2e582e94ee", "config_valid":"1","counter":"4","extensions":"active", "instance_id":"d0c3eb0d-f8e0-4bea-868b-18a2c61b438d","pid":"19764", "resident_size":"26416000",>"start_time":"1500629552","system_time":"223", "user_time":"476","uuid":"4C4C4544-004D-3610-8043-C2C04F4D3332", "version":"2.5.0","watcher":"19762"},"action":"added"}
{% endlog %}

#### {{ site.product.short_name }} configuration

Expand Down Expand Up @@ -138,16 +131,8 @@ log {

#### Outgoing message

>Outgoing message; message='{"_osquery":{"unixTime":"1500631481",
>"name":"pack_osquery-monitoring_osquery_info","hostIdentifier":"testhost",
>"decorations":{"username":"myuser","host_uuid":"4C4C4544-004D-3610-8043-C2C04F4D3332"},
>"columns":{"watcher":"19762","version":"2.5.0","uuid":"4C4C4544-004D-3610-8043-C2C04F4D3332",
>"user_time":"476","system_time":"223","start_time":"1500629552",
>"resident_size":"26416000","pid":"19764","instance_id":"d0c3eb0d-f8e0-4bea-868b-18a2c61b438d",
>"extensions":"active","counter":"4","config_valid":"1",
>"config_hash":"43cd1c6a7d0c283e21e026a53e619b2e582e94ee","build_platform":"ubuntu",
>"build_distro":"xenial"},"calendarTime":"Fri Jul 21 10:04:41 2017 UTC","action":"added"}}\x0a'
> Outgoing message; message='{"_osquery":{"unixTime":"1500631481", "name":"pack_osquery-monitoring_osquery_info","hostIdentifier":"testhost", "decorations":{"username":"myuser","host_uuid":"4C4C4544-004D-3610-8043-C2C04F4D3332"}, "columns":{"watcher":"19762","version":"2.5.0","uuid":"4C4C4544-004D-3610-8043-C2C04F4D3332", "user_time":"476","system_time":"223","start_time":"1500629552", "resident_size":"26416000","pid":"19764","instance_id":"d0c3eb0d-f8e0-4bea-868b-18a2c61b438d", "extensions":"active","counter":"4","config_valid":"1", "config_hash":"43cd1c6a7d0c283e21e026a53e619b2e582e94ee","build_platform":"ubuntu", "build_distro":"xenial"},"calendarTime":"Fri Jul 21 10:04:41 2017 UTC","action":"added"}}\x0a'

To configure a destination to send the log messages to Elasticsearch,
see elasticsearch-http: Sending messages to Elasticsearch HTTP Bulk API.
see elasticsearch-http: Sending messages to Elasticsearch HTTP Bulk API.
For other destinations, see destination: Forward, send, and store log messages.
Original file line number Diff line number Diff line change
Expand Up @@ -64,11 +64,13 @@ set in the message, and uses the IP address of the {{ site.product.short_name }}
the hostname (to use the hostname instead of the IP address, set the
use-dns() or use-fqdn() options in the Python source).

>msg_ietf = LogMessage.parse('<165>1 2003-10-11T22:14:15.003Z mymachine.example.com
>evntslog - ID47 [exampleSDID@32473 iut="3" eventSource="Application" eventID="1011"]
>An application event log entry', self.parse_options)
>msg_bsd = LogMessage.parse('<34>Oct 11 22:14:15 mymachine su: \'su root\' failed for
>lonvick on /dev/pts/8', self.parse_options)
```python
msg_ietf = LogMessage.parse('<165>1 2003-10-11T22:14:15.003Z mymachine.example.com \
evntslog - ID47 [exampleSDID@32473 iut="3" eventSource="Application" eventID="1011"] \
An application event log entry', self.parse_options)
msg_bsd = LogMessage.parse('<34>Oct 11 22:14:15 mymachine su: \'su root\' failed for \
lonvick on /dev/pts/8', self.parse_options)
```

## set_pri() method

Expand Down
8 changes: 5 additions & 3 deletions doc/_admin-guide/060_Sources/155_stdin/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,8 @@ used to collect a test message:
$ echo "this is a test message" | ./syslog-ng -Fe --no-caps
```

> [2017-11-14T13:47:16.757938] syslog-ng starting up; version='3.12.1'
> [2017-11-14T13:47:16.758195] syslog-ng shutting down; version='3.12.1'
> Nov 14 13:47:16 testserver this is a test message
{% log %}
[2017-11-14T13:47:16.757938] syslog-ng starting up; version='3.12.1'
[2017-11-14T13:47:16.758195] syslog-ng shutting down; version='3.12.1'
Nov 14 13:47:16 testserver this is a test message
{% endlog %}
6 changes: 3 additions & 3 deletions doc/_admin-guide/070_Destinations/310_syslog-ng/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,9 +28,9 @@ to properly handle the messages.

The following is a sample log message in EWMM format.

> <13>1 2018-05-13T13:27:50.993+00:00 my-host @syslog-ng - - -
> {"MESSAGE":"<34>Oct 11 22:14:15 mymachine su: 'su root' failed for username on
> /dev/pts/8","HOST_FROM":"my-host","HOST":"my-host","FILE_NAME":"/tmp/in","._TAGS":".source.s_file"}
{% log %}
<13>1 2018-05-13T13:27:50.993+00:00 my-host @syslog-ng - - - {"MESSAGE":"<34>Oct 11 22:14:15 mymachine su: 'su root' failed for username on /dev/pts/8","HOST_FROM":"my-host","HOST":"my-host","FILE_NAME":"/tmp/in","._TAGS":".source.s_file"}
{% endlog %}

**Declaration**

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,10 +20,14 @@ The following command will list the disk-buffer files in use:

The example output will look like the following:

> afsocket_dd_qfile(stream,10.21.10.20:601) ==> /opt/syslog-ng/var/syslog-ng-00000.rqf

**NOTE:** If you receive the following error message instead of the example
output, install a vim-common package on your system:
{: .notice--info}
{% log %}
afsocket_dd_qfile(stream,10.21.10.20:601) ==> /opt/syslog-ng/var/syslog-ng-00000.rqf
{% endlog %}

> **NOTE:** If you receive the following error message instead of the example
> output, install a vim-common package on your system:
>
> {% log %}
> xxd: command not found
> {% endlog %}
{: .notice--info}
Original file line number Diff line number Diff line change
Expand Up @@ -24,9 +24,11 @@ disk-buffer files used in {{ site.product.short_name }}:
dqtool cat /opt/syslog-ng/var/syslog-ng-00000.rqf
```

>Reliable disk-buffer state loaded; filename='/opt/syslog-ng/var/syslog-ng-00000.rqf',
>queue_length='2952', size='-437712'
>Jul 31 12:33:48.226 10.21.10.10 <382019-07-31T12:33:36 localhost prg00000[1234]:
>seq: 0000000838, thread: 0000, runid: 1564569216, stamp: 2019-07-31T12:33:36
>PADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADD
>...
{% log %}
Reliable disk-buffer state loaded; filename='/opt/syslog-ng/var/syslog-ng-00000.rqf',
queue_length='2952', size='-437712'
Jul 31 12:33:48.226 10.21.10.10 <382019-07-31T12:33:36 localhost prg00000[1234]:
seq: 0000000838, thread: 0000, runid: 1564569216, stamp: 2019-07-31T12:33:36
PADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADD
...
{% endlog %}
Original file line number Diff line number Diff line change
Expand Up @@ -156,12 +156,14 @@ To process the messages from an orphan disk-buffer file using a separate
Example: output displaying newly created empty disk-buffer file and
connection established to remote destination

>Follow-mode file source not found, deferring open; filename='/no_such_file_or.dir'
>Reliable disk-buffer state saved; filename='/tmp/qdisk/syslog-ng-00000.rqf', qdisk_length='0'
>No server license found, running in client mode;
>syslog-ng starting up; version='7.0.20', cfg-fingerprint='eaa03b9efb88b87d7c1b0ce7efd042ed8ac0c013', >cfg-nonce-ndx='0', cfg-signature='c0327a7f7e6418ce0399a75089377dfb662bb072'
>FIPS information; FIPS-mode='disabled'
>Syslog connection established; fd='7', server='AF_INET(10.21.10.20:514)', local='AF_INET(0.0.0.0:0)'
{% log %}
Follow-mode file source not found, deferring open; filename='/no_such_file_or.dir'
Reliable disk-buffer state saved; filename='/tmp/qdisk/syslog-ng-00000.rqf', qdisk_length='0'
No server license found, running in client mode;
syslog-ng starting up; version='7.0.20', cfg-fingerprint='eaa03b9efb88b87d7c1b0ce7efd042ed8ac0c013', >cfg-nonce-ndx='0', cfg-signature='c0327a7f7e6418ce0399a75089377dfb662bb072'
FIPS information; FIPS-mode='disabled'
Syslog connection established; fd='7', server='AF_INET(10.21.10.20:514)', local='AF_INET(0.0.0.0:0)'
{% endlog %}

7. To stop {{ site.product.short_name }}, press CTRL+C.

Expand Down
Loading