Syslog support
Syslog integration variants
This section explain different ways to recieve and process syslog with LibreNMS. Except of graylog, all Syslogs variants store their logs in the LibreNMS database. You need to enable the Syslog extension in config.php:
$config['enable_syslog'] = 1;
Traditional Syslog server
syslog-ng
apt-get install syslog-ng-core
yum install syslog-ng
Once syslog-ng is installed, create the config file (/etc/syslog-ng/conf.d/librenms.conf) and paste the following:
source s_net {
tcp(port(514) flags(syslog-protocol));
udp(port(514) flags(syslog-protocol));
};
destination d_librenms {
program("/opt/librenms/syslog.php" template ("$HOST||$FACILITY||$PRIORITY||$LEVEL||$TAG||$R_YEAR-$R_MONTH-$R_DAY $R_HOUR:$R_MIN:$R_SEC||$MSG||$PROGRAM\n") template-escape(yes));
};
log {
source(s_net);
source(s_src);
destination(d_librenms);
};
Next start syslog-ng:
service syslog-ng restart
If no messages make it to the syslog tab in LibreNMS, chances are you experience an issue with SELinux. If so, create a file mycustom-librenms-rsyslog.te , with the following content:
module mycustom-librenms-rsyslog 1.0;
require {
type syslogd_t;
type httpd_sys_rw_content_t;
type ping_exec_t;
class process execmem;
class dir { getattr search write };
class file { append getattr execute open read };
}
#============= syslogd_t ==============
allow syslogd_t httpd_sys_rw_content_t:dir { getattr search write };
allow syslogd_t httpd_sys_rw_content_t:file { open read append getattr };
allow syslogd_t self:process execmem;
allow syslogd_t ping_exec_t:file execute;
Then, as root, execute the following commands:
checkmodule -M -m -o mycustom-librenms-rsyslog.mod mycustom-librenms-rsyslog.te
semodule_package -o mycustom-librenms-rsyslog.pp -m mycustom-librenms-rsyslog.mod
semodule -i mycustom-librenms-rsyslog.pp
rsyslog
If you prefer rsyslog, here are some hints on how to get it working.
Add the following to your rsyslog config somewhere (could be at the top of the file in the step below, could be in rsyslog.conf if you are using remote logs for something else on this host)
# Listen for syslog messages on UDP:514
$ModLoad imudp
$UDPServerRun 514
Create a file called /etc/rsyslog.d/30-librenms.confand add the following depending on your version of rsyslog.
# Feed syslog messages to librenms
module(load="omprog")
template(name="librenms"
type="string"
string= "%fromhost%||%syslogfacility%||%syslogpriority%||%syslogseverity%||%syslogtag%||%$year%-%$month%-%$day% %timegenerated:8:25%||%msg%||%programname%\n")
action(type="omprog"
binary="/opt/librenms/syslog.php"
template="librenms")
& stop
#Feed syslog messages to librenms
$ModLoad omprog
$template librenms,"%fromhost%||%syslogfacility%||%syslogpriority%||%syslogseverity%||%syslogtag%||%$year%-%$month%-%$day% %timegenerated:8:25%||%msg%||%programname%\n"
*.* action(type="omprog" binary="/opt/librenms/syslog.php" template="librenms")
& stop
# Feed syslog messages to librenms
$ModLoad omprog
$template librenms,"%FROMHOST%||%syslogfacility-text%||%syslogpriority-text%||%syslogseverity%||%syslogtag%||%$YEAR%-%$MONTH%-%$DAY% %timegenerated:8:25%||%msg%||%programname%\n"
$ActionOMProgBinary /opt/librenms/syslog.php
*.* :omprog:;librenms
If your rsyslog server is receiving messages relayed by another syslog server, you may try replacing %fromhost% with %hostname%, since fromhost is the host the message was received from, not the host that generated the message. The fromhost property is preferred as it avoids problems caused by devices sending incorrect hostnames in syslog messages.
Local Logstash
If you prefer logstash, and it is installed on the same server as LibreNMS, here are some hints on how to get it working.
First, install the output-exec plugin for logstash:
/usr/share/logstash/bin/logstash-plugin install logstash-output-exec
Next, create a logstash configuration file (ex. /etc/logstash/conf.d/logstash-simple.conf), and add the following:
input {
syslog {
port => 514
}
}
output {
exec {
command => "echo `echo %{host},,,,%{facility},,,,%{priority},,,,%{severity},,,,%{facility_label},,,,``date --date='%{timestamp}' '+%Y-%m-%d %H:%M:%S'``echo ',,,,%{message}'``echo ,,,,%{program} | sed 's/\x25\x7b\x70\x72\x6f\x67\x72\x61\x6d\x7d/%{facility_label}/'` | sed 's/,,,,/||/g' | /opt/librenms/syslog.php &"
}
elasticsearch {
hosts => ["10.10.10.10:9200"]
index => "syslog-%{+YYYY.MM.dd}"
}
}
Replace 10.10.10.10 with your primary elasticsearch server IP, and set the incoming syslog port. Alternatively, if you already have a logstash config file that works except for the LibreNMS export, take only the "exec" section from output and add it.
Remote Logstash (or any json source)
If you have a large logstash / elastic installation for collecting and filtering syslogs, you can simply pass the relevant logs as json to the LibreNMS API "syslog sink". This variant may be more flexible and secure in transport. It does not require any major changes to existing ELK setup. You can also pass simple json kv messages from any kind of application or script (example below) to this sink.
For long term or advanced aggregation searches you might still use Kibana/Grafana/Graylog etc. It is recommended to keep config['syslog_purge'] short.
A schematic setup can look like this:
┌──────┐
│Device├─►┌───────────────────┐ ┌──────────────┐
└──────┘ │Logstash Cluster ├┬──────────────►│ElasticSearch ├┐
│ RabbitMQ ││ │ Cluster ││
┌──────┬──►│ Filtering etc ││ ───────┐ └┬─────────────┼│
│Device│ └┬──────────────────┼│ │ └──────────────┘
└──────┘ └───────────────────┘ ▼
~~~WAN~~~
│
┌─┼─┐
│┼┼┼│ LB / Firewall / etc
└─┼─┘
│
▼
┌────────────────────┐ ┌────────────────────┐
│LibreNMS Sink ├┬──►│LibreNMS Master │
│/api/v0/syslogsink/ ││ │ MariaDB │
└┬───────────────────┼│ └────────────────────┘
└────────────────────┘
A minimal Logstash http output configuration can look like this:
output {
....
#feed it to LibreNMS
http {
http_method => "post"
url => "https://sink.librenms.org/api/v0/syslogsink/ # replace with your librenms host
format => "json_batch" # put multiple syslogs in on HTTP message
retry_failed => false # if true, logstash is blocking if the API is unavailable, be careful!
headers => ["X-Auth-Token","xxxxxxxLibreNMSApiToken]
# optional if your mapping is not already done before or does not match. "msg" and "host" is mandatory.
# you might also use out the clone {} function to duplicate your log stream and a dedicated log filtering/mapping etc.
# mapping => {
# "host"=> "%{host}"
# "program" => "%{program}"
# "facility" => "%{facility_label}"
# "priority" => "%{syslog5424_pri}"
# "level" => "%{facility_label}"
# "tag" => "%{topic}"
# "msg" => "%{message}"
# "timestamp" => "%{@timestamp}"
# }
}
}
Sample test data:
curl -L -X POST 'https://sink.librenms.org/api/v0/syslogsink/' -H 'X-Auth-Token: xxxxxxxLibreNMSApiToken' --data-raw '[
{
"msg": "kernel: minimum Message",
"host": "mydevice.fqdn.com"
},
{
"msg": "Line protocol on Interface GigabitEthernet1/0/41, changed state to up",
"facility": 23,
"priority": "189",
"program": "LINEPROTO-5-UPDOWN",
"host": "172.29.10.24",
"@timestamp": "2022-12-01T20:14:28.257Z",
"severity": 5,
"level": "ERROR"
},
{
"msg": "kernel: a unknown host",
"host": "unknown.fqdn.com"
}
]'
msg and host are the minimum keys. Graylog
This variant method use a external Graylog installation and its database. Please refer to the dedicated Graylog documentation.
Client configuration
Below are sample configurations for a variety of clients. You should understand the config before using it as you may want to make some slight changes. Further configuration hints may be found in the file Graylog.md.
Replace librenms.ip with IP or hostname of your LibreNMS install.
Replace any variables in
syslog
*.* @librenms.ip
rsyslog
*.* @librenms.ip:514
Cisco ASA
logging enable
logging timestamp
logging buffer-size 200000
logging buffered debugging
logging trap notifications
logging host <outside interface name> librenms.ip
Cisco IOS
logging trap debugging
logging facility local6
logging librenms.ip
Cisco NXOS
logging server librenms.ip 5 use-vrf default facility local6
Juniper Junos
set system syslog host librenms.ip authorization any
set system syslog host librenms.ip daemon any
set system syslog host librenms.ip kernel any
set system syslog host librenms.ip user any
set system syslog host librenms.ip change-log any
set system syslog host librenms.ip source-address <management ip>
set system syslog host librenms.ip exclude-hostname
set system syslog time-format
Huawei VRP
info-center loghost librenms.ip
info-center timestamp debugging short-date without-timezone // Optional
info-center timestamp log short-date // Optional
info-center timestamp trap short-date // Optional
//This is optional config, especially if the device is in public ip and you dont'want to get a lot of messages of ACL
info-center filter-id bymodule-alias VTY ACL_DENY
info-center filter-id bymodule-alias SSH SSH_FAIL
info-center filter-id bymodule-alias SNMP SNMP_FAIL
info-center filter-id bymodule-alias SNMP SNMP_IPLOCK
info-center filter-id bymodule-alias SNMP SNMP_IPUNLOCK
info-center filter-id bymodule-alias HTTP ACL_DENY
Huawei SmartAX (GPON OLT)
loghost add librenms.ip librenms
loghost activate name librenms
Allied Telesis Alliedware Plus
log date-format iso // Required so syslog-ng/LibreNMS can correctly interpret the log message formatting.
log host x.x.x.x
log host x.x.x.x level <errors> // Required. A log-level must be specified for syslog messages to send.
log host x.x.x.x level notices program imish // Useful for seeing all commands executed by users.
log host x.x.x.x level notices program imi // Required for Oxidized Syslog hook log message.
log host source <eth0>
HPE/Aruba Procurve
configure
logging severity warning
logging facility local6
logging librenms.ip control-descr “LibreNMS”
logging notify running-config-change
write memory
If you have permitted udp and tcp 514 through any firewall then that should be all you need. Logs should start appearing and displayed within the LibreNMS web UI.
Windows
By Default windows has no native way to send logs to a remote syslog server.
Using this how to you can download Datagram-Syslog Agent to send logs to a remote syslog server (LibreNMS).
Note
Keep in mind you can use any agent or program to send the logs. We are just using this Datagram-Syslog Agent for this example.
You will need to download and install "Datagram-Syslog Agent" for this how to Link to Download
External hooks
Trigger external scripts based on specific syslog patterns being matched with syslog hooks. Add the following to your LibreNMS config.php to enable hooks:
$config['enable_syslog_hooks'] = 1;
The below are some example hooks to call an external script in the event of a configuration change on Cisco ASA, IOS, NX-OS and IOS-XR devices. Add to your config.php file to enable.
Cisco ASA
$config['os']['asa']['syslog_hook'][] = Array('regex' => '/%ASA-(config-)?5-111005/', 'script' => '/opt/librenms/scripts/syslog-notify-oxidized.php');
Cisco IOS
$config['os']['ios']['syslog_hook'][] = Array('regex' => '/%SYS-(SW[0-9]+-)?5-CONFIG_I/', 'script' => '/opt/librenms/scripts/syslog-notify-oxidized.php');
Cisco NXOS
$config['os']['nxos']['syslog_hook'][] = Array('regex' => '/%VSHD-5-VSHD_SYSLOG_CONFIG_I/', 'script' => '/opt/librenms/scripts/syslog-notify-oxidized.php');
Cisco IOSXR
$config['os']['iosxr']['syslog_hook'][] = Array('regex' => '/%GBL-CONFIG-6-DB_COMMIT/', 'script' => '/opt/librenms/scripts/syslog-notify-oxidized.php');
Juniper Junos
$config['os']['junos']['syslog_hook'][] = Array('regex' => '/UI_COMMIT:/', 'script' => '/opt/librenms/scripts/syslog-notify-oxidized.php');
Juniper ScreenOS
$config['os']['screenos']['syslog_hook'][] = Array('regex' => '/System configuration saved/', 'script' => '/opt/librenms/scripts/syslog-notify-oxidized.php');
Allied Telesis Alliedware Plus
Note: At least software version 5.4.8-2.1 is required. log host x.x.x.x level notices program imi may also be required depending on configuration. This is to ensure the syslog hook log message gets sent to the syslog server.
$config['os']['awplus']['syslog_hook'][] = Array('regex' => '/IMI.+.Startup-config saved on/', 'script' => '/opt/librenms/scripts/syslog-notify-oxidized.php');
HPE/Aruba Procurve
$config['os']['procurve']['syslog_hook'][] = Array('regex' => '/Running Config Change/', 'script' => '/opt/librenms/scripts/syslog-notify-oxidized.php');
Configuration Options
Syslog Clean Up
Can be set inside of config.php
$config['syslog_purge'] = 30;
The cleanup is run by daily.sh and any entries over X days old are automatically purged. Values are in days. See here for more Clean Up Options Link
Matching syslogs to hosts with different names
In some cases, you may get logs that aren't being associated with the device in LibreNMS. For example, in LibreNMS the device is known as "ne-core-01", and that's how DNS resolves. However, the received syslogs are for "loopback.core-nw".
To fix this issue, you can configure LibreNMS to translate the incoming syslog hostname into another hostname, so that the logs get associated with the correct device.
Example:
$config['syslog_xlate'] = array(
'loopback0.core7k1.noc.net' => 'n7k1-core7k1',
'loopback0.core7k2.noc.net' => 'n7k2-core7k2'
);