Setting up Syslog Monitor with High-Availability

Overview 

A High-Availability (HA) active-passive VM deployment requires two identical VMs: a primary (master) and a secondary (backup) VM with DataSet Agent installed.

The secondary VM will act as a backup of the primary (master) VM and will continue to process events if the primary (master) VM fails.

Normal operation (Syslog monitor 1 is active as a Primary(master))

Syslog monitor is 2 active as a Primary(backup).

Failover from Syslog Monitor 1 to 2 is done using a virtual IP which will point to the active Syslog monitor in case one of the servers/machines goes down.

Prerequisites 

To deploy the HA configuration, you will need:

• Two VMs in the same network. These must be different VMs installed on different machines with different names, but both must have the same Syslog monitor configuration. 

• 3 IP addresses. One for each VM and a virtual IP associated with the active VM.

• The router (VRRP) software Keepalived. This software will monitor the activity of the VMs and trigger failover when needed.

Recommend VM configuration

2 vCPU

Memory → 8 GB

Ubuntu 

Step-by-step instructions (This guide is designed for setup on Ubuntu )

Identify the primary and secondary VMs
You are starting this process with two identical VMs already set up. Later steps in this process require you to distinguish between the primary VM and the secondary (failover) VM. So, to begin, be sure it is clear which is which.

Install the Keepalived package on both VMs

sudo apt-get update

sudo apt-get install keepalived

Configured Keepalived

Create a Keepalived config file on both VMs

Create a new "/etc/keepalived/keepalived.conf" file in each of the nodes copying the following content:

Configuration example for the primary node

vrrp_instance don_vsyslog { #edit this to set instance name
state MASTER
interface enp0s1
virtual_router_id 1
unicast_src_ip 192.168.64.3 #edit this
unicast_peer {
192.168.64.4 #edit this
}

priority 100
advert_int 1
authentication {
auth_type PASS #edit if desired
auth_pass this_is_the_password_for_ha1_ha2_comms #edit this
}

virtual_ipaddress {
192.168.64.5 brd 192.168.64.255 dev enp0s1 #edit this
}
}

Configuration example for the secondary node

vrrp_instance don_vsyslog { edit this to set instance name
state BACKUP
interface enp0s1
virtual_router_id 1
unicast_src_ip 192.168.64.4 #edit this
unicast_peer {
192.168.64.3 #edit this
}

priority 99
advert_int 1
authentication {
auth_type PASS #edit if desired
auth_pass this_is_the_password_for_ha1_ha2_comms #edit this
}

virtual_ipaddress {
192.168.64.5 brd 192.168.64.255 dev enp0s1 #edit this
}
}

Then edit the lines that include #edit this considering this information:

Agent Syslog configuration (identical on both VMs):

// Configuration for the Scalyr Agent. For help:
//
// https://www.scalyr.com/help/scalyr-agent-2

{
// Enter a "Write Logs" api key for your account. These are available at https://www.scalyr.com/keys
api_key: "<your write log API key>",

// Fields describing this server. These fields are attached to each log message, and
// can be used to filter data from a particular server or group of servers.
server_attributes: {
// Fill in this field if you'd like to override the server's hostname.
// serverHost: "REPLACE THIS",

// You can add whatever additional fields you'd like.
// tier: "production"
}

// Log files to upload to Scalyr. You can use '*' wildcards here.
logs: [
// { path: "/var/log/httpd/access.log", attributes: {parser: "accessLog"} }
],

monitors: [
{
module: "scalyr_agent.builtin_monitors.syslog_monitor",
protocols: "tcp:1468, udp:514",
accept_remote_connections: true
}

]
}

Restart Keepalived with the new configuration 

With the new keepalived.conf files and the scripts set up on both VMs, you need to restart the Keepalived service in each VM to activate the new configuration.

sudo systemctl restart keepalived

Enable the service so that it starts automatically if the machine is rebooted.

sudo systemctl enable keepalived

Confirm that both VMs have registered the virtual IP 

This simply confirms that the keepalived.conf file is correct and is being read properly.

Run this command on the primary VM:

ip a

You should see the primary VM IP and the virtual IP in the command output. If you don't, review the .conf file to be sure the changes were saved.

Send test data to the virtual IP
We can use the netcat utility to send 100 events to the virtual IP in order to confirm that it is enabled for the VMs.

for i in `seq 1 100`; do echo "<14>Jan 10 10:00:00 xxx test event $i"|nc -w0 192.168.64.5 1468; done

Testing High-Availability

On the primary node, stop the Keepalived service to simulate VM failure

sudo systemctl stop keepalived

On the secondary node, you can check that the secondary VM becomes the primary one:

sudo systemctl status keepalived

You will be able to see that your events are still being ingested into the platform.

From the Primary node enable keepalive to return HA to active-passive mode

sudo systemctl start keepalived