ELK – Auto-delete older Logstash indices

The following is an approach to auto-delete Logstash indices in Elasticsearch every X days. The following steps are to be run on your ELK host.

Get curator-cli

sudo pip install elasticsearch-curator -U

Create script

cd ~/
vim elasticsearch_del.sh

My preference is to delete indices older than 30 days, change the 30 to your preference. Then save the file.

#!/bin/bash                                                                                                                                           
/usr/local/bin/curator_cli "$@" delete_indices --filter_list '[{"filtertype":"age","source":"creation_date","direction":"older","unit":"days","unit_count":30},{"filtertype":"pattern","kind":"prefix","value":"logstash"}]'

Now make the script executable:
chmod +x elasticsearch_del.sh

Then run the script to make sure it works – use the --dry-run argument to test (i.e. not actually take any action):

./elasticsearch_del.sh --dry-run

If you you’re happy with the output and want to run it for real:

./elasticsearch_del.sh

Setup a CRON schedule job

crontab -e

Add the following line – changing the schedule to your preference. This runs it every Saturday at 5pm:

0 17 * * SAT /home/db/elasticsearch_del.sh

Advertisements

DNS CAA Record Adoption – Scanner and Results

The CAB forum members have voted and are in favor of making the CAA Checking Mandatory. All CAs (Certificate Authorities) will need to comply with the CAA (Certificate Authority Authorization) verification by September 2017.

The details are described in RFC 6844 – Abstract:

The Certification Authority Authorization (CAA) DNS Resource Record

allows a DNS domain name holder to specify one or more Certification

Authorities (CAs) authorized to issue certificates for that domain.

CAA Resource Records allow a public Certification Authority to

implement additional controls to reduce the risk of unintended

certificate mis-issue. This document defines the syntax of the CAA

record and rules for processing CAA records by certificate issuers.

This announcement made me curious about to what degree top HTTPS sites have opt’d into this by including CAA resource records for their properties…

I put together a simple scanner that queries the DNS records for each of the HTTPS ready Alexa Top 1 million sites. [Update] Code is now available on github

I did an initial scan totaling ~670k DNS records, resulting in ~0.05% having CAA resource records.

Given the low percentage of adoption, I am curious to observe how this changes over time. To that end, I’ve setup the scan to run periodically and post the results to @CAA_bot on twitter. Follow the account if you’re interested in being updated on progress.

ELK + Netflow

This guide assumes you’ve got a running ELK stack, and is tailored for a docker installation based on docker-elk. See my ELK on docker guide here

Also, the Netflow source configuration specifcs are for a Ubiquity EdgeRouter – you’ll need to get the specifics for your device if different.

Ubiquity EdgeRouter Config

configure
set system flow-accounting interface eth0
set system flow-accounting netflow version 9
set system flow-accounting netflow server <ip> port 2055
set system flow-accounting netflow enable-egress
commit
save

Logstash configure

cd ~/docker-elk

Open ./logstash/pipeline/logstash.conf in your editor

input {                                                                                                                                                                                                                                                                                               
  udp {                                                                                                                                               
    port => 2055                                                                                                                                      
    codec => netflow {                                                                                                                                
      versions => [5, 9]                                                                                                                              
    }                                                                                                                                                 
    type => netflow                                                                                                                                   
  }                                                                                                                                                   
}                                                                                                                                                                                                                                                                                                   

output {                                                                                                                                              

  }                                                                                                                                                   
  if [type] == "netflow" {                                                                                                                            
      elasticsearch {                                                                                                                                 
        index => "logstash_nf-%{+YYYY.MM.dd}"                                                                                                         
        hosts => "elasticsearch:9200"                                                                                                                 
      }                                                                                                                                               
    } else {                                                                                                                                          
        elasticsearch {                                                                                                                               
                hosts => "elasticsearch:9200"                                                                                                         
        }                                                                                                                                             
    }                                                                                                                                                 
}

Open docker-compose.yml in your editor, and add the following to the logstash service to ensure the NetFlow port 2055 is routed to the logstash container Continue reading “ELK + Netflow”

ELK on docker + Syslog

Versions used:

  • Ubuntu 16.04
  • Docker version 17.03.1-ce, build c6d412e
  • Docker-Compose version 1.12.0-rc2, build 08dc2a4

Install Docker CE

Install a few basics that we need:

sudo apt-get install \
    apt-transport-https \
    ca-certificates \
    curl \
    software-properties-common

Add docker.com’s GPG key

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

Add the apt-get repo:

sudo add-apt-repository \
   "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
   $(lsb_release -cs) \
   stable"

Update package list

sudo apt-get update

Install ‘docker-ce’ package

sudo apt-get install docker-ce

Test to verify all is well

sudo docker run hello-world                                                                                                            

Install Docker-Compose

Continue reading “ELK on docker + Syslog”

timhaak/plex docker upgrade

I’m using the timhaak/plex docker image

Here is to upgrade:

sudo docker pull timhaak/plex

sudo docker rm plex

Get it running again:
sudo docker run -d --restart=always -e PLEX_ALLOWED_NETWORKS=<CIDR> --name <shortname> -h <hostname> -v <config-location>:/config -v <media-location>:/data/movies -p 32400:32400 -p 32400:32400/udp -p 32469:32469 -p 32469:3
2469/udp -p 1900:1900/udp -p 32410:32410/udp -p 32412:32412/udp -p 32413:32413/udp -p 32414:32414/udp timhaak/plex

Replace:

<shortname> with what you want the container to be called

<hostname> with what you want the PMS to be called

<config-location> with the location of your Plex config (note to self, mine is: /opt/plex-data/)

<media-location> with the location of your videos/media

Verify upgrade

Show running containers:

sudo docker ps

Take note of the container ID for plex

Get a bash shell to the running Plex container:

sudo docker exec -it <containerid> /bin/bash

Verify the version installed

dpkg-query -s plexmediaserver | grep "Version"

ChromeOS + OpenVPN (+ TLSAuth)

This is a guide to get OpenVPN (with TLS Auth) working for a ChromeOS client. Note this guide assumes you to have control of the OpenVPN server and associated configuration. This guide doesn’t explain the specifics of port forwarding on your router, or use of Dynamic DNS – if you’re doing all the below I’ll assume you know about doing those things – if not there are plenty of tutorials around.

Versions used:

  • ChromeOS 57.0.2987.115 beta – on Samsung Chromebook Plus
  • Ubuntu 14.04 LTS (Bit old I know, but systemd 😦 )
  • OpenVPN 2.3.2 (openvpn 2.3.2-7ubuntu3.1)

Install OpenVPN server and easy-rsa

sudo apt-get install openvpn easy-rsa
sudo mkdir /etc/openvpn/easy-rsa/
sudo cp -r /usr/share/easy-rsa/* /etc/openvpn/easy-rsa/

Create certificates

cd /etc/openvpn/easy-rsa

Edit vars file to update the values

  • Set KEY_SIZE to 2048
  • Also set KEY_COUNTRY, KEY_PROVINCE, KEY_CITY, KEY_ORG, KEY_EMAIL parameters. Don’t leave any of these parameters blank.

Run ./vars to load parameters
Run ./clean-all to clear keys and previous files

Now lets create our CA cert and key:

Run ./build-ca. The majority of the defaults will be loaded of the var specified values, but you must enter the Common Name (CN) – enter a name that identifies your CA. MyVPN-CA for example. This will create two files 1) ca.crt your CA cert (public) and 2) ca.key you CA private key (secret!)

Now to create the server cert and key:

Run ./build-key-server server. Like the previous command most values can be defaulted. When prompted for CN, enter server. Then select yes for both Sign Certificate and Commit. This will create two files 1) server.crt your servers cert (public) and 2) server.key your servers private key (secret!)

Time for the client(s) cert and key(s):

Run ./build-key-client client1. When prompted for CN, enter a name unique for each client – e.g. client1. Then select yes for both Sign Certificate and Commit. This will create two files 1) client1.crt your clients cert (public) and 2) client1.key your clients private key (secret!)

Now we need to put the client cert and key into a format understood by ChromeOS, namely pkcs12. Run openssl pkcs12 -export -in client1.crt -inkey client1.key -certfile ca.crt -name MyClient -out client1.p12. Enter an export passphrase. This will create a file called client1.p12.

You can repeat the above each client, and just increment the client number: client2, client3 etc…

Now to generate the Diffie Hellman parameters. Run ./build-dh – this may take a few to many minutes. This will create a file called dh2048.pem – this is not secret.

Continue reading “ChromeOS + OpenVPN (+ TLSAuth)”