Home

Documents

Autoban

This is a short guide for protecting internet facing Linux servers from attack. The guide assumes Red Hat 7, but should mostly work with other distrubutions. There are three components to the solution.

Docker, KVM & iptables

Setting up a server with KVM and docker. If you're using a bridge interface to have your VMs talk to your network, you might notice that after a docker installation, your VMs have suddenly no connection. So what is going on? It turns out, docker adds a bunch of iptables rules by default which prevent communication. These will interfere with an already existing bridge, and suddenly your VMs will report no network. To prevent this you can disable iptables in /etc/docker/daemon.json by including the following entry. 
{
	"iptables" : false
}

Firewalld

I failed to find a clear explanation in the documentation, but from the implemented behaviour it looks like that:

  1. Interfaces and sources are used to select which zones are active
  2. Both are ignored for the default zone (always active)

So I believe the most effective use of firewalld is:

  1. lock down the default zone, say "public" - no open ports or services available
  2. in another zone, say "work" - define source and open ports

For example, assuming default zone is public and has no open ports, add source and port range to "work" zone:

    # firewall-cmd --zone=work --add-source=192.168.0.0.24
    # firewall-cmd --zone=work --add-port=8080-8090/tcp

Now check the active zones (default zone is always active):

    # firewall-cmd --get-active-zones

You will get:

     work
	    sources: 192.168.0.0/24

So "work" zone rules will apply to the particular subnet. Of courxe use --permanent option to make the behaviour stick.

In turn any ports or services you have in "public" (default) zone will apply to all interfaces and source addresses.

    # firewall-cmd --list-all-zones

public (default)
interfaces:
sources:
services:
port:
masquerade: no
forward-ports:
icmp-blocks:
rich rules:

work (active) 
interfaces:
sources: 192.168.0.0/24
services: dhcpv6-client ipp-client ssh
port: 8080-8090/tcp
masquerade: no
forward-ports:
icmp-blocks:
rich rules:

The same system works for interfaces. Say by adding "ens3" to "work" zone:

    # firewall-cmd --zone=work --add-interface=ens3

For KVM servers make the following change to allow the free flow of traffic to virtual clients.

	# firewall-cmd --permanent --remove-interface br0 --zone=public
	# firewall-cmd --permanent --add-interface br0 --zone=trusted
	# firewall-cmd --reload

FreeIPA PKI

The integrated PKI Service is provided via the Dogtag project. PKI signs and publishes certificates for FreeIPA hosts and services. It also provides CRL and OCSP services for all software validating the published certificate. FreeIPA management framework provides API to request, show and find certificates.

As the certificates used by FreeIPA client hosts and services have limited validity, the infrastructure also needs to handle reliable renewal of the certificates. For that purpose, a Certmonger daemon is running on all clients and handles the renewal in a transparent way for the services using it.

Blending in PKI infrastructure

FreeIPA server PKI can be configured in several configurations to fit into potentially existing PKI infrastructure

Communication with PKI

FreeIPA clients and their services are neither expected nor allowed to communicate with PKI directly. They are supposed to utilize the FreeIPA server API instead, using the standard Kerberos authentication. FreeIPA web service then validates the request and passes it to the PKI service, authenticating with an own agent certificate (ipaCert stored in /etc/httpd/alias/)

Requesting a new certificate

Certificate can be requested either manually by a privileged user who is then able to request it for any chosen hostname (cn) or by the host itself, which can request a certificate for it's own hostname, ideally via Certmonger.

Manual certificate requests
On a FreeIPA client, run the following commands to request a new certificate which can then be used by a mod_nss Apache module to secure a HTTPS traffic with a certificate published by FreeIPA CA:

  1. Create a Kerberos principal for the service that will use/own the certificate:
      # ipa service-add HTTP/`hostname`
  2. Create NSS certificate database which will hold the certificate 
      # mkdir -p /etc/httpd/alias; cd /etc/httpd/alias
  # certutil -N -d .
  3. Set correct directory ownership and SELinux context (on platforms running on SELinux):
      # chown :apache *.db && chmod g+rw *.db
  # semanage fcontext -a -t cert_t "/etc/httpd/alias(/.*)?"
  # restorecon -FvvR /etc/httpd/alias/
  4. Add FreIPA CA certificate
      # certutil -A -d . -n 'EXAMPLE.COM IPA CA' -t CT,, -a < /etc/ipa/ca.crt
  5. Request a signed certificate for the service
      # certutil -R -d . -a -g 2048 -s CN=`hostname`,O=EXAMPLE.COM > web.csr
  # ipa cert-request --principal=HTTP/`hostname` web.csr
  # ipa cert-show $SERIAL_NUMBER --out=web.crt
  # certutil -A -d . -n Server-Cert -t u,u,u -i web.crt
  6. Optionally show and validate the certificate
    
  # certutil -L -d . -n Server-Cert
  # certutil -V -u V -d . -n Server-Cert

Obviously, this procedure has a disadvantage of the certificate not being tracked by the Certmonger and thus not being automatically renewed before it's validity ends.

Automated certificate requests with Certmonger To request a new (HTTP) certificate for a FreeIPA client, the procedure is slightly easier. The biggest benefit is that the certificate is automatically renewed before the validation ends:

  1. Create a Kerberos principal for the service that will use/own the certificate:
      # ipa service-add HTTP/`hostname`
  2. Create NSS certificate database which will hold the certificate
      # mkdir -p /etc/httpd/alias; cd /etc/httpd/alias
  # certutil -N -d .
  3. In case you created the database with a PIN (asked interactively in the previous step), remember it or store it to text file:
      # echo $PIN > pwdfile.txt
  # chmod o-rwx pwdfile.txt
  4. Set correct directory ownership and SELinux context (on platforms running on SELinux):
      # chown :apache *.db && chmod g+rw *.db
  # semanage fcontext -a -t cert_t "/etc/httpd/alias(/.*)?"
  # restorecon -FvvR /etc/httpd/alias/
  5. Add FreeIPA CA certificate
      # certutil -A -d . -n 'EXAMPLE.COM IPA CA' -t CT,, -a < /etc/ipa/ca.crt
  6. Request a signed certificate for the service and see the entry in Certmonger. In case you created a NSS database with a PIN (see the step 3.), use -P $PIN or -p /etc/httpd/alias/pwdfile.txt option to tell certmonger about it:
      # ipa-getcert request -d /etc/httpd/alias -n Server-Cert -K HTTP/`hostname`
        -N CN=`hostname`,O=EXAMPLE.COM -g2048 -p /etc/httpd/alias/pwdfile.txt

    • SAN names: in FreeIPA 4.0 and later, you can add optional SAN DNS names to your request with -D. Note that you need to first create respective host or service objects and configure that given host can manage them with service-add-host or host-add-managedby command. These objects are being verified when FreeIPA cert-req command authorizes the SAN names.

  7. Check the status of the requested certificate. If request succeeded, it will be in a MONITORING state:
      # ipa-getcert list -d /etc/httpd/alias/ -n Server-Cert
  8. Optionally show and validate the certificate
      # certutil -L -d . -n Server-Cert
  # certutil -V -u V -d . -n Server-Cert

Documentation

GPG Sign RPM

# How to sign your custom RPM package with GPG key

# Step: 1
# Generate gpg key pair (public key and private key)
#
# You will be prompted with a series of questions about encryption.
# Simply select the default values presented. You will also be asked
# to create a Real Name, Email Address and Comment (comment optional).
# 
# If you get the following response:
# -----------------------------------------------------------------------
# We need to generate a lot of random bytes. It is a good idea to perform
# some other action (type on the keyboard, move the mouse, utilize the
# disks) during the prime generation; this gives the random number
# generator a better chance to gain enough entropy.
# -----------------------------------------------------------------------
# Open up a separate terminal, ssh into your server and run this command:
# ls -R /

gpg --gen-key

# Step: 2
# Verify your gpg keys were created

gpg --list-keys

# Step: 3
# Export your public key from your key ring to a text file.
#
# You will use the information for Real Name and Email you used to
# create your key. I used Fernando Aleman and faleman@email.com

gpg --export -a 'Fernando Aleman' > RPM-GPG-KEY-faleman

# Step: 4
# Import your public key to your RPM DB
#
# If you plan to share your custom built RPM packages with others, make sure
# to have your public key file available online so others can verify RPMs

sudo rpm --import RPM-GPG-KEY-faleman

# Step: 5
# Verify the list of gpg public keys in RPM DB

rpm -q gpg-pubkey --qf '%{name}-%{version}-%{release} --> %{summary}\n'

# Step: 6
# Configure your ~/.rpmmacros file
#
# You can use the following command to edit if you are on the server:
# vi ~/.rpmmacros
#
# %_signature => This will always be gpg
# %_gpg_path  => Enter full path to .gnupg in your home directory
# %_gpg_name  => Use the Real Name you used to create your key
# %_gpbin     => run `which gpg` (without ` marks) to get full path 

%_signature gpg
%_gpg_path /root/.gnupg
%_gpg_name Fernando Aleman
%_gpgbin /usr/bin/gpg

# Step: 7
# Sign your custom RPM package
#
# You can sign each RPM file individually:

rpm --addsign git-1.7.7.3-1.el6.x86_64.rpm

# Or you can `cd` into your RPMS folder and sign them all:

rpm --addsign *.rpm

# Step: 8
# Check the signature to make sure it was signed
#
# Watch for 'gpg OK' as in this example:
# git-1.7.7.3-1.el6.x86_64.rpm: (sha1) dsa sha1 md5 gpg OK

rpm --checksig git-1.7.7.3-1.el6.x86_64.rpm

# Tip!
# Sign package during build
#
# To sign a package while it's being built, simply add '--sign'

rpmbuild -ba --sign git.spec

LetsEncrypt

Certs from Let's Encrypt have a maximum validity duration of 90 days. There are many plugins available for Certbot which can be used to automate the certificate update process. Unfortunately, my experience with these plugins has not been favorable. At the time of this writting, a plugin for Apache servers that use mod_nss is not available. Also, the available plugins to not work well with Apache security modules such as authnz_pam, mod_security and mod_evasive. I therefore decided, at least for now, to manually update certificates using the following process. The instructions apply to RedHat/CentOS 7 & 8 systems.

Apache mod_md method

mod_md - Let's Encrypt for Apache

Certbot standalone method with http challenge

Install certbot from the EPEL repository, if not already installed.

  1. Change to the Letsencrypt directory
    cd /etc/letsencrypt/live/<DOMAIN>
  2. Shutdown httpd service
    systemctl stop httpd.service
  3. To get updated LetsEencrypt certs run the following command.
    certbot certonly --standalone --preferred-challenges http \
	-d <DOMAIN1> -d <DOMAIN2>
    The certificate and chain will be saved at:
    /etc/letsencrypt/live/<DOMAIN>/
  4. Make pkcs12 file
    openssl pkcs12 -export -in cert.pem --inkey privkey.pem \
	-certfile chain.pem -password file: <PWFILE> -out cert.pfx \
	-name "Server-Cert"
  5. Remove existing cert and key
    certutil -F -d /etc/httpd/alias -n Server-Cert
  6. Import pkcs12 file
    pk12util -d /etc/httpd/alias/ -w <PWFILE> -i cert.pfx
  7. Start Apache
    systemctl start httpd.service

Certbot manual method with dns challenge

If you’d like to obtain a certificate running certbot on a machine other than your target webserver or perform the steps for domain validation yourself, you can use the manual plugin. While hidden from the UI, you can use the plugin to obtain a certificate by specifying certonly and --manual on the command line. This requires you to copy and paste commands into another terminal session, which may be on a different computer.

When using the dns challenge, certbot will ask you to place a TXT DNS record with specific contents under the domain name consisting of the hostname for which you want a certificate issued, prepended by _acme-challenge.

  1. Request cert from letsencrypt 
    sudo certbot certonly --manual --preferred-challenge dns -d *.ohares.us
  2. Create dns TXT record
    If your DNS provider happens to be NO-IP, creating the TXT record is a manual process. Log into no-ip.com, then navigate to "manage dns" and modify your domainname record. Paste the challenge code returned in step #1 into the data field.
Additionally you can specify scripts to prepare for validation and perform the authentication procedure and/or clean up after it by using the --manual-auth-hook and --manual-cleanup-hook flags. This is described in more depth in the hooks section.

NO-IP

Instruction for recieving or renewing a GeoTrust certificate from No-IP.com.

Note, the No-IP website does not include the ability to "renew" a certificate. Instead you wait until your existing cert is about to expire, and then purchase a new one. No-IPs' maximum certificate term is two years.

If you don't already have a private key, create a private key and signing request as follows:

openssl req -out www.ohares.us.csr -new -newkey rsa:2048 -nodes \
	-keyout www.ohares.us.key

If you already have a private key, generate a certificate signing request as follows:

openssl req -new -out www.ohares.us.csr -key www.ohares.us.key \
          -config www.ohares.us.cnf -sha256

To the get the CSR signed,go to https://my.noip.com/, and follow their instructions.

After you download your new certificate, it must be converted to NSSDB format for import

openssl pkcs12 -export -in www.ohares.us.crt -inkey www.ohares.us.key \
        -certfile GeoTrust.crt -password file:pw.txt -out cert.pfx \
		-name 'Server-Cert'

Remove existing cert and key

certutil -F -d /etc/httpd/alias -n Server-Cert

Import new pkcs12 file

pk12util -d /etc/httpd/alias -w pw.txt -i cert.pfx

Restart httpd service

systemctl start httpd.service

RHEL Developer Subscription

Cannot Activate RHEL Server with Free Developer Subscription

I signed up for the free Red Hat developer license, but I can't activate my RHEL server install. My account says I have an active Red Hat Enterprise Linux Developer Suite subscription, but when I go to attach my system to it, I get a message saying I don't have any matching subscriptions.

It happens randomly every now and then ... You should be able to solve it by executing :

sudo subscription-manager remove --all
sudo subscription-manager unregister
sudo subscription-manager clean

sudo subscription-manager register
sudo subscription-manager refresh
sudo subscription-manager attach --auto

Replace software RAID device

This guide shows how to remove a failed hard drive from a Linux RAID1 array (software RAID), and how to add a new hard disk to the RAID1 array without losing data.

  1. Preliminary Note

    2. In this example I have two hard drives, /dev/sda and /dev/sdb, with the partitions /dev/sda1 and /dev/sda2 as well as /dev/sdb1 and /dev/sdb2.

    /dev/sda1 and /dev/sdb1 make up the RAID1 array /dev/md0.

    /dev/sda2 and /dev/sdb2 make up the RAID1 array /dev/md1.

    /dev/sdb has failed, and we want to replace it.

  2. How Do I Tell If A Hard Disk Has Failed?

    3. If a disk has failed, you will probably find a lot of error messages in the log files, e.g. /var/log/messages or /var/log/syslog.

    You can also run

    	cat /proc/mdstat

    and instead of the string [UU] you will see [U_] if you have a degraded RAID1 array.

  3. Removing The Failed Disk

    4. To remove /dev/sdb, we will mark /dev/sdb1 and /dev/sdb2 as failed and remove them from their respective RAID arrays (/dev/md0 and /dev/md1).

    First we mark /dev/sdb1 as failed:

    
	server1:~# mdadm --manage /dev/md0 --fail /dev/sdb1

    Result:

    
	server1:~# cat /proc/mdstat
	Personalities : [linear] [multipath] [raid0] [raid1] [raid5] [raid4] [raid6] [raid10]
	md0 : active raid1 sda1[0] sdb1[2](F)
		24418688 blocks [2/1] [U_]
	  
	md1 : active raid1 sda2[0] sdb2[1]
		24418688 blocks [2/2] [UU]
 
	unused devices:
    Then we remove /dev/sdb1 from /dev/md0: 
    
	server1:~# mdadm --manage /dev/md0 --remove /dev/sdb1
	mdadm: hot removed /dev/sdb1
    And "cat /proc/mdstat" should show this: 
    
	server1:~# cat /proc/mdstat
	Personalities : [linear] [multipath] [raid0] [raid1] [raid5] [raid4] [raid6] [raid10]
	md0 : active raid1 sda1[0]
		24418688 blocks [2/1] [U_]
 
	md1 : active raid1 sda2[0] sdb2[1]
		24418688 blocks [2/2] [UU]
 
	unused devices:
    Now we do the same steps again for /dev/sdb2 (which is part of /dev/md1): server1:~# mdadm --manage /dev/md1 --fail /dev/sdb2

    Result:

    
	server1:~# cat /proc/mdstat
	Personalities : [linear] [multipath] [raid0] [raid1] [raid5] [raid4] [raid6] [raid10]
	md0 : active raid1 sda1[0]
		24418688 blocks [2/1] [U_]
	 
	md1 : active raid1 sda2[0] sdb2[2](F)
		24418688 blocks [2/1] [U_]
	 
	unused devices:
    Repeat for /dev/sdb2 
    
	server1:~# mdadm --manage /dev/md1 --remove /dev/sdb2
	mdadm: hot removed /dev/sdb2
    Result:> 
    
server1:~# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid5] [raid4] [raid6] [raid10]
md0 : active raid1 sda1[0]
	24418688 blocks [2/1] [U_]
 
md1 : active raid1 sda2[0]
	24418688 blocks [2/1] [U_]

	unused devices:

    Then power down the system and replace the old /dev/sdb hard drive with a new one
    (it must have at least the same size as the old one - if it's only a few MB smaller than the old one then rebuilding the arrays will fail).

  4. Adding The New Hard Disk
    5. After you have changed the hard disk /dev/sdb, boot the system. The first thing we must do now is to create the exact same partitioning as on /dev/sda. We can do this with one simple command: 
    	sfdisk -d /dev/sda | sfdisk /dev/sdb
    You can run 
    	fdisk -l
    to check if both hard drives have the same partitioning now. Next we add /dev/sdb1 to /dev/md0 and /dev/sdb2 to /dev/md1: mdadm --manage /dev/md0 --add /dev/sdb1 
    
	server1:~# mdadm --manage /dev/md0 --add /dev/sdb1
	mdadm: re-added /dev/sdb1

	server1:~# mdadm --manage /dev/md1 --add /dev/sdb2
	mdadm: re-added /dev/sdb2
    Now both arays (/dev/md0 and /dev/md1) will be synchronized. Run 
    cat /proc/mdstat
    to see when it's finished. During the synchronization the output will look like this: 
    
server1:~# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid5] [raid4] [raid6] [raid10]
	md0 : active raid1 sda1[0] sdb1[1]
		24418688 blocks [2/1] [U_]
	[=>...................]  recovery =  9.9% (2423168/24418688) finish=2.8min speed=127535K/sec
																		 
	md1 : active raid1 sda2[0] sdb2[1]
		24418688 blocks [2/1] [U_]
	[=>...................]  recovery =  6.4% (1572096/24418688) finish=1.9min speed=196512K/sec
																	  
	unused devices:
    When the synchronization is finished, the output will look like this: 
    
server1:~# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid5] [raid4] [raid6] [raid10]
	md0 : active raid1 sda1[0] sdb1[1]
		24418688 blocks [2/2] [UU]

	md1 : active raid1 sda2[0] sdb2[1]
		24418688 blocks [2/2] [UU]

	unused devices:
    That's it, you have successfully replaced /dev/sdb!