Saturday, 18 April 2015

Multicast IPsec protected OSPFv2/OSPFv3 areas


1. Short introduction.

The presumption that your backbone area is the safest segment of your corporate network is untenable, especially when your backbone is based on MAN service. Usually the MAN operators are used to provide to the clients virtual networks (logical layer of Layer 2 frames transport) based on 802.1q VLAN or VPLS. It is a shared service because the clients' networks are logically defined transport of frames across the operator's physical network. Therefore any successful attack on the operator's active equipment creates is direct risk of compromising the clients' networks.

OSPF is still the most used protocol for intra-AS routing. Usually the routers connected to the corporate backbone networks are configured as OSPF speakers (they speak either OSPFv2 and OSPFv3, or both). The OSPFv2 protocol provides both simple and cryptographic authentication, and the cryprographic ones is based on the MD5 hash function. In contrast the authetication in OSPFv3 is moved out of the protocol. One very serious reason the authentication is moved is because the OSPF applications are critical, they are part of the user space, and process multicast messages. The operating system kernel escalates the multicast messages received for the OSPF group address to the OSPF application regardless they are properly autheticated or not. Therefore if an intruder gain an access to an OSPF area it may send a lot of packets to the OSPF listeners and rise the load of their protocol processing applications. Given that the OSPF algorithm strongly depends on the ability of processing as fast as possible the incomming update messages, any intentionally caused high load may affect the dynamic routing over the whole OSPF area.

One possible way to prevent the unautheticated OSPF messages from reaching the OSPF application and affect its performance is to employ the kernel to perform the packet authentication. In other words the kernel should filter the unauthenticated OSPF messages. That can be achieved by implementing an IPsec transport protocol so the kernel should escalate to the OSPF application only the packets passed a successful ESP/AH authentication. Such an idea sounds very reasonable but it should solve somehow an importent principle problem - IKE cannot exchange (yet) keys over multicast. Hence if IPsec is going to be implemented to secure OSPF a static keys must be configured on all OSPF listeners in the area.

Below it is decribed how to use IPsec to protect and authenticate the communication between OSPF speakers running quagga daemons ospfd and ospf6d.


2. Generating the keys.

Two keys for each direction of the communication should be generated - one for the encryption algorithm and one for the message digest algorithm. Bellow are presented command lines that can be used to generated that keys:

  • 128-bits key (AES, SHA1):

    $ echo -n "0x" && dd if=/dev/urandom count=16 bs=1 2> /dev/null | xxd -ps -c 16

  • 160-bits key (RIPEMD):

    $ echo -n "0x" && dd if=/dev/urandom count=20 bs=1 2> /dev/null | xxd -ps -c 20

  • 256-bits key (AES, SHA256):

    $ echo -n "0x" && dd if=/dev/urandom count=32 bs=1 2> /dev/null | xxd -ps -c 32

  • 448-bits key (Blowfish):

    $ echo -n "0x" && dd if=/dev/urandom count=56 bs=1 2> /dev/null | xxd -ps -c 56

  • 512-bits key (AES, SHA512):

    $ echo -n "0x" && dd if=/dev/urandom count=64 bs=1 2> /dev/null | xxd -ps -c 64


3. Loading the keys and configuring the IPsec policy.

The most convenient way of configuring IPsec for multicast with pre-shared keys on Linux is to use the package ipsec-tools. It provides the IKEv2 daemon racoon and the policy manager setkey. Since only pre-shared keys will be configured and used there is no need to run the racoon. Only setkey should be run once when both keys and policies need to be loaded into the memory. On the hard disk they should be kept in the file setkey.conf, by following the example bellow:


# OSPFv2

spdadd[0][0] any -P in ipsec esp/transport//require ;
spdadd[0][0] any -P out ipsec esp/transport//require ;
spdadd[0][0] any -P in ipsec esp/transport//require ;
spdadd[0][0] any -P out ipsec esp/transport//require ;
add esp 0x10003 -m transport -E blowfish-cbc 0xd7e13bc95b0d7a40b4591e06a22bdd40c7dd0632b1cff938b9bc947d03a6dc14091e69de309b3c9d6627ee871317a39cd85d65402b674e2e -A hmac-sha256 0x2ff9702ace1e5986135074bb4be183537f85d255a250dfa46d01edfa625038ca ;
add esp 0x10005 -m transport -E blowfish-cbc 0xd5c69b9088ec8054724cdd84e5fb82ca39f5ff5979e6f33ebf453d568320c7df19a39c771397143cd54d55b858c5ca27e17fb01105da183f -A hmac-sha256 0x46693831b5c989b186fd7ba884d8bcea503a9ea4fc835958c3166051604eb3ab ;

# OSPFv3
spdadd ::/0 ff02::5 any -P in ipsec esp/transport//require ;
spdadd ::/0 ff02::5 any -P out ipsec esp/transport//require ;
spdadd ::/0 ff02::6 any -P in ipsec esp/transport//require ;
spdadd ::/0 ff02::6 any -P out ipsec esp/transport//require ;
add :: ff02::5 esp 0x10003 -m transport -E blowfish-cbc 0x8f46177b518ad6f43e519ac1bc63657bef66390e013bad95b835536e3eb7f509a867f203a21a70ddd32f92d8fd1a7dd71c1fc5f083c44f9e -A hmac-sha256 0x70249c37454edf0a5696a4f9f487604cf6a9ef06580a18bd2baa380b35b622ec ;
add :: ff02::6 esp 0x10005 -m transport -E blowfish-cbc 0xd555e5ec47dcc407c3650efdda92210e24de57ce96a0305c251881f606adcb7d38b7d03d5e06fac45cbdbf5572fc948d733957852931c2f9 -A hmac-sha256 0x1e8ec58b12aa6a5ac51f80af34ffc6082f2cc8f1efaac49f480e9e413cc0f424 ;

In RHEL and CentOS the file setkey.conf is usually located in the directory /etc/racoon, if the package ipsec-tools is installed from RPM packages. To load the keys and the policy setkey should be run like this:

# setkey -f setkey.conf

To check if they are successfully loaded into the memory and configured in the kernel, run:

# ip xfrm state

It is good idea to create a service (RHEL/CentOS <=6) or systemd configuration (RHEL/CentOS >=7) for loading the policy at a boot time or reload it later. Note that it is vital to load the policy simultaneous on all OSPF speakers to prevent OSPF area splitting.


4. Debugging.

If ospf6d is properly started and the IPsec policy is correctly loaded, the router should join the OSPF area and start exchange routing information. The received routes can be listed by executing:

# vtysh -c "sh ipv6 ospf6 route"

An example output is provided bellow:

*N E1 2001:67c:20d0:20::/64 fe80::16da:e9ff:fef1:195a eth0 33d03:07:59
*N E1 2001:67c:20d0:22::/64 fe80::201:3ff:fed8:8b39 eth0 33d03:09:44
*N E1 2001:67c:20d0:23::/64 fe80::210:5aff:fef2:8484 eth0 33d03:09:44
*N E1 2001:67c:20d0:24::/64 fe80::16da:e9ff:fef1:195a eth0 33d03:07:59
*N E1 2001:67c:20d0:25::/64 fe80::16da:e9ff:fef1:195a eth0 33d03:07:59
*N IA 2001:67c:20d0:2f::ffff:0/112   :: eth0 33d03:09:44

If no routes appear for a long time in the list there might be a problem with the IPsec configuration. So check the content of setkey.conf, correct and reload the policies, and restart the ospf6d daemon.

To check if there is an IPsec communication on ff02::5 tcpdump should be run in order to capture and analyze the packets arriving on the ethernet interface the router is connected to the OSPF area (eth0 for instance):

# tcpdump -n -i eth0 host ff02::5

and if there is IPsec multicast communication exchange it will be diplayed as:

17:45:41.770145 IP6 fe80::20e:cff:fe84:945b > ff02::5: ESP(spi=0x00010003,seq=0x6df6a), length 84
17:45:41.770498 IP6 fe80::16da:e9ff:fef1:195a > ff02::5: ESP(spi=0x00010003,seq=0x702e0), length 84
17:45:41.770647 IP6 fe80::210:5aff:fef2:8484 > ff02::5: ESP(spi=0x00010003,seq=0x5fafa), length 84
17:45:41.771146 IP6 fe80::201:3ff:fed8:8b39 > ff02::5: ESP(spi=0x00010003,seq=0x75ca0), length 84
17:45:51.770765 IP6 fe80::20e:cff:fe84:945b > ff02::5: ESP(spi=0x00010003,seq=0x6df6b), length 84
17:45:51.771100 IP6 fe80::16da:e9ff:fef1:195a > ff02::5: ESP(spi=0x00010003,seq=0x702e1), length 84
17:45:51.771297 IP6 fe80::210:5aff:fef2:8484 > ff02::5: ESP(spi=0x00010003,seq=0x5fafb), length 84
17:45:51.771749 IP6 fe80::201:3ff:fed8:8b39 > ff02::5: ESP(spi=0x00010003,seq=0x75ca1), length 84

Note that the source of the packets is always the link-local address of the router interface attached to the OSPF area. So by checking the output of tcpdump to estimate which router participate in the IPsec multicast communication.

IPv6 Deployment in Sofia University Network (2007-2013)

IPv6 Deployment in Sofia University Network (2007-2013)

The slides were presented at the Third South East Europe (SEE 3)/RIPE NCC Regional Meeting (14-15 April 2014, Sofia, Bulgaria). You can download the presentation on PDF here:

Monday, 6 April 2015

Link-state BGP prefix announcements with quagga

Link-state BGP prefix announcements with quagga

The border routers should be excluded from the process of route origination in BGP unless they are configured to do a route aggregation. In all other cases they should only redistribute as BGP prefixes the routes created (originated) by the access routers inside the autonomous system. Most often the access routers run OSPF software and therefore the routes they originate are send to the borders through OSPF areas. It is a common practice to originate routes for the connected to the router networks based on the link-state of the interfaces. In that case if the interface of interest is up and if an IP address is configured on it, the network the configured IP address belongs to may become originated (if the configuration of the access router allows that). This post is intended to show how to do such kind of orignation by using quagga and BGP (not OSPF).

Lets describe the schema employed for the origination of connected routes. When an IP address is configured on a network interface (and the link state of that interface is "up") the operating system kernel marks the network the IP belongs to, as a connected (directly accessible) and creates the correponsing route in the routing table. That route represents the network as directly accessed through some interface name (without next-hop IP address). Routes directed by an network interface instead of next-hop IP address are connected routes. Accroding to both linux kernel and quagga specifications thay appear as supplied to the routing table by the routing protocol "connected". Because each connected route is related ony to a network interface it will be part of the kernel routing table as long as the link state of the interface is "up". When the interface is going down the kernel should delete from the routing table all routes (not only the connected ones) connected through it. The kernel should do the same if the IP address used for connecting the network is delted from the interface configuration.

The configuration bellow gives an idea how to configure bgpd to redistribute the connected routes into iBGP as prefixes. Both IPv4 and IPv6 route origination are described.

hostname bgpd
password zebra
log file /var/log/quagga/bgpd.log
router bgp 5421
 bgp router-id
 redistribute connected route-map REDISTRIBUTE_CONNECTED_IPV4
 neighbor BORDER_ROUTERS_V4 peer-group
 neighbor BORDER_ROUTERS_V4 remote-as 5421
 neighbor BORDER_ROUTERS_V4 next-hop-self
 neighbor BORDER_ROUTERS_V4 soft-reconfiguration inbound
 neighbor BORDER_ROUTERS_V6 peer-group
 neighbor BORDER_ROUTERS_V6 remote-as 5421
 neighbor peer-group BORDER_ROUTERS_V4
 neighbor peer-group BORDER_ROUTERS_V4
 address-family ipv6
 redistribute connected route-map REDISTRIBUTE_CONNECTED_IPV6
 neighbor BORDER_ROUTERS_V6 activate
 neighbor BORDER_ROUTERS_V6 next-hop-self
 neighbor 2001:67c:20d0:0:62:44:127:17 peer-group BORDER_ROUTERS_V6
 neighbor 2001:67c:20d0:0:62:44:127:18 peer-group BORDER_ROUTERS_V6
ip prefix list REDISTRIBUTE_IPV4 seq 5 permit
ipv6 prefix list REDISTRIBUTE_IPV6 permit 2001:67c:20d0:20::/64
route-map OUTGOING_IPV6_PREFIXES permit 20
 match ipv6 address prefix-list REDISTRIBUTE_CONNECTED_IPV6
 set community 5421:19
route-map INCOMMING_IPV4_PREFIXES accept 10
route-map INCOMMING_IPV6_PREFIXES accept 10
 match ip address prefix list REDISTRIBUTE_IPV4
 set origin igp
 match ipv6 address prefix list REDISTRIBUTE_IPV6
 set origin igp
line vty

How it works? When quagga is installed and configured as a routing protocol its zebra daemon takes in account the content on the kernel routing table and the changes happening there. For instancee if the IP addresses and 2001:67c:20d0:29::/64 are configured on the interface enp3s0 the networks and 2001:67c:20d0:20::/64 will appear as connected routes in the routing kernel table. That information becomes available to zebra (through NETLINK) and it forwards it to the other routing protocol daemons (such as ospfd, ospf6d, ripd, ripngd, bgpd, isisd). When bgpd receives updates from zebra it classify them by protocol (the one the kernel set as an atribute to the respective routes). Then it checks if the configuration (as the one given above) requires route redistribution. Because "redistribute connected" appears as a part of the bgp router configuration, those routes that matching the protocol "connected" will be processed. But only those of them that can successfully pass through the prefix lists (REDISTRIBUTE_IPV4 or REDISTRIBUTE_IPV6) will appear as BGP routes and will be eventyally announced to the border routers.

Note that the good practise in using quagga instists that the IP addresses should be configured on the respective interfaces by using the zebra configuration. For instance:

zebra@lcpe-gw(config)# interface enp3s0
zebra@lcpe-gw(config-if)# ipv6 address
zebra@lcpe-gw(config-if)# ipv6 address 2001:67c:20d0:29::/64

Sunday, 8 March 2015

How to install CentOS 7 on HP ProLiant BL460c G1/G5

1. Introduction

HP ProLiant BL460c G1/G5 is a very stable and reliable server platform widely employed by the dedicated servers providers. It has built-in HP Smart Array Controller that can be used under Linux by means of HP Smart Array CCISS driver. It provides the kernel modules cciss.ko. Unfortunately Red Hat Inc. discontinued the support for HP Smart Array CCISS driver and since then Red Hat Enterprise Linux 7 (RHEL 7) and CentOS 7 does not include cciss.ko in their kernel packages. Therefore the distribution installer Anaconda cannot recognize the SAS controller during the installation and cannot use the disks attached to it.

One possible way to overcome that problem is to adopt the kmod-cciss RPM package provided by the ELRepo repository. That package provides the module cciss.ko which is missing in the kernel modules tree, but it is not very trivial to place it into the initrd image and rebuild the installer disk. An easier and safer way is to use the original installer (without modifications), and instruct Anaconda to read the module from so called "driver disk", during the pre-installation process. The instructions given bellow cover only that kind of installation. Their execution requires a physical access to the server or access to iLO with virtual media support. The BIOS of the machine must allow the use of USB floopy and USB DVD as boot devices.

If you don't have a physical access to the server or it is a dedicated server owned by some hosting operator, it is mandatory to have an access to iLO. Some of the hosting companies you can find on the Internet does not give you an access to iLO. It is better to avoid such companies. Note that iLO has an important role in the administration of HP server systems!

2. Building the driver disk

To build a driver disk containing the cciss.ko module from the package kmod-cciss the following routine must be executed (on a system running CentOS or RHEL):

$ export CURR=`pwd`
$ export SQUASH_ROOT=${CURR}/squashfs-root
$ export VER=3.6.26-1
$ export RPM_PKG=kmod-cciss-${VER}.el7.elrepo.x86_64.rpm
$ mkdir -p ${SQUASH_ROOT}/rpms/x86_64/
$ echo "HP Smart Array CCISS driver ver. 3.6.26" > ${SQUASH_ROOT}/rhdd3
$ wget --directory-prefix=${SQUASH_ROOT}/rpms/x86_64/${RPM_PKG}
$ createrepo --basedir ${SQUASH_ROOT}/rpms/x86_64/ .
$ touch ${SQUASH_ROOT}/.rundepmod
$ rpm2cpio ${SQUASH_ROOT}/rpms/x86_64/${RPM_PKG} | cpio -imVd ./lib/*
$ cd ${CURR}
$ mksquashfs ${SQUASH_ROOT} ${CURR}/kmod-cciss-${VER}-driver-disk.img
$ rm -fr ${SQUASH_ROOT}

Thus created file kmod-cciss-3.6.26-1-driver-disk.img contains the driver disk image. To list its content use the command line:

$ unsquashfs -ls kmod-cciss-3.6.26-1-driver-disk.img

3. Preparing the installation media

The installation process requires to have locally as files both driver disk and the latest CentOS 7 installer.

3.1. Connecting the virtual media applet provided by iLO (default installation media)

If iLO Virtual Media is accessible both installer and driver disk files should be selected and connected as virtual devices in the window of the applet:

To attach the driver disk as a virtual floppy go to "Virtual Floppy/USBkey" section of the applet window. Click there on "Local Image File" radio button, press "Browse" and then use the emerged file manager window to select the driver disk file. Finally, click on "Connect".

Almost the same sequence of operations need to be performed in "Virtual CD/DVD-ROM" section of the applet window in order to attach the CemtOS 7 installer iso file as virtual DVD.

3.2. Connecting physical media devices (requires physical access to the server)

It is not a common practice to have the server supplied or equipped with floppy and CD/DVD device. So if the virtual media provided by iLO is not accessible at the time USB Key and USB DVD device need to be pugged into the machine USB ports. Also the BIOS settings need to be adjusted to make possible booting from a USB DVD device.

The driver disk need to be transferred to a USB Key by using dd tool:

$ dd if=kmod-cciss-3.6.26-1-driver-disk.img of=/dev/sdb

(if the USB Key device is presented into the local system as /dev/sdb) and the CentOS 7 installer should be burned to a DVD blank.

4. Starting the installation

Reboot the machine. In the first screen shown by the installer:

press Tab. Then supply the installer with loading parameters:

Lets explain them:


This is the URL of the repository containing the CentOS 7 package collection. It is better to use a repository instead of transfering the packages from the local install media (the iso file) through iLO.


It tells Anaconda to use driver disk. The driver disk need to be attached as USB Floppy.


It tells Anaconda to redirect the video ouput to a VNC session. That redirection takes plase when the installer switches to the interactive mode (where the user can chose the parameters of the installation: language, the disk partitioning, packages, e.t.c.).


The password for accessing the VNC session.


The video resolution for the VNC session.

ip= netmask= gateway= nameserver=

The IP address, netmask, gateway, and nameserver. Anaconda needs them for accessing the respository (see "inst.repo" above) and to provide a network access to the VNC session.

After pressing "Enter" the Anacoda is starting the installation process and soon it will ask for selecting the driver disk:

The driver disk (the USB Floppy) has a type "squashfs". To mark it for a further use do enter "2" and press "Enter". Then Anaconda will display the list of the drivers provided by the disk:

Enter the driver number (in the illustration the number is 1 since only one driver is provided) and then press "Enter".

Now the driver is selected and the should be loaded. Enter "c" and press "Enter" for loading the driver. The process of loading will be displayed:

Enter "c" and press "Enter" to continue. After that Anaconda will connect to the repository and download the installer components. When finish it will start the VNC session and display a message:

Finally a VNC viewer should be started and connected to the address and session number displayed before by Anaconda.

After pressing the "Connect" button a box for entering the VNC password will appear. If the password is entered correctly the VNC viewer will access the graphical installer window of Anaconda:

Now the installation could be run. After finishing the installation do not forget to disconnect the virtual of physical media before restarting the server.