_Firewall-1 down for maintenance_

 Author: Christian ALT                 go to security alerts         go to ports definition

To reduce your down time when you upgrade your firewall or when you perform some maintenance tasks:

Use a spare station to upgrade FW-1.

Here are the details of our experience with spare machines for firewall-1. We have already made several software upgrades of FW-1 almost on the fly. The way we proceed is to prepare a spare machine with the definitions of the interfaces, the objects, the rules, the translation part if used, the databases, the routing definitions, DNS functions.

 Then we are able to remove the running fw-1 and we replace it by the spare. The traffic is then stopped for about 3-20 minutes depending on other network components, routers, DNS...There is a lot of dynamic involved on some sites we did, with heavy loaded lines. We could reduce the interruption time by having access to the network elements to about 5 minutes.

You must not think that to put back the original machine is an easy job. With the spare machine the dynamic of the network changed. You have again to wait for a while before the net knows about your machine.

 It means that with a spare machine you can upgrade or perform maintenance tasks with two interruptions of about 15 minutes (average).

 This allows you to upgrade your firewall offline and test it before having it back to production.

 We have now several machines ready for that kind of exercise and could switch firewalls with up to 5 interfaces. We want to share our experience, and let people know what they can expect with a spare.

Your spare machine does not have to be exactly the same as the original one, you can work with less services and restricted access. You can for example forbid the outgoing traffic from your internal users and just allow the access to your web server and to the incoming and outgoing mail.

What you have to be careful about

The name of the spare should be the same as the original.

If you have DNS server functions, copy the DNS databases, named.boot, resolv.conf files

If you run it under Solaris watch for /etc/nsswitch, /etc/defaultrouter, /etc/notrouter

Copy the objects, copy the users definitions

You can either copy the current policy to the spare or configure it with a different policy. The easiest being reducing the offered services, and building a more restrictive policy.

IP addresses and subnetmasks, hostname must match the original one.

The routing table must be the same

For address translation, do not forget to redefine the static entries in the arp cache with the new mac addresses of your interfaces.

Shutdown the spare and restart it. Watch for the loaded policy, the routing table, the translation table.

Take care not to have the spare running at the same time as the original firewall : you will get a mess with address conflicts.

Some special adaptation might be necessary for some sites.

Remove the cables from the original to the spare. From now on your spare is still unknown to most of the servers on the internal net and to the routers internal and external. This is due to the change of the mac addresses. Most of your computers will keep the old translation of the IP address of the firewall to the mac address.

To accelerat the knowledge of the new mac address on each required computer perform an :

arp -d IP#_firewall, and then ping the firewall from that computer.

 The DNS server might be unhappy with the request coming, look at the DNS resolution

A valuable tool is a network analyser (snoop for Solaris), activat it on each ineterface.

Look if the traffic is running smoothly, look for the correctness of the address translation.

Once the update process is done and you wish to switch back, be aware that the cache ARP must be re-initialised on the required computers.

The DNS should also be restarted.

.