When I first started installing "firewalls" in the 1980’s they were very much seen in the context of forest fires and "fire breaks". Firewalls provided a barrier across which the "fire" could not travel. We built "hardened" hosts, computers with all but the most essential software removed. The "hardened host" was was connected to the internet to provide the services which you wanted to make available and only those services. IP forwarding was turned of which meant that incoming network traffic could not be sent onto other destinations. Access from the LAN to the firewall was provided only for maintenance and in some configurations for users to access proxy servers which would connect to specified services on the internet on the users behalf..
As firewalls became ever more complex software applications there was quite a debate as to whether a firewall was necessary at all. If the only inbound ports open were the ones on which you provided services what was the point of blocking other trafic?
Well rightly or wrongly firewalls got more and more sophisticated and came to be regarded as an essential tool to protect the network. Other techniques, such as host hardening and perimeter networks became less of a priority. Without the firewall many organisation are now hopelessly exposed. The exposure of the enterprise to risk alsor exposes its clients, customers and partners to a similar risk.
Well configured and properly maintained, firewalls are of course highly effective in keeping the undesirables out. The best firewall in world however, is useless if it is turned off!
I cannot even recall the number of times I have been told that a network communications problem "must" be down to the firewall. I was reminded of this recently when a client was experiencing problems with email and a Courier POP3 mail server. It was a system that had been in place for some years. The system was built on a remote server in Germany and I had implemented an iptables firewall a couple of years previously. The firewall restricted access to ports 110 and 995 which were used to access their email to the clients site and that of the technical support staff. E-mail is crucial to the clients business and so, when problems occurred a consultant, from 1and1, the company providing the server, was brought in to accelerate the time to "fix". The consultant was quite clear and quite emphatic, the firewall was causing the problem!
It didn’t matter that the problem was intermittent, that the configuration of the firewall hadn’t changed, that the logs did not show any blocked POP3 traffic. It wasn’t relevant that a tool called "netstat" showed successful connections or that a sniffer (a tool that can capture network traffic) on the network showed the relevant communications taking place. No the problem must be the firewall. It was blocking traffic and must be turned off.
It reminded me of a similar situation when I was working for the putative "biggest bank in the world", in the City. I was managing 6 Checkpoint "Firewall 1" firewalls for the arm of the bank that dealt with exotic derivatives. The firewalls protected gateways to the internet, extra-net gateways to Frankfurt, Hong Kong, San Francisco, New York, Singapore and Tokyo, and data feeds from Bloomberg and Reuters. When one of the data feeds went down it was only minutes before the panic buttons started being pushed. The trading floor was in uproar, the bank could be loosing hundreds of thousands of dollars within minutes, it must be fixed instantly.
I was asked to check the firewall. I did. Nothing from the data feeds was being blocked, nothing had changed on the firewall that day. We could see from the firewall logs that it wasn’t a problem, sniffing on the network interface could show that no traffic was arriving from the data feed source but none of this held any sway with the network manager. It had to be the firewall and if it wasn’t, well the only way to prove it was to take the firewall down!
Well the long and short of it was that with tears of frustration in my eyes I ultimately carried out the instruction to "turn off" the firewall, it was that or clear my desk. And that was my priority you see, keeping my job.
The exposure of this multimillion pound enterprise and via it’s City of London hub, the world wide corporation, was less important than the priority of making profits on the trading floor. There was no time for rational debate the whole system, all around the world had to be exposed to the internet in order to "fix" the problem instantly.
Part of the problem I think is the demise of the systems administrator. The old systems administrator was responsible for everything, the servers, the network, telecoms, applications and system security. Because the systems admin was responsible for everything he had to understand it all and how each element fitted into the bigger picture. Now with so many functional divisions within a company’s IT provision no one in the organisation really has a handle on how it all works together.
The "networks" team primary ethos is to enable high speed communications. Firewalls get in the way of that process, firewalls stop network traffic! This could be why they are so often identified as the prime suspect when things stop working, they are something of an anathema to "networks". The other side of the coin of course is that the network security team, working independently of "networks" and "production services" tinker away at the firewall "improving security" often completely oblivious to he potential impact on the users or the business. They’ve only got to make a wrong call once or twice in a year, to become the bête noire for the next ten.
Oh and in case your wondering, turning the firewall off made no difference to the data feed at all, the problem, as I knew, lay elsewhere. I have to say the network manager was big enough to approach me later in the day and explain what the problem really turned out to be, it wasn’t an apology exactly but it was much more than might have been expected. Were the banks systems penetrated while the firewall was down? Nobody knows, and I suspect nobody involved, wanted to know it wasn't the priority. What was important was the traders access to the data feeds whatever the risks.