How to switch https traffic from 1 web server to another one. IPTABLES issue
Here is my configuration. I have varnish listening to port 80 and 2 web server (tomcat) on port 8080 and 8081 and https 4430 and 4431.
HTTP traffic goes from varnish to 1 web server. When I need to upgrade my app, I shut down one tomcat, upgrade the code, restart and change the varnish configuration to go to the upgraded tomcat.
I need to do the same for HTTP traffic. It's a small amount of traffic and I don't need caching, so I though I could just use iptables.
So I added a rule like this:
iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 443 -j REDIRECT --to-port 4430
and when I need to switch to the other tomcat, I do:
iptables -t nat -D PREROUTING -i eth0 -p tcp --dport 443 -j REDIRECT --to-port 4430 iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 443 -j REDIRECT --to-port 4431
It seemed to work but after I few days all the machine has connection problems. Varnish is not connecting to tomcat, dns lookup do not work, etc.
I found in /var/log/messages thousands of:
Mar 27 10:04:15 ip-10-72-254-31 kernel: [2112599.753509] nf_conntrack: table full, dropping packet.
Shutting down iptables seems to work to fix the connection problems.
So, can I use iptables in the way I'm using it? How can I prevent "table full, dropping packet" issue? What else can I use to redirect my https traffic?
We use iptables extensively and never faced NAT tables filling up; The only difference I see is that we are using slightly different rules using the DNAT target (even for localhost) rather than REDIRECT.
/sbin/iptables -I PREROUTING -t nat -i eth0 -p tcp --dport 443 -j DNAT --to :4430
(our kernels are 2.6.32 and higher - iptables 220.127.116.11 and higher)
You can try making some adjustments to the nf_conntrack tacble (which tracks all opened connections).
The first one would be to reduce the time a connection can be tracked to a more sensible value, like one day. Something like:
sysctl -w net.netfilter.nf_conntrack_tcp_timeout_established=86400
You can also try to increase the size of the table
sysctl -w net.netfilter.nf_conntrack_max=65535
Restarting your network interfaces should be enough to use these new values.
Finally you will probably need to make those modifications to sysctl.conf to survive the next reboot.
To answer "What else can I use to redirect my https traffic?": You can use rinetd. Like iptables, it can be used to redirect TCP connections at port level, but it's way simpler to configure.
In essence, you would put this line to /etc/rinetd.conf:
10.72.254.31 443 10.72.254.31 4430
And exchange it for this when you want to switch over to the second web server:
10.72.254.31 443 10.72.254.31 4431
and after every such change, restart rinetd with:
service rinetd restart
Just make sure that nothing is listening on port 443 when enabling these, as else forwarding will fail and your /var/log/rinetd.log will be flooded with messages like 22/Mar/2013:19:35:47 0.0.0.0 0 (null) 0 0 0 accept-failed - [details].