How to switch https traffic from 1 web server to another one. IPTABLES issue

Here is my configuration. I have varnish listening to port 80 and 2 web server (tomcat) on port 8080 and 8081 and https 4430 and 4431.

HTTP traffic goes from varnish to 1 web server. When I need to upgrade my app, I shut down one tomcat, upgrade the code, restart and change the varnish configuration to go to the upgraded tomcat.

I need to do the same for HTTP traffic. It's a small amount of traffic and I don't need caching, so I though I could just use iptables.

So I added a rule like this:

iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 443 -j REDIRECT --to-port 4430

and when I need to switch to the other tomcat, I do:

iptables -t nat -D PREROUTING -i eth0 -p tcp --dport 443 -j REDIRECT --to-port 4430 
iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 443 -j REDIRECT --to-port 4431 

It seemed to work but after I few days all the machine has connection problems. Varnish is not connecting to tomcat, dns lookup do not work, etc.

I found in /var/log/messages thousands of:

Mar 27 10:04:15 ip-10-72-254-31 kernel: [2112599.753509] nf_conntrack: table full, dropping packet.

Shutting down iptables seems to work to fix the connection problems.

So, can I use iptables in the way I'm using it? How can I prevent "table full, dropping packet" issue? What else can I use to redirect my https traffic?

Answers


We use iptables extensively and never faced NAT tables filling up; The only difference I see is that we are using slightly different rules using the DNAT target (even for localhost) rather than REDIRECT.

/sbin/iptables -I PREROUTING -t nat -i eth0 -p tcp --dport 443 -j DNAT --to :4430

(our kernels are 2.6.32 and higher - iptables 1.4.3.1 and higher)

EDIT

You can try making some adjustments to the nf_conntrack tacble (which tracks all opened connections).

The first one would be to reduce the time a connection can be tracked to a more sensible value, like one day. Something like:

sysctl -w net.netfilter.nf_conntrack_tcp_timeout_established=86400

You can also try to increase the size of the table

sysctl -w net.netfilter.nf_conntrack_max=65535

Restarting your network interfaces should be enough to use these new values.

Finally you will probably need to make those modifications to sysctl.conf to survive the next reboot.


To answer "What else can I use to redirect my https traffic?": You can use rinetd. Like iptables, it can be used to redirect TCP connections at port level, but it's way simpler to configure.

In essence, you would put this line to /etc/rinetd.conf:

10.72.254.31  443    10.72.254.31  4430

And exchange it for this when you want to switch over to the second web server:

10.72.254.31  443    10.72.254.31  4431

and after every such change, restart rinetd with:

service rinetd restart

Just make sure that nothing is listening on port 443 when enabling these, as else forwarding will fail and your /var/log/rinetd.log will be flooded with messages like 22/Mar/2013:19:35:47 0.0.0.0 0 (null) 0 0 0 accept-failed - [details].


Need Your Help

What is the proper package.json manner to publish a npm package developed with Browserify?

node.js npm browserify

I am publishing a npm package developed with Browserify, and wonder what is the proper manner to construct a package.json.

How use correctly Rcpp::pt( )

c++ r rcpp

I am reading the chapter 4 of "Seamless R and C++ Integration with Rcpp" and I had a little problem.