Re: [Nolug] DNS Server Problem with TCP

From: Dustin Puryear <dustin_at_puryear-it.com>
Date: Sun, 26 Oct 2008 20:21:33 -0500
Message-ID: <4905179D.5020501@puryear-it.com>

With Linux, the drivers are either built into the kernel or supplied as
modules. In both situations a kernel update will update the driver.

Be sure to track 'netstat -np' before and AFTER the problem. We want to
see if the number of TCP connections gets too high. Your system can only
support so many (as large a number as it may be), and I wonder if
something is grabbing all those sockets.

It could also still be a driver issue, but TCP is a tad high up the
stack. But who knows.

--
Dustin Puryear
President and Sr. Consultant
Puryear Information Technology, LLC
225-706-8414 x112
http://www.puryear-it.com
Author, "Best Practices for Managing Linux and UNIX Servers"
  http://www.puryear-it.com/pubs/linux-unix-best-practices/
John Souvestre wrote:
> Hi Dustin.
> 
> I forced a kernel update this morning.  So this would have upgraded (if
> necessary) the driver?
> 
> The failure mode seems to be just TCP.  UDP keeps working.  The DNS server runs
> - for the most part.  It just won't do large domains or zone transfers.  I can
> still ping it with ICMP but not TCP.
> 
> Here's some results before the failure.  Next time it fails I'll be checking
> again.
> 
> - - -
> 
> "netstat -tupan | grep :53\ "
> 
> tcp        0      0 199.254.148.41:53           
> ... 0.0.0.0:*                   LISTEN      2714/named
> tcp        0      0 127.0.0.1:53                
> ... 0.0.0.0:*                   LISTEN      2714/named
> udp        0      0 199.254.148.41:53           
> ... 0.0.0.0:*        2714/named
> udp        0      0 127.0.0.1:53                
> ... 0.0.0.0:*        2714/named
> 
> - - -
> 
> "lsof -i:53"
> 
> COMMAND  PID  USER   FD   TYPE DEVICE SIZE NODE NAME
> named   2714 named   20u  IPv4   5751       UDP localhost.localdomain:domain
> named   2714 named   21u  IPv4   5752       TCP localhost.localdomain:domain
> (LISTEN)
> named   2714 named   22u  IPv4   5756       UDP ns1.idsno.net:domain
> named   2714 named   23u  IPv4   5757       TCP ns1.idsno.net:domain (LISTEN)
> 
> - - -
> 
> John
> 
>    John Souvestre - Southern Star & Integrated Data Systems - (504) 355-0609
> 
>  > -----Original Message-----
>  > From: owner-nolug@stoney.redfishnetworks.com [mailto:owner-
>  > nolug@stoney.redfishnetworks.com] On Behalf Of Dustin Puryear
>  > Sent: Saturday, October 25, 2008 3:00 PM
>  > To: nolug@nolug.org
>  > Subject: Re: [Nolug] DNS Server Problem with TCP
>  > 
>  > up2date, by default, will skip kernel updates. kernel updates includes
>  > module updates, which means no driver updates. :)
>  > 
>  > I would suspect the card or the driver since DNS (UDP) and ping (TCP) fail.
>  > 
>  > The next time this happens you can do:
>  > 
>  > # netstat -lnp
>  > 
>  > And then:
>  > 
>  > # netstat -np
>  > 
>  > You may have a bazillion connections OR you may have a bad card/driver.
>  > 
>  > --
>  > Dustin Puryear
>  > President and Sr. Consultant
>  > Puryear Information Technology, LLC
>  > 225-706-8414 x112
>  > http://www.puryear-it.com
>  > 
>  > Author, "Best Practices for Managing Linux and UNIX Servers"
>  >   http://www.puryear-it.com/pubs/linux-unix-best-practices/
>  > 
>  > 
>  > John Souvestre wrote:
>  > > Hi Dustin.
>  > >
>  > > No, the hardware is different.  Ns1 is a Dell and ns2 is a Compaq.  They
>  > are
>  > > both set up about the same.  They are just used as name servers.
>  > >
>  > > I like your driver theory.  It would explain why restart named doesn't
>  > help.
>  > > Would RH's up2date keep the driver up to date or is that something has to
>  > be
>  > > done some other way?
>  > >
>  > > Thanks,
>  > >
>  > > John
>  > >
>  > >    John Souvestre - Southern Star & Integrated Data Systems - (504) 355-
>  > 0609
>  > >
>  > >  > -----Original Message-----
>  > >  > From: owner-nolug@stoney.redfishnetworks.com [mailto:owner-
>  > >  > nolug@stoney.redfishnetworks.com] On Behalf Of Dustin Puryear
>  > >  > Sent: Saturday, October 25, 2008 2:06 PM
>  > >  > To: nolug@nolug.org
>  > >  > Subject: Re: [Nolug] DNS Server Problem with TCP
>  > >  >
>  > >  > Oh, wait. Did you say you are using identical hardware? The same NIC? If
>  > >  > not, could it be a driver or hardware issue with the card and it flakes
>  > out?
>  > >  >
>  > >  > Also, the reason I was asking about running network services is that DNS
>  > >  > may be a red herring (as you implied with the 'ping' comment).
>  > >  >
>  > >  > --
>  > >  > Dustin Puryear
>  > >  > President and Sr. Consultant
>  > >  > Puryear Information Technology, LLC
>  > >  > 225-706-8414 x112
>  > >  > http://www.puryear-it.com
>  > >  >
>  > >  > Author, "Best Practices for Managing Linux and UNIX Servers"
>  > >  >   http://www.puryear-it.com/pubs/linux-unix-best-practices/
>  > >  >
>  > >  >
>  > >  > John Souvestre wrote:
>  > >  > > Hi Dustin.
>  > >  > >
>  > >  > > I should add that restarting named doesn't help.
>  > >  > >
>  > >  > > John
>  > >  > >
>  > >  > >    John Souvestre - Southern Star & Integrated Data Systems - (504)
>  > 355-
>  > >  > 0609
>  > >  > >
>  > >  > >  > -----Original Message-----
>  > >  > >  > From: John Souvestre [mailto:johns@sstar.com]
>  > >  > >  > Sent: Saturday, October 25, 2008 11:26 AM
>  > >  > >  > To: 'nolug@nolug.org'
>  > >  > >  > Subject: RE: [Nolug] DNS Server Problem with TCP
>  > >  > >  >
>  > >  > >  > Hi Dustin.
>  > >  > >  >
>  > >  > >  > redhat-release-4ES-8.0.el4_7.1 is the version on both ns1 and ns2.
>  > >  > >  >
>  > >  > >  > bind-9.2.4-30.el4 on both.  But I don't think that bind is part of
>  > the
>  > >  > >  > problem since TCP pings to the box fail also when the problem
>  > starts.
>  > >  > >  >
>  > >  > >  > Thanks,
>  > >  > >  >
>  > >  > >  > John
>  > >  > >  >
>  > >  > >  >    John Souvestre - Southern Star & Integrated Data Systems - (504)
>  > 355-
>  > >  > 0609
>  > >  > >  >
>  > >  > >  >  > -----Original Message-----
>  > >  > >  >  > From: owner-nolug@stoney.redfishnetworks.com [mailto:owner-
>  > >  > >  >  > nolug@stoney.redfishnetworks.com] On Behalf Of Dustin Puryear
>  > >  > >  >  > Sent: Saturday, October 25, 2008 10:38 AM
>  > >  > >  >  > To: nolug@nolug.org
>  > >  > >  >  > Subject: Re: [Nolug] DNS Server Problem with TCP
>  > >  > >  >  >
>  > >  > >  >  > The exact same version of RH? And what is the kernel version on
>  > both?
>  > >  > >  >  > And the version of bind on both?
>  > >  > >  >  >
>  > >  > >  >  > --
>  > >  > >  >  > Dustin Puryear
>  > >  > >  >  > President and Sr. Consultant
>  > >  > >  >  > Puryear Information Technology, LLC
>  > >  > >  >  > 225-706-8414 x112
>  > >  > >  >  > http://www.puryear-it.com
>  > >  > >  >  >
>  > >  > >  >  > Author, "Best Practices for Managing Linux and UNIX Servers"
>  > >  > >  >  >   http://www.puryear-it.com/pubs/linux-unix-best-practices/
>  > >  > >  >  >
>  > >  > >  >  >
>  > >  > >  >  > John Souvestre wrote:
>  > >  > >  >  > > Hi all.
>  > >  > >  >  > >
>  > >  > >  >  > > I'm having a strange problem with one of our DNS servers
>  > >  > >  > (ns1.idsno.net).
>  > >  > >  >  > Our
>  > >  > >  >  > > ns2.idsno.net server is set up almost identically and it has
>  > no
>  > >  > problem.
>  > >  > >  >  > >
>  > >  > >  >  > > The problem is with the ability to do DNS TCP transfers.  What
>  > >  > makes
>  > >  > >  > this
>  > >  > >  >  > really
>  > >  > >  >  > > strange is that TCP works just fine when the box is rebooted.
>  > But
>  > >  > a few
>  > >  > >  >  > days
>  > >  > >  >  > > later, it stops responding to TCP.  I've seen this happen a
>  > half
>  > >  > dozen
>  > >  > >  >  > times.
>  > >  > >  >  > >
>  > >  > >  >  > > We're running RH.  As far as I can tell, IPTables is not being
>  > used
>  > >  > on
>  > >  > >  > the
>  > >  > >  >  > box.
>  > >  > >  >  > >
>  > >  > >  >  > > I don't see anything interesting in /var/log/messages.
>  > >  > >  >  > >
>  > >  > >  >  > > The box is behind a PIX but so is the other one.  Port 53 UDP
>  > and
>  > >  > TCP
>  > >  > >  > are
>  > >  > >  >  > both
>  > >  > >  >  > > enabled.
>  > >  > >  >  > >
>  > >  > >  >  > > Does anyone have any ideas?
>  > >  > >  >  > >
>  > >  > >  >  > > Thanks!
>  > >  > >  >  > >
>  > >  > >  >  > > John
>  > >  > >  >  > >
>  > >  > >  >  > >    John Souvestre - Southern Star & Integrated Data Systems -
>  > (504)
>  > >  > 355-
>  > >  > >  >  > 0609
>  > >  > >  >  > >
>  > >  > >  >  > >
>  > >  > >  >  > > ___________________
>  > >  > >  >  > > Nolug mailing list
>  > >  > >  >  > > nolug@nolug.org
>  > >  > >  >  > >
>  > >  > >  >  > > --
>  > >  > >  >  > > This message was scanned by ESVA and is believed to be clean.
>  > >  > >  >  > > Click here to report this message as spam.
>  > >  > >  >  > > http://esva.puryear-it.com/cgi-bin/learn-msg.cgi?id=
>  > >  > >  >  > >
>  > >  > >  >  > >
>  > >  > >  >  > ___________________
>  > >  > >  >  > Nolug mailing list
>  > >  > >  >  > nolug@nolug.org
>  > >  > >
>  > >  > > ___________________
>  > >  > > Nolug mailing list
>  > >  > > nolug@nolug.org
>  > >  > >
>  > >  > > --
>  > >  > > This message was scanned by ESVA and is believed to be clean.
>  > >  > > Click here to report this message as spam.
>  > >  > > http://esva.puryear-it.com/cgi-bin/learn-msg.cgi?id=
>  > >  > >
>  > >  > >
>  > >  > ___________________
>  > >  > Nolug mailing list
>  > >  > nolug@nolug.org
>  > >
>  > > ___________________
>  > > Nolug mailing list
>  > > nolug@nolug.org
>  > >
>  > > --
>  > > This message was scanned by ESVA and is believed to be clean.
>  > > Click here to report this message as spam.
>  > > http://esva.puryear-it.com/cgi-bin/learn-msg.cgi?id=
>  > >
>  > >
>  > ___________________
>  > Nolug mailing list
>  > nolug@nolug.org
> 
> ___________________
> Nolug mailing list
> nolug@nolug.org
> 
> --
> This message was scanned by ESVA and is believed to be clean.
> Click here to report this message as spam. 
> http://esva.puryear-it.com/cgi-bin/learn-msg.cgi?id=
> 
> 
___________________
Nolug mailing list
nolug@nolug.org
Received on 10/26/08

This archive was generated by hypermail 2.2.0 : 12/19/08 EST