We provide our software as SAAS offering and use ftp for handling documents. All of a sudden, we started having ftp trouble at one of our data center.
Then it spread to other data centers also.
When ever we had a issue with ftp, it would take nearly 30 second for the login prompt to come up. From the get go we suspected a dns issue but we weren't sure which functionality of the ftp server was causing it.
We turned off the reverse dns look up, then turned off logging and still it didn't help. we got a memory dump during the one of the outage and saw some calls for dns lookup but couldn't make much head way with out symbol files.
We switched to another ftp server product and the problem disappeared.
Finally the ftp server support came through and asked us to turn off additional flags that were causing dns look up. After we turned it off, the problem disappeared.
We had turned on SMTP on the ftp server for monitoring unauthenticated login and it was causing reverse DNS look up issue and even after turning off this feature, a bug in the ftp server that would cause dns reverse lookup even though all the reverse dns look up features were turned off.