This only happens when a network connection has been established, but then it takes, in my Windows 7 VM, (roughly) 15 seconds to shut down.
So I assume it is the winsock cleanup which takes a while. I guess it may be possible to background the process (or return to operation in non-graphical mode) while it keeps shutting down the socket so the user can continue with something else in the meantime.
Another approach my be to just ignore the socket state and have the operating system on both ends of the connection deal with closing the connection and liberating the socket (I would think it does this generally, based on a timeout) on application exit (only). This can seem programmatically dirty but I assume that's how many other applications (not) handle it, too.