Fix server-side EndOfStreamException and CancelledKeyExceptions warnings

[csfrancis]

This fixes the server warnings generated in zookeeper.log:

2013-11-19 01:57:53,625 [myid:] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@349] - caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x14252332fe501c9, likely client has closed socket
        at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
        at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
        at java.lang.Thread.run(Thread.java:724)
2013-11-19 01:57:53,625 [myid:] - INFO  [ProcessThread(sid:0 cport:-1)::PrepRequestProcessor@476] - Processed session termination for sessionid: 0x14252332fe501c9
2013-11-19 01:57:53,626 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:58253 which had sessionid 0x
14252332fe501c9
2013-11-19 01:57:53,627 [myid:] - ERROR [SyncThread:0:NIOServerCnxn@180] - Unexpected Exception:
java.nio.channels.CancelledKeyException
        at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
        at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)
        at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:153)
        at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1076)
        at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:404)
        at org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:167)
        at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:101)

These warnings are caused by the Zookeeper C API client disconnecting before a response can be sent back from the server. The patch adds a function, wait_for_close that is called from zookeeper_close, which will block until a response is received for the close request, thereby allowing the connection to be gracefully closed.

[eric]

Is this something you've had a chance to submit to https://issues.apache.org/jira/browse/ZOOKEEPER?

I'm concerned about the poll() in the patch — it is the responsibility of the caller of the C library to be polling the file descriptors, it isn't something that the internals should do themselves.

Doing it this way could cause the Ruby VM to hang for up to 3 seconds while that poll() waits to complete, which isn't a great result...

[csfrancis]

I haven't yet submitted it to the Zookeeper JIRA. I can understand your concern about the poll() call. Part of my reasoning for doing it this way is that the close is already a blocking operation:

        /* make sure the close request is sent; we set timeout to an arbitrary
         * (but reasonable) number of milliseconds since we want the call to block*/
        rc=adaptor_send_queue(zh, 3000);

The call to adaptor_send_queue(), which in the ST adaptor case calls flush_send_queue(), will also potentially block for up to 3 seconds. Perhaps I could set the timeout value to be a bit lower?