net: one dead descriptor must not cost every peer its turn

From the first of two wedges captured on the pool. It is a different
proximate fault from the second one, and this is the part that turns a
single bad socket into a node-wide outage.

A failed select() zeroes both descriptor sets, so no peer is serviced on
that pass. That is correct as far as it goes. What was missing is that
nothing stopped the offending descriptor coming straight back:

  - the invalid socket was only marked fDisconnect, and a node is not
    removed from vNodes until it has drained or DISCONNECT_DRAIN_SECS has
    elapsed -- and it cannot drain, because its socket is gone;
  - the select set was rebuilt from vNodes with no check, so the same dead
    descriptor went back in on the very next pass and failed select()
    again.

Measured in the capture: 1,027 consecutive 'its socket is no longer valid'
passes across 61 seconds -- the full drain timeout -- one bad socket, and
for the whole of it the node read nothing from anybody. The peer that
caused it had 57 bytes unsent, so it took the slow path out.

Now the descriptor is forgotten as soon as getsockopt says it is not ours:
select() succeeds again on the next pass and every other peer is served
normally while the dead node ages out as before.

Deliberately forgotten rather than closed. getsockopt has just said this
descriptor is no longer ours; calling close() on it is precisely how an
unrelated socket gets closed instead, which is the fault fixed in e742d0b.

(cherry picked from commit f14c513b4e39d6203b73584f6dc01acbb5a9546e)

0 files changed, 0 insertions, 0 deletions