Failure Notification Latencies
The latency of failure notification for an actual failure is comprised of two parts: the time for a node in the system to decide that a failure has occurred, and the time for FUSE to propagate that information to all group mem- bers. The time to detect a failure depends on the type of failure, and on how node and link failures are monitored. We perform two experiments to characterize these costs: our first experiment investigates the latency of explicitly signaled failures, and our second investigates the latency of failure notification when nodes crash.
To measure the notification latency of explicitly sig- naled failures, we chose a group member at random from the same set of groups used for the group creation ex- periments, and had it explicitly signal failure. As expected, the notification latencies are sig- nificantly lower than the group creation latencies. This improvement is a result of three factors. First, our mes- saging layer maintains a cache of recently used TCP con- nections rather than opening a new TCP connection each time a message is sent. During this experiment, the failure notifications travel over cached TCP connections because these connections were recently used to perform group creation. Second, failure notifications only require a one- way message, not a round-trip. Third, creation blocks at the root until all members have been contacted, and thus a single node in the group that is far away in the network will delay the entire create operation. In contrast, notifica- tion takes effect at each member as soon as the notification has arrived. The maximum notification time observed for any FUSE group was 1165 ms.
