Proximity Optimised Scheduling

Multi-tier Holistic Resilience

As the size and complexity of the HPC cluster increases, the mean time between a failure also increases. Checkpointing is the current solution, with limited innovation specifically to reduce the likelihood and impact of a failure. The EuroEXA precursor project, ExaNEST specifically considered the efficiency of checkpointing by leveraging the contained abstraction of a node through virtualization. EuroEXA has focused on reducing the performance impact of checkpointing while investigating solutions to reduce to occurrence and severity of any failure. This includes techniques to resolve failures locally, while communicating across the stack as necessary.

Infrastructure Resilience

The infrastructure includes the mechanics, the power, and the cooling of the IT of a HPC deployment. Along with the performance advances in EuroEXA in exposing locality, it creates a hierarchy against which failures can be managed. No longer are all node equal to all other nodes in a system, we created a hierarchy both physically in the hardware and in the design of the firmware/software: Nodes are independently powered and network together in units of four; Four such units create a blade with its independent hardware management and localised networking. Eight blades create a network group and creates the next level of network isolation. Four network groups create the cabinet which provides redundancy of power supply and the unit of system replication. The liquid cooling system is designed with redundant paths and pumps with an innovative solution to prevent leaks and a subsequent catastrophic failures by running at sub-atmospheric pressures, ensuring leaks pull in air, that can then be vented, and moving leaks to be simply a scheduled maintenance issue.

Topology Resilience

Today the HPC network topology is a web of equally interconnected nodes. Any loss in a region of the network causes significant re-routing pressures, and likely application failure. The EuroEXA network topology is built using three different approaches which together reduces the impact of any failure in the topology. The direct links at blade and network group protects those resources from failures elsewhere in the system topology with these topologies regions adopting a geographical addressing scheme able compensate for disconnects independent of the implemented higher-level topology decisions otherwise driven by system size.

Communications Resilience

The ExaNET communications protocol used across the topology has been designed with resilience on the end-to-end communications. The hardware designs have been partitioned to ensure physical and link level failures are isolated so that routing paths can be adjusted – simplified by the locality hierarchy of the overall topology. Novel approaches to flow control and back pressure management also reduces failures due to potential overdriving the network as a consequence of adopting a hierarchy of increasing bandwidths necessary to reduce port counts and the total hop count across the topology.

Checkpointing Efficiency

The industries ongoing enhancements in how to checkpoint a node, EuroEXA focuses on increasing the efficiency and hence “down time” associated with checkpointing. Adopting the learnings from ExaNEST virtualised checkpointing, the traditional centralised HPC storage system is unable to scale in support of checkpointing at exascale. The fastest storage solutions in the IO500 are currently capable of a few 100GBytes/s. Current TOP500 systems are 100,000’s of nodes, meaning less than 1Mbyte bandwidth per node to share machine snapshots between nodes for resilience. Leading to low fidelity checkpoints, and significant impact to overall performance. The EuroEXA system creates the HPC distributed storage system between and across all nodes which would provide a 100,000 node system the ability to snapshot at over 200TBytes/s with the EuroEXA network design able to distribute checkpoints for resilience without overloading the interconnect.

Memory Resilience

The physical protection of data in DRAM has been limited to EEC, typically with the ability to fix a single bit and identify a 2nd bit of failure. Any further errors going unnoticed. Although bit-flipping errors are reduced on EEC DRAM, research has shown such multi-bit errors can and do occur. Adding ECC to a memory both reduces the effective capacity, but given the complexity is its generation (10 level XOR) limits performance. The EuroEXA memory compression technologies show that compression could reduce the cost of ECC protection, while the design showing that being more intelligent around how memory is accessed can provide more time for manipulation of the data stream. We see this as a foundation to enable a future investigation of software defined, multi-bit EEC schemes able to move beyond the single correct/single detect schemes while maintaining capacity and performance.

Serdes and Device Resilience

The foundation of a HPC system are the analogue and digital devices from which everything else is built. The security and resilience of these devices is fundamental to the correct operation of the higher-level circuits and cores. EuroEXA investigated various structures and approaches to increase the resilience to wear and security issues – which together would reduce the occurrence of failures, and hence increase system level resilience.