Scaling out with ExaNet

ExaNet is the EuroEXA approach for large-scale, multi-tired interconnect. 

ExaNet at a glance:

  • Hybrid Torus Topology for inter-node communication 
    • Quadrant Level → all to all
    • Blade Level → Dragonfly/Full Crossbar
    • Network Group →  3D Torus 
    • Rack Level →  Fat Tree
  • Light-Weight Custom Communication Protocol (low latency/high bw)
  • Reliability and QoS
    • Traffic congestion control 
    • End-to-end reliability guarantees by retransmission 
    • Fault-awareness at system level 
  • RDMA offload 
  • Optimized communication libraires 
    • API: user-space software stack
    • MPIpoint-to-point and collective 

Blade

NetworkGroup

  • 8 Blades
  • 3.2Tbps
  • Torus Topology 

Rack

  • 4 NG
  • 6.4Tbps
  • Fat Tree
  • 110kW

System

  • 2MW in modular facility
  • PUE 1.0x
  • Heat Reuse Capable
  • Low Cost Facilities