Project > Infrastructure Development
Infrastructure Development
EuroEXA has developed and demonstrated advanced infrastructures for HPC – not only maximising compute performance, but also improving energy efficiency across every element of the system.
To achieve this, we harnessed advanced chassis-level liquid cooling technologies. This maximises thermal density, reducing the energy consumed by the interconnect between elements. It’s a ground-breaking approach that delivers a step-change in efficiency because it reduces the transmission distance between computing elements and, therefore, reduces communications latencies too. This technology also maximises thermal output, offering the opportunity to capture and reuse waste heat – which can help offset the running costs of the system.
EuroEXA’s design approach offers a meta-modular infrastructure – with modular data centre container facilities that can be transported by road. And, because it works with open compute standards, it offers interchangeable compute across modular cabinets, chassis and compute elements.
The energy efficiency is boosted further thanks to the innovative dynamic power delivery, which consolidates and phase-balances power conversion, minimising the number of power-conversion steps between the AC supply and the compute element. Finally, this liquid-cooling technology also provides enhanced resilience, delivering a robust infrastructure to help the platform withstand the rigours of real-world HPC applications.
Powerful Performance with enhanced efficiency
Iceotope’s K:UL Infrastructure™ drives compute performance by delivering and distributing DC power when it is needed, while ensuring that any heat the system generates is removed rapidly – allowing the electronic components to operate at their maximum operating frequency at all times.
This infrastructure is designed around the goal of minimising energy consumption – targeting a PUE of 1.03, which equates to just 3% of energy consumed beyond the IT. Factor in those opportunities for recapturing reusable heat, and we can in fact target “PUE parity”, as the amount of energy that be reused is greater than the energy used in the infrastructure supporting the IT.
Thermal Density
With the K:UL Infrastructure™ chassis-level liquid-cooling technology, the system immerses the electronic components in an advanced dielectric coolant. With a process that could be likened to a shower or dishwasher, the heat-generating components are cooled by a constant flow of liquid, circulating at a high rate to remove heat rapidly.
This technology delivers a Thermal Density of over 100kW in a single cabinet – similar to a fossil-fuel burning heating system for an office block. This not only delivers the substantial thermal output to offer viable heat-reuse applications, but also maximises compute effectiveness by reducing the distance that high frequency data needs to cover across the system.
Thermal Output
The EuroEXA system delivers the thermal output that can be harnessed for other purposes - reusing the energy to heat buildings or for industrial processes. The low grade heat output targets a temperature of 55C or more in water, with a return temperature 10C less. This is combined with a thermal density of over 100kW in a single cabinet, rivalling fossil-fuel based heating systems.
This combination offers a viable system for both computation and energy reuse, reducing operating costs across the system. The system targets “PUE parity” when factoring in reusable heat, as the energy that can be captured and reused is greater than the amount of energy needed by the infrastructure supporting the IT.
Infrastructure Meta-Modularity
EuroEXA uses a meta-modular approach to adapt to the latest Open Hardware specifications. Drawing inspiration from the Open Compute Project, EuroEXA’s hardware includes a number of modular components, with different levels of modularity across both the infrastructure and hardware – from the building to the electronics.
At the infrastructure level, EuroEXA uses a Modular Data Centre facility that can be stacked in 3 dimensions, minimising the distance between computing elements while maximising scalability. At the middle level, EuroEXA uses Open Compute Project cabinets, with a modular power bus, modular manifolds and a 21” rack form factor.
This meta-modular approach enables our vision for an infrastructure that can swiftly adopt new hardware technologies, building an extensive system for scaling out to ExaScale, while offering maximum flexibility and variety of hardware.
Dynamic Power Delivery
The EuroEXA platform stands apart from most computing infrastructures in the way it delivers power to hardware. Using a dynamic delivery approach, it takes power conversion and supply from seven stages of conversion to just one. Most infrastructures start with AC power, and then convert to DC power in a three-phase, three-stage power supply.
This power is then used to charge batteries, before conversion back to AC power by an inverter for supply to cabinets. Then, within each cabinet, there is a power distribution unit that splits the AC power to individual power supplies that again go through a three-stage process to convert to DC power, before providing this power to the servers.
The EuroEXA approach takes three-phase AC power and uses a special cabinet-mounted one-stage three-phase power supply developed by the ExaPower project.This converts power directly to 48V DC, which is connected to a 48V DC power distribution in the cabinet, which has batteries attached. This DC power then goes directly to the EuroEXA servers.
The resilience of liquid cooling
The liquid cooling system offers a range of advantages over alternatives in terms of resilience. It’s an approach that offers the ability to rapidly maintain the system, as well as riding through faults and scheduling repairs at a later date. The facility-level liquid cooling loop, which connects to all of the servers by quick disconnect valves, operates at sub atmospheric pressure.
This means that if a fault occurs, coolant will not leak out – instead, air leaks in. Then, the pumping system has a mechanism for separating air from the coolant before venting it – ensuring that the cooling system continues to operate. EuroEXA Testbeds 2 and 3 offer the first demonstration of this technological approach in a chassis-level liquid-cooling solution.