This news blog provides news about the e-IRG and related e-Infrastructure topics.


World's first ARM-based Isambard supercomputer is ready for science

At the Workshops Day at ISC'18 in Frankfurt, Germany, we talked with e-IRG delegate Simon McIntosh-Smith, a professor of high-performance computing at the University of Bristol about the first ARM-based supercomputer. With some of his colleagues, Simon McIntosh-Smith designed and built Isambard, the world's first production-ready ARM-based supercomputer. The team worked together with Cray. The machine is actually a Cray XC50. It is a real production machine. It is not going to be just a testbed. One is going to run Isambard as a real science cluster. People in the UK will apply for time on this machine and go through a science review process. If the development team likes what they want to do, they will be allowed time on the Isambard machine. Hopefully, they will log in and use it like they would any other big high-performance machine.

The difference though is under the hood. Isambard will actually be a first generation of high-performance computing optimized ARM-based CPUs. The machine was due to arrive at the end of July 2018 and Simon McIntosh-Smith hoped to open up for science this Summer.

What is the size of the machine?

It is a full cabinet of the Cray XC50 hardware. It has 168 dual socket nodes. The Cavium ThunderX2 ARM CPUs that are used are 32-core each at 64 cores per node. So, this is something like just over 10.700 cores for the whole machine. Isambard uses the Cray Aries interconnect between all those 168 nodes.

If you position Isambard in the European supercomputing scale, it is tier 0?

In the UK, one would call it tier 2. The PRACE machines are tier 0. The national machines like Archer in the UK are the tier 1 machines. Isambard is a tier 2 class machine, it is a regional machine. It is operated by four universities together, called the GW4 Alliance - the Great Western four university of Bath, Bristol, Cardiff and Exeter, all in the southwest of the UK.  

This is not a machine you can buy from a catalogue?

Actually, you pretty much can. That is the other exciting thing. It is just going to be an XC50 standard product. You can go to Cray and buy one of these off their list. Cray is not the only company and not the only system vendor doing ARM-based machines now. HPE and Bull/Atos are doing them too. Fujitsu is having one coming soon. Most of the main vendors now have this on their list. It is becoming more of a commodity standard thing that is just starting to happen now.  

This is a good sign that the technology is catching on. At the ARM workshop at ISC'18, you presented some benchmarks?

Isambard is focused on real science and it is part of the science ecosystem in the UK. So, the team took ten of the most heavily used science codes on the tier 1 machine, Archer. The team has been porting and optimizing those on Isambard. The developers had some early access nodes which are basically a sort of white boxes with the right Cavium ThunderX2 CPUs in them and a pre-release of the Cray software stack. They got all of the Cray compilers, libraries and profilers, as well as all of the open source tools. They had things like GNU compilers and Clang-based compilers as well. They went through all the important science codes. They are all already compiled and running well for the prototype early access. Those are lots of chemistry codes, things like VASP and CASTEP, as well as molecular dynamics codes like GROMACS and NAMD. They have got CFD codes in the top 10 like OpenFOAM and lots of atmosphere and weather codes. In the UK, the unified model from the Met Office is one of the key codes. They also got NEMO, which is an ocean code. All those are real science codes which are already running on this prototype machine.

The team has been presenting itself across the board. Performance was very competitive with the latest x86 and Skylake from Intel. It was really a great fact. It all pretty much worked out of the box. It was good to learn that the software ecosystems are mature. The performance was good. This is really important because that means it is a genuine option for people buying new machines. It is really exciting to see what this means for the team's options when they will be buying their new machines. They will have a bit more variety and more competition. This is good for the team and the users.  

As for the machine itself, the processors are ARM?

Yes. They are Cavium ThunderX2 processors which are based on ARM.

What is the network connect of this system?

Since Isambard is a Cray XC50, it has the Aries interconnect. That is what they have been using with Intel-based systems for a long time. Also, for the team, interestingly, that is the same interconnect as within the Archer tier 1 machine. When some comparisons are being performed, there are already real results on Archer with Aries and the Cray tool chain. Now, the team can run some tests on Isambard with Aries and the Cray tool chain. The only real difference is the processor. So, the team has gone from an x86 to an ARM-based Cavium ThunderX2 processor and almost everything else is the same. The team is really seeing the difference, going just from one to the other, they can use some sort of apples-to-apples comparison which is very useful for the team.

Surely, you are looking forward to the scientific results of the machine?

It should be really good, hopefully, if the acceptance goes well. Isambard is bound to be open for science this Summer. Hopefully, people will very quickly get used to the fact that ARM works just like anything else that they use and just start using it for real science.