Fastest data-sorting operation in a cloud competition

In 2016, the Chinese company Tencent Corporation achieved the fastest data-sorting operation; using a 512-node OpenPOWER cluster, they designed and implemented a procedure capable of sorting 100 Terabytes (TB) in 98.8 seconds in the Indy GraySort category competition. This corresponded to a sort rate of 60.7 TB/min (terabytes per minute).

As data plays a crucial role in our lives, fast processing of large amounts data is very important. The “sorting” operation is a key step in data-processing, which can be expensive and time-consuming.

A Terabyte corresponds to 1,000,000,000,000 bytes; this is the equivalent of about 210 DVDs or 1,423 CDs full of data. The sorting performance of this record is about 100 times faster than the sorting speed obtained 10 years ago by IBM Almaden Research (sort rate of 125GB/min).

For this challenging task, there has been a considerable improvement after only a decade. With this performance, the Tencent Corporation broke the record previously set by AliCloud, which in 2015 sorted 100 TB of data in 329 seconds (18.24 TB/min).

In 2016, the Tencent Corporation also broke the record for the fastest data-sorting operation in the Daytona GraySort competition, a less specific and more general-purpose sort; in this category, they achieved a sort rate of 44.8 TB/min.