TY - GEN
T1 - ShenTu
T2 - 2018 International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2018
AU - Lin, Heng
AU - Zhu, Xiaowei
AU - Yu, Bowen
AU - Tang, Xiongchao
AU - Xue, Wei
AU - Chen, Wenguang
AU - Zhang, Lufei
AU - Hoefler, Torsten
AU - Ma, Xiaosong
AU - Liu, Xin
AU - Zheng, Weimin
AU - Xu, Jingfang
N1 - Publisher Copyright:
© 2018 IEEE.
PY - 2018/7/2
Y1 - 2018/7/2
N2 - Graphs are an important abstraction used in many scientific fields. With the magnitude of graph-structured data constantly increasing, effective data analytics requires efficient and scalable graph processing systems. Although HPC systems have long been used for scientific computing, people have only recently started to assess their potential for graph processing, a workload with inherent load imbalance, lack of locality, and access irregularity. We propose ShenTu8 the first general-purpose graph processing framework that can efficiently utilize an entire Petascale system to process multi-trillion edge graphs in seconds. ShenTu embodies four key innovations: hardware specialization, supernode routing, on-chip sorting, and degree-aware messaging, which together enable its unprecedented performance and scalability. It can traverse a record-size 70-trillion-edge graph in seconds. Furthermore, ShenTu enables the processing of a spam detection problem on a 12-trillion edge Internet graph, making it possible to identify trustworthy and spam webpages directly at the fine-grained page level.
AB - Graphs are an important abstraction used in many scientific fields. With the magnitude of graph-structured data constantly increasing, effective data analytics requires efficient and scalable graph processing systems. Although HPC systems have long been used for scientific computing, people have only recently started to assess their potential for graph processing, a workload with inherent load imbalance, lack of locality, and access irregularity. We propose ShenTu8 the first general-purpose graph processing framework that can efficiently utilize an entire Petascale system to process multi-trillion edge graphs in seconds. ShenTu embodies four key innovations: hardware specialization, supernode routing, on-chip sorting, and degree-aware messaging, which together enable its unprecedented performance and scalability. It can traverse a record-size 70-trillion-edge graph in seconds. Furthermore, ShenTu enables the processing of a spam detection problem on a 12-trillion edge Internet graph, making it possible to identify trustworthy and spam webpages directly at the fine-grained page level.
KW - Application programming interfaces
KW - Big data applications
KW - Data analysis
KW - Graph theory
KW - Supercomputers
UR - https://www.scopus.com/pages/publications/85064134571
U2 - 10.1109/SC.2018.00059
DO - 10.1109/SC.2018.00059
M3 - Conference contribution
AN - SCOPUS:85064134571
T3 - Proceedings - International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2018
SP - 706
EP - 716
BT - Proceedings - International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2018
PB - Institute of Electrical and Electronics Engineers Inc.
Y2 - 11 November 2018 through 16 November 2018
ER -