Optimizing center performance through coordinated data staging, scheduling and recovery

Zhe Zhang; Chao Wang; Sudharshan S. Vazhkudai; Xiaosong Ma; Gregory G. Pike; John W. Cobb; Frank Mueller

doi:10.1145/1362622.1362696

Optimizing center performance through coordinated data staging, scheduling and recovery

Zhe Zhang^*
, Chao Wang
, Sudharshan S. Vazhkudai
, Xiaosong Ma
, Gregory G. Pike
, John W. Cobb
, Frank Mueller

^*Corresponding author for this work

North Carolina State University
Oak Ridge National Laboratory

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

16 Citations (Scopus)

Abstract

Procurement and the optimized utilization of Petascale supercomputers and centers is a renewed national priority. Sustained performance and availability of such large centers is a key technical challenge significantly impacting their usability. Storage systems are known to be the primary fault source leading to data unavailability and job resubmissions. This results in reduced center performance, partially due to the lack of coordination between I/O activities and job scheduling. In this work, we propose the coordination of job scheduling with data staging/offloading and on-demand staged data reconstruction to address the availability of job input data and to improve centerwide performance. Fundamental to both mechanisms is the efficient management of transient data: in the way it is scheduled and recovered. Collectively, from a center's standpoint, these techniques optimize resource usage and increase its data/service availability. From a user's standpoint, they reduce the job turnaround time and optimize the allocated time usage. (c) 2007 ACM.

Original language	English
Title of host publication	Proceedings of the 2007 ACM/IEEE Conference on Supercomputing, SC'07
Publisher	Association for Computing Machinery
Pages	55
Number of pages	1
ISBN (Print)	9781595937643
DOIs	https://doi.org/10.1145/1362622.1362696
Publication status	Published - 16 Nov 2007
Externally published	Yes
Event	2007 ACM/IEEE Conference on Supercomputing, SC 2007 - Reno, NV, United States Duration: 10 Nov 2007 → 16 Nov 2007

Publication series

Name	Proceedings of the 2007 ACM/IEEE Conference on Supercomputing, SC'07

Conference

Conference	2007 ACM/IEEE Conference on Supercomputing, SC 2007
Country/Territory	United States
City	Reno, NV
Period	10/11/07 → 16/11/07

Keywords

Coordinated scheduling
Data scheduling
Data staging
HPC center performance optimization
Transient data recovery

Access to Document

10.1145/1362622.1362696

Cite this

Zhang, Z., Wang, C., Vazhkudai, S. S., Ma, X., Pike, G. G., Cobb, J. W., & Mueller, F. (2007). Optimizing center performance through coordinated data staging, scheduling and recovery. In Proceedings of the 2007 ACM/IEEE Conference on Supercomputing, SC'07 (pp. 55). Article 55 (Proceedings of the 2007 ACM/IEEE Conference on Supercomputing, SC'07). Association for Computing Machinery. https://doi.org/10.1145/1362622.1362696

@inproceedings{a6732311ed7549658ab60cf2690e6f5a,

title = "Optimizing center performance through coordinated data staging, scheduling and recovery",

abstract = "Procurement and the optimized utilization of Petascale supercomputers and centers is a renewed national priority. Sustained performance and availability of such large centers is a key technical challenge significantly impacting their usability. Storage systems are known to be the primary fault source leading to data unavailability and job resubmissions. This results in reduced center performance, partially due to the lack of coordination between I/O activities and job scheduling. In this work, we propose the coordination of job scheduling with data staging/offloading and on-demand staged data reconstruction to address the availability of job input data and to improve centerwide performance. Fundamental to both mechanisms is the efficient management of transient data: in the way it is scheduled and recovered. Collectively, from a center's standpoint, these techniques optimize resource usage and increase its data/service availability. From a user's standpoint, they reduce the job turnaround time and optimize the allocated time usage. (c) 2007 ACM.",

keywords = "Coordinated scheduling, Data scheduling, Data staging, HPC center performance optimization, Transient data recovery",

author = "Zhe Zhang and Chao Wang and Vazhkudai, \{Sudharshan S.\} and Xiaosong Ma and Pike, \{Gregory G.\} and Cobb, \{John W.\} and Frank Mueller",

year = "2007",

month = nov,

day = "16",

doi = "10.1145/1362622.1362696",

language = "English",

isbn = "9781595937643",

series = "Proceedings of the 2007 ACM/IEEE Conference on Supercomputing, SC'07",

publisher = "Association for Computing Machinery",

pages = "55",

booktitle = "Proceedings of the 2007 ACM/IEEE Conference on Supercomputing, SC'07",

address = "United States",

note = "2007 ACM/IEEE Conference on Supercomputing, SC 2007 ; Conference date: 10-11-2007 Through 16-11-2007",

}

Zhang, Z, Wang, C, Vazhkudai, SS, Ma, X, Pike, GG, Cobb, JW & Mueller, F 2007, Optimizing center performance through coordinated data staging, scheduling and recovery. in Proceedings of the 2007 ACM/IEEE Conference on Supercomputing, SC'07., 55, Proceedings of the 2007 ACM/IEEE Conference on Supercomputing, SC'07, Association for Computing Machinery, pp. 55, 2007 ACM/IEEE Conference on Supercomputing, SC 2007, Reno, NV, United States, 10/11/07. https://doi.org/10.1145/1362622.1362696

Optimizing center performance through coordinated data staging, scheduling and recovery. / Zhang, Zhe; Wang, Chao; Vazhkudai, Sudharshan S. et al.
Proceedings of the 2007 ACM/IEEE Conference on Supercomputing, SC'07. Association for Computing Machinery, 2007. p. 55 55 (Proceedings of the 2007 ACM/IEEE Conference on Supercomputing, SC'07).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Optimizing center performance through coordinated data staging, scheduling and recovery

AU - Zhang, Zhe

AU - Wang, Chao

AU - Vazhkudai, Sudharshan S.

AU - Ma, Xiaosong

AU - Pike, Gregory G.

AU - Cobb, John W.

AU - Mueller, Frank

PY - 2007/11/16

Y1 - 2007/11/16

N2 - Procurement and the optimized utilization of Petascale supercomputers and centers is a renewed national priority. Sustained performance and availability of such large centers is a key technical challenge significantly impacting their usability. Storage systems are known to be the primary fault source leading to data unavailability and job resubmissions. This results in reduced center performance, partially due to the lack of coordination between I/O activities and job scheduling. In this work, we propose the coordination of job scheduling with data staging/offloading and on-demand staged data reconstruction to address the availability of job input data and to improve centerwide performance. Fundamental to both mechanisms is the efficient management of transient data: in the way it is scheduled and recovered. Collectively, from a center's standpoint, these techniques optimize resource usage and increase its data/service availability. From a user's standpoint, they reduce the job turnaround time and optimize the allocated time usage. (c) 2007 ACM.

AB - Procurement and the optimized utilization of Petascale supercomputers and centers is a renewed national priority. Sustained performance and availability of such large centers is a key technical challenge significantly impacting their usability. Storage systems are known to be the primary fault source leading to data unavailability and job resubmissions. This results in reduced center performance, partially due to the lack of coordination between I/O activities and job scheduling. In this work, we propose the coordination of job scheduling with data staging/offloading and on-demand staged data reconstruction to address the availability of job input data and to improve centerwide performance. Fundamental to both mechanisms is the efficient management of transient data: in the way it is scheduled and recovered. Collectively, from a center's standpoint, these techniques optimize resource usage and increase its data/service availability. From a user's standpoint, they reduce the job turnaround time and optimize the allocated time usage. (c) 2007 ACM.

KW - Coordinated scheduling

KW - Data scheduling

KW - Data staging

KW - HPC center performance optimization

KW - Transient data recovery

UR - https://www.scopus.com/pages/publications/56749179540

U2 - 10.1145/1362622.1362696

DO - 10.1145/1362622.1362696

M3 - Conference contribution

AN - SCOPUS:56749179540

SN - 9781595937643

T3 - Proceedings of the 2007 ACM/IEEE Conference on Supercomputing, SC'07

SP - 55

BT - Proceedings of the 2007 ACM/IEEE Conference on Supercomputing, SC'07

PB - Association for Computing Machinery

T2 - 2007 ACM/IEEE Conference on Supercomputing, SC 2007

Y2 - 10 November 2007 through 16 November 2007

ER -

Zhang Z, Wang C, Vazhkudai SS, Ma X, Pike GG, Cobb JW et al. Optimizing center performance through coordinated data staging, scheduling and recovery. In Proceedings of the 2007 ACM/IEEE Conference on Supercomputing, SC'07. Association for Computing Machinery. 2007. p. 55. 55. (Proceedings of the 2007 ACM/IEEE Conference on Supercomputing, SC'07). doi: 10.1145/1362622.1362696

Optimizing center performance through coordinated data staging, scheduling and recovery

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this