View in EDS

A Single-Shot Generalized Device Placement for Large Dataflow Graphs.

Saved in:

Bibliographic Details
Title:	A Single-Shot Generalized Device Placement for Large Dataflow Graphs.
Authors:	Zhou, Yanqi¹ (AUTHOR), Roy, Sudip¹ (AUTHOR), Abdolrashidi, Amirali¹ (AUTHOR), Wong, Daniel Lin-Kit¹ (AUTHOR), Ma, Peter¹ (AUTHOR), Xu, Qiumin¹ (AUTHOR), Mirhoseini, Azalia¹ (AUTHOR), Laudon, James¹ (AUTHOR)
Source:	IEEE Micro. Sep/Oct2020, Vol. 40 Issue 5, p26-36. 11p.
Subjects:	IEEE Computer Society, Recurrent neural networks, Artificial neural networks, Deep learning
Abstract:	With increasingly complex neural network architectures and heterogeneous device characteristics, finding a reasonable graph partitioning and device placement strategy is challenging. There have been prior attempts at learned approaches for solving device placement, these approaches are computationally expensive, unable to handle large graphs consisting over 50000 nodes, and do not generalize well to unseen graphs. To address all these limitations, we propose an efficient single-shot, generalized deep RL method (SGDP) based on a scalable sequential attention mechanism over a graph neural network that is transferable to new graphs. On a diverse set of representative deep learning models, our method on average achieves 20% improvement over human placement and 18% improvement over the prior art with 15× faster convergence. We are the first to demonstrate super human performance on 8-layer recurrent neural network language model and 8-layer GNMT consisting of over 50000 nodes, on 8-GPUs. We provide rationales and sensitivity study on model architecture selections. [ABSTRACT FROM AUTHOR]
	Copyright of IEEE Micro is the property of IEEE and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)
Database:	Engineering Source

Description
Abstract:	With increasingly complex neural network architectures and heterogeneous device characteristics, finding a reasonable graph partitioning and device placement strategy is challenging. There have been prior attempts at learned approaches for solving device placement, these approaches are computationally expensive, unable to handle large graphs consisting over 50000 nodes, and do not generalize well to unseen graphs. To address all these limitations, we propose an efficient single-shot, generalized deep RL method (SGDP) based on a scalable sequential attention mechanism over a graph neural network that is transferable to new graphs. On a diverse set of representative deep learning models, our method on average achieves 20% improvement over human placement and 18% improvement over the prior art with 15× faster convergence. We are the first to demonstrate super human performance on 8-layer recurrent neural network language model and 8-layer GNMT consisting of over 50000 nodes, on 8-GPUs. We provide rationales and sensitivity study on model architecture selections. [ABSTRACT FROM AUTHOR]
ISSN:	02721732
DOI:	10.1109/MM.2020.3015188