View in EDS

A Single-Shot Generalized Device Placement for Large Dataflow Graphs.

Saved in:
Bibliographic Details
Title: A Single-Shot Generalized Device Placement for Large Dataflow Graphs.
Authors: Zhou, Yanqi1 (AUTHOR), Roy, Sudip1 (AUTHOR), Abdolrashidi, Amirali1 (AUTHOR), Wong, Daniel Lin-Kit1 (AUTHOR), Ma, Peter1 (AUTHOR), Xu, Qiumin1 (AUTHOR), Mirhoseini, Azalia1 (AUTHOR), Laudon, James1 (AUTHOR)
Source: IEEE Micro. Sep/Oct2020, Vol. 40 Issue 5, p26-36. 11p.
Subjects: IEEE Computer Society, Recurrent neural networks, Artificial neural networks, Deep learning
Abstract: With increasingly complex neural network architectures and heterogeneous device characteristics, finding a reasonable graph partitioning and device placement strategy is challenging. There have been prior attempts at learned approaches for solving device placement, these approaches are computationally expensive, unable to handle large graphs consisting over 50000 nodes, and do not generalize well to unseen graphs. To address all these limitations, we propose an efficient single-shot, generalized deep RL method (SGDP) based on a scalable sequential attention mechanism over a graph neural network that is transferable to new graphs. On a diverse set of representative deep learning models, our method on average achieves 20% improvement over human placement and 18% improvement over the prior art with 15× faster convergence. We are the first to demonstrate super human performance on 8-layer recurrent neural network language model and 8-layer GNMT consisting of over 50000 nodes, on 8-GPUs. We provide rationales and sensitivity study on model architecture selections. [ABSTRACT FROM AUTHOR]
Copyright of IEEE Micro is the property of IEEE and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)
Database: Engineering Source
FullText Text:
  Availability: 0
Header DbId: egs
DbLabel: Engineering Source
An: 145693359
AccessLevel: 6
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: A Single-Shot Generalized Device Placement for Large Dataflow Graphs.
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Zhou%2C+Yanqi%22">Zhou, Yanqi</searchLink><relatesTo>1</relatesTo> (AUTHOR)<br /><searchLink fieldCode="AR" term="%22Roy%2C+Sudip%22">Roy, Sudip</searchLink><relatesTo>1</relatesTo> (AUTHOR)<br /><searchLink fieldCode="AR" term="%22Abdolrashidi%2C+Amirali%22">Abdolrashidi, Amirali</searchLink><relatesTo>1</relatesTo> (AUTHOR)<br /><searchLink fieldCode="AR" term="%22Wong%2C+Daniel+Lin-Kit%22">Wong, Daniel Lin-Kit</searchLink><relatesTo>1</relatesTo> (AUTHOR)<br /><searchLink fieldCode="AR" term="%22Ma%2C+Peter%22">Ma, Peter</searchLink><relatesTo>1</relatesTo> (AUTHOR)<br /><searchLink fieldCode="AR" term="%22Xu%2C+Qiumin%22">Xu, Qiumin</searchLink><relatesTo>1</relatesTo> (AUTHOR)<br /><searchLink fieldCode="AR" term="%22Mirhoseini%2C+Azalia%22">Mirhoseini, Azalia</searchLink><relatesTo>1</relatesTo> (AUTHOR)<br /><searchLink fieldCode="AR" term="%22Laudon%2C+James%22">Laudon, James</searchLink><relatesTo>1</relatesTo> (AUTHOR)
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="JN" term="%22IEEE+Micro%22">IEEE Micro</searchLink>. Sep/Oct2020, Vol. 40 Issue 5, p26-36. 11p.
– Name: Subject
  Label: Subjects
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22IEEE+Computer+Society%22">IEEE Computer Society</searchLink><br /><searchLink fieldCode="DE" term="%22Recurrent+neural+networks%22">Recurrent neural networks</searchLink><br /><searchLink fieldCode="DE" term="%22Artificial+neural+networks%22">Artificial neural networks</searchLink><br /><searchLink fieldCode="DE" term="%22Deep+learning%22">Deep learning</searchLink>
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: With increasingly complex neural network architectures and heterogeneous device characteristics, finding a reasonable graph partitioning and device placement strategy is challenging. There have been prior attempts at learned approaches for solving device placement, these approaches are computationally expensive, unable to handle large graphs consisting over 50000 nodes, and do not generalize well to unseen graphs. To address all these limitations, we propose an efficient single-shot, generalized deep RL method (SGDP) based on a scalable sequential attention mechanism over a graph neural network that is transferable to new graphs. On a diverse set of representative deep learning models, our method on average achieves 20% improvement over human placement and 18% improvement over the prior art with 15× faster convergence. We are the first to demonstrate super human performance on 8-layer recurrent neural network language model and 8-layer GNMT consisting of over 50000 nodes, on 8-GPUs. We provide rationales and sensitivity study on model architecture selections. [ABSTRACT FROM AUTHOR]
– Name: AbstractSuppliedCopyright
  Label:
  Group: Ab
  Data: <i>Copyright of IEEE Micro is the property of IEEE and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract.</i> (Copyright applies to all Abstracts.)
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=egs&AN=145693359
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1109/MM.2020.3015188
    Languages:
      – Code: eng
        Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 11
        StartPage: 26
    Subjects:
      – SubjectFull: IEEE Computer Society
        Type: general
      – SubjectFull: Recurrent neural networks
        Type: general
      – SubjectFull: Artificial neural networks
        Type: general
      – SubjectFull: Deep learning
        Type: general
    Titles:
      – TitleFull: A Single-Shot Generalized Device Placement for Large Dataflow Graphs.
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Zhou, Yanqi
      – PersonEntity:
          Name:
            NameFull: Roy, Sudip
      – PersonEntity:
          Name:
            NameFull: Abdolrashidi, Amirali
      – PersonEntity:
          Name:
            NameFull: Wong, Daniel Lin-Kit
      – PersonEntity:
          Name:
            NameFull: Ma, Peter
      – PersonEntity:
          Name:
            NameFull: Xu, Qiumin
      – PersonEntity:
          Name:
            NameFull: Mirhoseini, Azalia
      – PersonEntity:
          Name:
            NameFull: Laudon, James
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 09
              Text: Sep/Oct2020
              Type: published
              Y: 2020
          Identifiers:
            – Type: issn-print
              Value: 02721732
          Numbering:
            – Type: volume
              Value: 40
            – Type: issue
              Value: 5
          Titles:
            – TitleFull: IEEE Micro
              Type: main
ResultId 1