View in EDS HTML Full Text PDF Full Text

Xpert: Empowering Incident Management with Query Recommendations via Large Language Models.

Saved in:
Bibliographic Details
Title: Xpert: Empowering Incident Management with Query Recommendations via Large Language Models.
Authors: Jiang, Yuxuan1 jyuxuan@umich.edu, Zhang, Chaoyun2 chaoyunzhang@microsoft.com, He, Shilin2 shilin.he@microsoft.com, Yang, Zhihao3 zhihaoyang@stu.pku.edu.cn, Ma, Minghua2 minghuama@microsoft.com, Qin, Si2 si.qin@microsoft.com, Kang, Yu2 yu.kang@microsoft.com, Dang, Yingnong4 dang.yingnong@microsoft.com, Rajmohan, Saravan5 saravan.rajmohan@microsoft.com, Lin, Qingwei2 qlin@microsoft.com, Zhang, Dongmei2 dongmeiz@microsoft.com
Source: ICSE: International Conference on Software Engineering. 2024, p1-13. 13p.
Subjects: Query languages (Computer science), Language models, Domain-specific programming languages, Machine learning, Artificial intelligence
Abstract: Large-scale cloud systems play a pivotal role in modern IT infrastructure. However, incidents occurring within these systems can lead to service disruptions and adversely affect user experience. To swiftly resolve such incidents, on-call engineers depend on crafting domain-specific language (DSL) queries to analyze telemetry data. However, writing these queries can be challenging and time-consuming. This paper presents a thorough empirical study on the utilization of queries of KQL, a DSL employed for incident management in a large-scale cloud management system at Microsoft. The findings obtained underscore the importance and viability of KQL queries recommendation to enhance incident management. Building upon these valuable insights, we introduce Xpert, an end-to-end machine learning framework that automates KQL recommendation process. By leveraging historical incident data and large language models, Xpert generates customized KQL queries tailored to new incidents. Furthermore, Xpert incorporates a novel performance metric called Xcore, enabling a thorough evaluation of query quality from three comprehensive perspectives. We conduct extensive evaluations of Xpert, demonstrating its effectiveness in offline settings. Notably, we deploy Xpert in the real production environment of a large-scale incident management system in Microsoft, validating its efficiency in supporting incident management. To the best of our knowledge, this paper represents the first empirical study of its kind, and Xpert stands as a pioneering DSL query recommendation framework designed for incident management. [ABSTRACT FROM AUTHOR]
Copyright of ICSE: International Conference on Software Engineering is the property of Association for Computing Machinery and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)
Database: Engineering Source
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
Text:
  Availability: 1
Header DbId: egs
DbLabel: Engineering Source
An: 185196361
AccessLevel: 6
PubType: Conference
PubTypeId: conference
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Xpert: Empowering Incident Management with Query Recommendations via Large Language Models.
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Jiang%2C+Yuxuan%22">Jiang, Yuxuan</searchLink><relatesTo>1</relatesTo><i> jyuxuan@umich.edu</i><br /><searchLink fieldCode="AR" term="%22Zhang%2C+Chaoyun%22">Zhang, Chaoyun</searchLink><relatesTo>2</relatesTo><i> chaoyunzhang@microsoft.com</i><br /><searchLink fieldCode="AR" term="%22He%2C+Shilin%22">He, Shilin</searchLink><relatesTo>2</relatesTo><i> shilin.he@microsoft.com</i><br /><searchLink fieldCode="AR" term="%22Yang%2C+Zhihao%22">Yang, Zhihao</searchLink><relatesTo>3</relatesTo><i> zhihaoyang@stu.pku.edu.cn</i><br /><searchLink fieldCode="AR" term="%22Ma%2C+Minghua%22">Ma, Minghua</searchLink><relatesTo>2</relatesTo><i> minghuama@microsoft.com</i><br /><searchLink fieldCode="AR" term="%22Qin%2C+Si%22">Qin, Si</searchLink><relatesTo>2</relatesTo><i> si.qin@microsoft.com</i><br /><searchLink fieldCode="AR" term="%22Kang%2C+Yu%22">Kang, Yu</searchLink><relatesTo>2</relatesTo><i> yu.kang@microsoft.com</i><br /><searchLink fieldCode="AR" term="%22Dang%2C+Yingnong%22">Dang, Yingnong</searchLink><relatesTo>4</relatesTo><i> dang.yingnong@microsoft.com</i><br /><searchLink fieldCode="AR" term="%22Rajmohan%2C+Saravan%22">Rajmohan, Saravan</searchLink><relatesTo>5</relatesTo><i> saravan.rajmohan@microsoft.com</i><br /><searchLink fieldCode="AR" term="%22Lin%2C+Qingwei%22">Lin, Qingwei</searchLink><relatesTo>2</relatesTo><i> qlin@microsoft.com</i><br /><searchLink fieldCode="AR" term="%22Zhang%2C+Dongmei%22">Zhang, Dongmei</searchLink><relatesTo>2</relatesTo><i> dongmeiz@microsoft.com</i>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="JN" term="%22ICSE%3A+International+Conference+on+Software+Engineering%22">ICSE: International Conference on Software Engineering</searchLink>. 2024, p1-13. 13p.
– Name: Subject
  Label: Subjects
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Query+languages+%28Computer+science%29%22">Query languages (Computer science)</searchLink><br /><searchLink fieldCode="DE" term="%22Language+models%22">Language models</searchLink><br /><searchLink fieldCode="DE" term="%22Domain-specific+programming+languages%22">Domain-specific programming languages</searchLink><br /><searchLink fieldCode="DE" term="%22Machine+learning%22">Machine learning</searchLink><br /><searchLink fieldCode="DE" term="%22Artificial+intelligence%22">Artificial intelligence</searchLink>
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Large-scale cloud systems play a pivotal role in modern IT infrastructure. However, incidents occurring within these systems can lead to service disruptions and adversely affect user experience. To swiftly resolve such incidents, on-call engineers depend on crafting domain-specific language (DSL) queries to analyze telemetry data. However, writing these queries can be challenging and time-consuming. This paper presents a thorough empirical study on the utilization of queries of KQL, a DSL employed for incident management in a large-scale cloud management system at Microsoft. The findings obtained underscore the importance and viability of KQL queries recommendation to enhance incident management. Building upon these valuable insights, we introduce Xpert, an end-to-end machine learning framework that automates KQL recommendation process. By leveraging historical incident data and large language models, Xpert generates customized KQL queries tailored to new incidents. Furthermore, Xpert incorporates a novel performance metric called Xcore, enabling a thorough evaluation of query quality from three comprehensive perspectives. We conduct extensive evaluations of Xpert, demonstrating its effectiveness in offline settings. Notably, we deploy Xpert in the real production environment of a large-scale incident management system in Microsoft, validating its efficiency in supporting incident management. To the best of our knowledge, this paper represents the first empirical study of its kind, and Xpert stands as a pioneering DSL query recommendation framework designed for incident management. [ABSTRACT FROM AUTHOR]
– Name: AbstractSuppliedCopyright
  Label:
  Group: Ab
  Data: <i>Copyright of ICSE: International Conference on Software Engineering is the property of Association for Computing Machinery and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract.</i> (Copyright applies to all Abstracts.)
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=egs&AN=185196361
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1145/3597503.3639081
    Languages:
      – Code: eng
        Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 13
        StartPage: 1
    Subjects:
      – SubjectFull: Query languages (Computer science)
        Type: general
      – SubjectFull: Language models
        Type: general
      – SubjectFull: Domain-specific programming languages
        Type: general
      – SubjectFull: Machine learning
        Type: general
      – SubjectFull: Artificial intelligence
        Type: general
    Titles:
      – TitleFull: Xpert: Empowering Incident Management with Query Recommendations via Large Language Models.
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Jiang, Yuxuan
      – PersonEntity:
          Name:
            NameFull: Zhang, Chaoyun
      – PersonEntity:
          Name:
            NameFull: He, Shilin
      – PersonEntity:
          Name:
            NameFull: Yang, Zhihao
      – PersonEntity:
          Name:
            NameFull: Ma, Minghua
      – PersonEntity:
          Name:
            NameFull: Qin, Si
      – PersonEntity:
          Name:
            NameFull: Kang, Yu
      – PersonEntity:
          Name:
            NameFull: Dang, Yingnong
      – PersonEntity:
          Name:
            NameFull: Rajmohan, Saravan
      – PersonEntity:
          Name:
            NameFull: Lin, Qingwei
      – PersonEntity:
          Name:
            NameFull: Zhang, Dongmei
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 05
              Text: 2024
              Type: published
              Y: 2024
          Titles:
            – TitleFull: ICSE: International Conference on Software Engineering
              Type: main
ResultId 1