View in EDS PDF Full Text

A hybrid approach for content extraction with text density and visual importance of DOM nodes.

Saved in:
Bibliographic Details
Title: A hybrid approach for content extraction with text density and visual importance of DOM nodes.
Authors: Song, Dandan1, Sun, Fei, Liao, Lejian1 liaolj@bit.edu.cn
Source: Knowledge & Information Systems. Jan2015, Vol. 42 Issue 1, p75-96. 22p.
Subjects: Data mining, Document Object Model (Web development technology), Web databases, Database searching, Object-oriented programming
Abstract: Additional contents in web pages, such as navigation panels, advertisements, copyrights and disclaimer notices, are typically not related to the main subject and may hamper the performance of Web data mining. They are traditionally taken as noises and need to be removed properly. To achieve this, two intuitive and crucial kinds of information-the textual information and the visual information of web pages-is considered in this paper. Accordingly, Text Density and Visual Importance are defined for the Document Object Model (DOM) nodes of a web page. Furthermore, a content extraction method with these measured values is proposed. It is a fast, accurate and general method for extracting content from diverse web pages. And with the employment of DOM nodes, the original structure of the web page can be preserved. Evaluated with the CleanEval benchmark and with randomly selected pages from well-known Web sites, where various web domains and styles are tested, the effect of the method is demonstrated. The average F1-scores with our method were 8.7 % higher than the best scores among several alternative methods. [ABSTRACT FROM AUTHOR]
Copyright of Knowledge & Information Systems is the property of Springer Nature and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)
Database: Engineering Source
FullText Links:
  – Type: pdflink
Text:
  Availability: 0
Header DbId: egs
DbLabel: Engineering Source
An: 100240026
AccessLevel: 6
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: A hybrid approach for content extraction with text density and visual importance of DOM nodes.
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Song%2C+Dandan%22">Song, Dandan</searchLink><relatesTo>1</relatesTo><br /><searchLink fieldCode="AR" term="%22Sun%2C+Fei%22">Sun, Fei</searchLink><br /><searchLink fieldCode="AR" term="%22Liao%2C+Lejian%22">Liao, Lejian</searchLink><relatesTo>1</relatesTo><i> liaolj@bit.edu.cn</i>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="JN" term="%22Knowledge+%26+Information+Systems%22">Knowledge & Information Systems</searchLink>. Jan2015, Vol. 42 Issue 1, p75-96. 22p.
– Name: Subject
  Label: Subjects
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Data+mining%22">Data mining</searchLink><br /><searchLink fieldCode="DE" term="%22Document+Object+Model+%28Web+development+technology%29%22">Document Object Model (Web development technology)</searchLink><br /><searchLink fieldCode="DE" term="%22Web+databases%22">Web databases</searchLink><br /><searchLink fieldCode="DE" term="%22Database+searching%22">Database searching</searchLink><br /><searchLink fieldCode="DE" term="%22Object-oriented+programming%22">Object-oriented programming</searchLink>
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Additional contents in web pages, such as navigation panels, advertisements, copyrights and disclaimer notices, are typically not related to the main subject and may hamper the performance of Web data mining. They are traditionally taken as noises and need to be removed properly. To achieve this, two intuitive and crucial kinds of information-the textual information and the visual information of web pages-is considered in this paper. Accordingly, Text Density and Visual Importance are defined for the Document Object Model (DOM) nodes of a web page. Furthermore, a content extraction method with these measured values is proposed. It is a fast, accurate and general method for extracting content from diverse web pages. And with the employment of DOM nodes, the original structure of the web page can be preserved. Evaluated with the CleanEval benchmark and with randomly selected pages from well-known Web sites, where various web domains and styles are tested, the effect of the method is demonstrated. The average F1-scores with our method were 8.7 % higher than the best scores among several alternative methods. [ABSTRACT FROM AUTHOR]
– Name: AbstractSuppliedCopyright
  Label:
  Group: Ab
  Data: <i>Copyright of Knowledge & Information Systems is the property of Springer Nature and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract.</i> (Copyright applies to all Abstracts.)
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=egs&AN=100240026
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1007/s10115-013-0687-x
    Languages:
      – Code: eng
        Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 22
        StartPage: 75
    Subjects:
      – SubjectFull: Data mining
        Type: general
      – SubjectFull: Document Object Model (Web development technology)
        Type: general
      – SubjectFull: Web databases
        Type: general
      – SubjectFull: Database searching
        Type: general
      – SubjectFull: Object-oriented programming
        Type: general
    Titles:
      – TitleFull: A hybrid approach for content extraction with text density and visual importance of DOM nodes.
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Song, Dandan
      – PersonEntity:
          Name:
            NameFull: Sun, Fei
      – PersonEntity:
          Name:
            NameFull: Liao, Lejian
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 01
              Text: Jan2015
              Type: published
              Y: 2015
          Identifiers:
            – Type: issn-print
              Value: 02191377
          Numbering:
            – Type: volume
              Value: 42
            – Type: issue
              Value: 1
          Titles:
            – TitleFull: Knowledge & Information Systems
              Type: main
ResultId 1