Style spectroscope: improve interpretability and controllability through Fourier analysis.
Saved in:
| Title: | Style spectroscope: improve interpretability and controllability through Fourier analysis. |
|---|---|
| Authors: | Jin, Zhiyu1 (AUTHOR), Shen, Xuli1 (AUTHOR), Li, Bin1 (AUTHOR) libin@fudan.edu.cn, Xue, Xiangyang1 (AUTHOR) |
| Source: | Machine Learning. Jun2024, Vol. 113 Issue 6, p3485-3503. 19p. |
| Subjects: | Spectroscope, Fourier analysis, Fourier transforms, Algorithms |
| Abstract: | Universal style transfer (UST) infuses styles from arbitrary reference images into content images. Existing methods, while enjoying many practical successes, are unable of explaining experimental observations, including different performances of UST algorithms in preserving the spatial structure of content images. In addition, methods are limited to cumbersome global controls on stylization, so that they require additional spatial masks for desired stylization. In this work, we first provide a systematic Fourier analysis on a general framework for UST. We present an equivalent form of the framework in the frequency domain. The form implies that existing algorithms treat all frequency components and pixels of feature maps equally, except for the zero-frequency component. We connect Fourier amplitude and phase with a widely used style loss and a well-known content reconstruction loss in style transfer, respectively. Based on such equivalence and connections, we can thus interpret different structure preservation behaviors between algorithms with Fourier phase. Given the interpretations, we propose two plug-and-play manipulations upon style transfer methods for better structure preservation and desired stylization. Both qualitative and quantitative experiments demonstrate the improved performance of our manipulations upon mainstreaming methods without any additional training. Specifically, the metrics are improved by 6% in average on the content images from MS-COCO dataset and the style images from WikiArt dataset. We also conduct experiments to demonstrate (1) the abovementioned equivalence, (2) the interpretability based on Fourier amplitude and phase and (3) the controllability associated with frequency components. [ABSTRACT FROM AUTHOR] |
| Copyright of Machine Learning is the property of Springer Nature and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.) | |
| Database: | Engineering Source |
|
Full text is not displayed to guests.
Login for full access.
|
|
| FullText | Links: – Type: pdflink Text: Availability: 1 |
|---|---|
| Header | DbId: egs DbLabel: Engineering Source An: 177194535 AccessLevel: 6 PubType: Academic Journal PubTypeId: academicJournal PreciseRelevancyScore: 0 |
| IllustrationInfo | |
| Items | – Name: Title Label: Title Group: Ti Data: Style spectroscope: improve interpretability and controllability through Fourier analysis. – Name: Author Label: Authors Group: Au Data: <searchLink fieldCode="AR" term="%22Jin%2C+Zhiyu%22">Jin, Zhiyu</searchLink><relatesTo>1</relatesTo> (AUTHOR)<br /><searchLink fieldCode="AR" term="%22Shen%2C+Xuli%22">Shen, Xuli</searchLink><relatesTo>1</relatesTo> (AUTHOR)<br /><searchLink fieldCode="AR" term="%22Li%2C+Bin%22">Li, Bin</searchLink><relatesTo>1</relatesTo> (AUTHOR)<i> libin@fudan.edu.cn</i><br /><searchLink fieldCode="AR" term="%22Xue%2C+Xiangyang%22">Xue, Xiangyang</searchLink><relatesTo>1</relatesTo> (AUTHOR) – Name: TitleSource Label: Source Group: Src Data: <searchLink fieldCode="JN" term="%22Machine+Learning%22">Machine Learning</searchLink>. Jun2024, Vol. 113 Issue 6, p3485-3503. 19p. – Name: Subject Label: Subjects Group: Su Data: <searchLink fieldCode="DE" term="%22Spectroscope%22">Spectroscope</searchLink><br /><searchLink fieldCode="DE" term="%22Fourier+analysis%22">Fourier analysis</searchLink><br /><searchLink fieldCode="DE" term="%22Fourier+transforms%22">Fourier transforms</searchLink><br /><searchLink fieldCode="DE" term="%22Algorithms%22">Algorithms</searchLink> – Name: Abstract Label: Abstract Group: Ab Data: Universal style transfer (UST) infuses styles from arbitrary reference images into content images. Existing methods, while enjoying many practical successes, are unable of explaining experimental observations, including different performances of UST algorithms in preserving the spatial structure of content images. In addition, methods are limited to cumbersome global controls on stylization, so that they require additional spatial masks for desired stylization. In this work, we first provide a systematic Fourier analysis on a general framework for UST. We present an equivalent form of the framework in the frequency domain. The form implies that existing algorithms treat all frequency components and pixels of feature maps equally, except for the zero-frequency component. We connect Fourier amplitude and phase with a widely used style loss and a well-known content reconstruction loss in style transfer, respectively. Based on such equivalence and connections, we can thus interpret different structure preservation behaviors between algorithms with Fourier phase. Given the interpretations, we propose two plug-and-play manipulations upon style transfer methods for better structure preservation and desired stylization. Both qualitative and quantitative experiments demonstrate the improved performance of our manipulations upon mainstreaming methods without any additional training. Specifically, the metrics are improved by 6% in average on the content images from MS-COCO dataset and the style images from WikiArt dataset. We also conduct experiments to demonstrate (1) the abovementioned equivalence, (2) the interpretability based on Fourier amplitude and phase and (3) the controllability associated with frequency components. [ABSTRACT FROM AUTHOR] – Name: AbstractSuppliedCopyright Label: Group: Ab Data: <i>Copyright of Machine Learning is the property of Springer Nature and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract.</i> (Copyright applies to all Abstracts.) |
| PLink | https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=egs&AN=177194535 |
| RecordInfo | BibRecord: BibEntity: Identifiers: – Type: doi Value: 10.1007/s10994-023-06435-5 Languages: – Code: eng Text: English PhysicalDescription: Pagination: PageCount: 19 StartPage: 3485 Subjects: – SubjectFull: Spectroscope Type: general – SubjectFull: Fourier analysis Type: general – SubjectFull: Fourier transforms Type: general – SubjectFull: Algorithms Type: general Titles: – TitleFull: Style spectroscope: improve interpretability and controllability through Fourier analysis. Type: main BibRelationships: HasContributorRelationships: – PersonEntity: Name: NameFull: Jin, Zhiyu – PersonEntity: Name: NameFull: Shen, Xuli – PersonEntity: Name: NameFull: Li, Bin – PersonEntity: Name: NameFull: Xue, Xiangyang IsPartOfRelationships: – BibEntity: Dates: – D: 01 M: 06 Text: Jun2024 Type: published Y: 2024 Identifiers: – Type: issn-print Value: 08856125 Numbering: – Type: volume Value: 113 – Type: issue Value: 6 Titles: – TitleFull: Machine Learning Type: main |
| ResultId | 1 |
