ASSOCIATED LEARNING: A METHODOLOGY TO DECOMPOSE END-TO-END BACKPROPAGATION ON CNN, RNN, AND TRANSFORMER

Dennis Y. Wu; Di Nan Lin; Vincent F. Chen; Hung Hsuan Chen

ASSOCIATED LEARNING: A METHODOLOGY TO DECOMPOSE END-TO-END BACKPROPAGATION ON CNN, RNN, AND TRANSFORMER

Dennis Y. Wu, Di Nan Lin, Vincent F. Chen, Hung Hsuan Chen

Department of Computer Science and Information Engineering

Research output: Contribution to conference › Paper › peer-review

Abstract

We study associated learning (AL), an alternative methodology to end-to-end backpropagation (BP). We introduce the workflow to convert a neural network into an AL-form network such that AL can be used to learn parameters for various types of neural networks. We compare AL and BP on some of the most successful neural networks: convolutional neural networks, recurrent neural networks, and Transformers. Experimental results show that AL consistently outperforms BP on open datasets. We discuss possible reasons for AL's success, its limitations, and AL's newly discovered properties. Our implementation is available at https://github.com/Hibb-bb/AL.

Original language	English
State	Published - 2022
Event	10th International Conference on Learning Representations, ICLR 2022 - Virtual, Online Duration: 25 Apr 2022 → 29 Apr 2022

Conference

Conference	10th International Conference on Learning Representations, ICLR 2022
City	Virtual, Online
Period	25/04/22 → 29/04/22

Cite this

@conference{07b019f398174f3e8fbe321f4a88db9b,

title = "ASSOCIATED LEARNING: A METHODOLOGY TO DECOMPOSE END-TO-END BACKPROPAGATION ON CNN, RNN, AND TRANSFORMER",

abstract = "We study associated learning (AL), an alternative methodology to end-to-end backpropagation (BP). We introduce the workflow to convert a neural network into an AL-form network such that AL can be used to learn parameters for various types of neural networks. We compare AL and BP on some of the most successful neural networks: convolutional neural networks, recurrent neural networks, and Transformers. Experimental results show that AL consistently outperforms BP on open datasets. We discuss possible reasons for AL's success, its limitations, and AL's newly discovered properties. Our implementation is available at https://github.com/Hibb-bb/AL.",

author = "Wu, {Dennis Y.} and Lin, {Di Nan} and Chen, {Vincent F.} and Chen, {Hung Hsuan}",

note = "Publisher Copyright: {\textcopyright} 2022 ICLR 2022 - 10th International Conference on Learning Representationss. All rights reserved.; 10th International Conference on Learning Representations, ICLR 2022 ; Conference date: 25-04-2022 Through 29-04-2022",

year = "2022",

language = "???core.languages.en_GB???",

}

TY - CONF

T1 - ASSOCIATED LEARNING

T2 - 10th International Conference on Learning Representations, ICLR 2022

AU - Wu, Dennis Y.

AU - Lin, Di Nan

AU - Chen, Vincent F.

AU - Chen, Hung Hsuan

PY - 2022

Y1 - 2022

N2 - We study associated learning (AL), an alternative methodology to end-to-end backpropagation (BP). We introduce the workflow to convert a neural network into an AL-form network such that AL can be used to learn parameters for various types of neural networks. We compare AL and BP on some of the most successful neural networks: convolutional neural networks, recurrent neural networks, and Transformers. Experimental results show that AL consistently outperforms BP on open datasets. We discuss possible reasons for AL's success, its limitations, and AL's newly discovered properties. Our implementation is available at https://github.com/Hibb-bb/AL.

AB - We study associated learning (AL), an alternative methodology to end-to-end backpropagation (BP). We introduce the workflow to convert a neural network into an AL-form network such that AL can be used to learn parameters for various types of neural networks. We compare AL and BP on some of the most successful neural networks: convolutional neural networks, recurrent neural networks, and Transformers. Experimental results show that AL consistently outperforms BP on open datasets. We discuss possible reasons for AL's success, its limitations, and AL's newly discovered properties. Our implementation is available at https://github.com/Hibb-bb/AL.

UR - http://www.scopus.com/inward/record.url?scp=85150382595&partnerID=8YFLogxK

M3 - 會議論文

AN - SCOPUS:85150382595

Y2 - 25 April 2022 through 29 April 2022

ER -

ASSOCIATED LEARNING: A METHODOLOGY TO DECOMPOSE END-TO-END BACKPROPAGATION ON CNN, RNN, AND TRANSFORMER

Abstract

Conference

Fingerprint

Cite this