On the Information Capacity of Hinge Functions

Authors

G. Alirezaei, R. Mathar,

Abstract

        So called hinge functions play an important role in many applications, e.g., deep learning, support vector machines, regression, classification and others. A thorough theory, which
explains why some of these applications are so successful for their respective purpose, is still missing. This paper aims at filling a knowledge gap by answering the question of how much information can be conveyed across a neural node, e.g., which uses the hinge loss function on weighted agglomerated information from precedent nodes. We hope that insight from artificial neural networks may also have implications for understanding biological information processing. As key results in this paper, an elegant representation of mutual information is derived and, furthermore, some structural properties of a channel, that consists of a certain input signal which is overlaid by additive noise and is filtered by a hinge function, are investigated. Determining the capacity of this channel in an explicit form, although an important fundamental problem, seems to be extremely difficult and unsolved as of today. Thus, necessary and sufficient conditions for an optimal input signal along with upper bounds on the capacity are deduced. Furthermore, we conjecture that exponentially distributed input signals are asymptotically capacity-achieving in the high SNR regime.

BibTEX Reference Entry 

@inproceedings{AlMa16,
	author = {Gholamreza Alirezaei and Rudolf Mathar},
	title = "On the Information Capacity of Hinge Functions",
	pages = "443-447",
	booktitle = "2016 International Symposium on Information Theory and its Applications (ISITA'16)",
	address = {Monterey, California, USA},
	month = Oct,
	year = 2016,
	hsb = RWTH-2016-09659,
	}

Downloads

 Download paper  Download bibtex-file

This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights there in are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. In most cases, these works may not be reposted without the explicit permission of the copyright holder.