History
Claude E. Shannon (1916–2001) founded information theory with his classic paper "A Mathematical Theory of Communication," published in the Bell System Technical Journal in July and October of 1948. At a beginning of his paper, Shannon asserted that "The fundamental problem of communication is that of reproducing at one point, either exactly or approximately, a message selected at another point." His theory first considered communication as a strictly declared mathematical condition within statistics and gave communications engineers a way to determine a capacity of a communication channel in terms of the common currency of bits. This condition is known as a channel coding problem. A transmission a share of a theory is non caring by having the meaning (semantics) of the message conveyed.

Another placed of ideas inside info theory relates to data compression. Utilizing a technical indicator description for data, information theory quantifies the total of bits needed to describe a information. There are ii formulations for the compression condition -- within lossless data compression the data must exist as reconstructed exactly, whereas lossy data compression examines how several bits come required to reconstruct the information to inside a specified fidelity level. This fidelity level is measured by the work known as the distortion work. Around principles theory this is known as rate distortion theory. Two lossless & lossy source codes develop bits at a output which may be utilized when the inputs to the channel codes mentioned above.

This section of references theory into compression & transmission is justified per information transmission theorems, or source-channel separation theorems that justify a utilise of bits when the universal currency for principles inside numerous contexts. Still, these theorems sole hang on to in the situation in which of these transmitting user wishes to communicate to 1 getting user. Inside scenarios sustaining supplementary than 1 transmitter (a multiple-access channel), more than of these receiver (a broadcast channel) or go-between "helpers" (a relay channel), or further general networks, compression followed by transmission might there are no protracted become optimum. Network information theory refers to these multi-agent communication system.

These are usually believed that a modern discipline of data theory began sustaining a publication of Shannon's article "The Mathematical Theory of Communication" in the Bell System Technical Journal in July and October of 1948. This functiin drew on earliest publications by Harry Nyquist and Ralph Hartley. In the run of working out the theory of communications that can be applied by electrical engineers to design better telecommunication equipment, Shannon defined the measure of facts content of the message

& entropy, or mean total of facts by the message:

(in which p(m_{i personally}) is the probability of message m_i) that, whilst applied to an trading tools source, can determine a capacity of a channel compulsory to transmit the source. Whenever a logarithm around the formulthe is taken to base 2, so it gives a measures of message data & entropy in bits. Shannon's measure of entropy come to become taken as a measure of the facts contained around the message, when opposed to the part of the message that is strictly determined (hence predictable) by inherent structures, like redundancy in the structure of languages or even even the technical indicator properties of a language on to the frequencies of occurrence of different letter or word pairs, triplets, etc. View Markov chains.

Recently, still, it has emerged that entropy was defined & utilized when you took a 2nd Globe War by Alan Turing at Bletchley Park. Turing known as it "weight of evidence" & measured it inside units known as bans & decibans. (This is does'nt to exist as confused using a weight of grounds to believe defined by I.J. Good and described in the article statistical inference, which Turing also tackled and named "log-odds" or "lods".) Turing and Shannon collaborated during the war but it appears that they independently created the concept. (Information come given within Alan Turing: The Enigma by Andrew Hodges.)

Relation with thermodynamic entropy

Information entropy as defined by Shannon and added upon by more physicists is closely related thermodynamic entropy.

Boltzmann and Gibbs did considerable work in statistical thermodynamics. This function was a inspiration for adopting a term entropy within reference theory. There are deep relationships between entropy in the thermodynamic & informational senses. E.g., Maxwell's demon needs information to reverse thermodynamic entropy and getting that information exactly balances out the thermodynamic gain that the demon would otherwise achieve.

Among more utile measures of principles is mutual information, a measure of the technical indicator dependence between deuce random variables. Reciprocal reference is defined for deuce cases $X$ & $Y$ as

in which $H(X,\; Y)$ is the joint entropy, defined as

& $H(X|Y)$ is the conditional entropy of $X$ conditioned in researching $Y$. Intrinsically, a reciprocal references may be intuitively considered a total of uncertainty inside $X$ that is eliminated by observations of $Y$ & contrariwise. Whilst a random variables within wonder come continuous like than distinct, a sums may be replaced by having integrals & densities utilized in situ of probability mass functions.

Reciprocal reference is closely related the log-likelihood ratio test in a context of contingency tables and the Multinomial distribution and to Pearson's χ² test: mutual information can be considered a statistic for assessing independence between a pair of variables, and has a well-specified asymptotic distribution. Likewise, reciprocal tools may be expressed through the Kullback-Leibler divergence:

Shannon facts is appropriate for with measurements of uncertainty all over an disordered space. An guide measure of tools was created by Fisher for with measurements of uncertainty all over an orderly space. For instance, Shannon info is utilized above a space of alphabetic letters, when letters don't stand 'distances' between the two. For facts all about the value of the continuous parameter like a human's height, Fisher references is utilized, when guessed heights wash have a easily-chiseled few feet away.

A. N. Kolmogorov introduced an information measure that is based on the shortest algorithm that can recreate it; see Kolmogorov complexity.

Extensions in progress
Inside 1995, Tim Palmer signalled two unwritten assumptions just about Shannon's definition of reference that manufactured it unsuitable per se to quantum mechanics:

the supposition that there exists such the tool as an discernible state (for even example the upper face a die or a coin) prior to a observation begins

A fact that caring this state doesn't depend on the the correct sequence where observations come processed (commutativity)

A article Conceptual inadequacy of the Shannon references inside quantum measuring [http://scitation.aip.org/getabs/servlet/GetabsServlet?prog=normal&id=PLRAAN000063000002022113000001&idtype=cvips&gifs=yes], published within 2001 by Anton Zeilinger [http://www.quantum.univie.ac.at/zeilinger/] and Caslav Brukner, synthesized and developed these remarks. A thus-alleged Zeilinger's principle suggests that the quantization observed in QM could be bound to information quantification (one cannot observe less than one bit, and what is not observed is by definition "random").

However these claims remain extremely controversial. For the elaborate discussion of the pertinence of the Shannon data within quantum mechanics & an argument that Zeilinger's principle can't tell you quantisation, watch Timpson [http://www.philosophy.leeds.ac.uk/Staff/CT/Index.htm] 2003 [http://arxiv.org/abs/quant-ph/0112178] & likewise Hall 2000 [http://arxiv.org/abs/quant-ph/0007116] & Mana 2004 [http://arxiv.org/abs/quant-ph/0302049]

For the tutorial in quantum trading tools view [http://members.aol.com/jmtsgibbs/infothry.htm].

Controversies

Dr. William Dembski, a proponent of Intelligent Design, controversially suggested that what he called "specified" Shannon information is relevant to making a "Design inference" that is an inference that something was in some sense planned by a intelligent agent. A theory is virtually always rejected per scientific community, though occasionally sense it will exist as breathe to produce algorithms which could detect intelligence in purely naturalistic settings, & that Dembski's idea may actually own occasionally utility, though non in the way he arranged.

Applications
Reference theory is widely utilized in Coding theory Cryptography and cryptanalysis Data communications Data compression Detection theory Estimation theory Composer James Tenney, among others such as his teacher Lejaren Hiller, has used data theory in the composition of musical works like Ergodos.