Shannon Source Coding Theorem

香农奈奎斯特采样定理
香农-奈奎斯特采样定理（Shannon-Nyquist Sampling Theorem）是一项基本的信号处理原理，它规定了一个连续时间信号的采样频率应该至少是该信号中最高频率成分的两倍，以便在离散时间中完整地重构原始信号。

这个定理是由克劳德·香农（Claude Shannon）和哈里·奈奎斯特（Harry Nyquist）在20世纪初提出的。

具体来说，香农-奈奎斯特采样定理表述如下：
如果一个连续时间信号的最高频率成分为f_max，那么为了在离散时间中准确地重建原始信号，采样频率f_s（采样率）必须满足：
f_s ≥ 2 * f_max
这意味着采样频率应至少是信号中最高频率的两倍。

如果采样频率不满足这个条件，就会出现所谓的"混叠"或"奈奎斯特折叠"，导致信号在离散时间中无法准确还原。

香农-奈奎斯特采样定理在数字信号处理、通信系统、音频处理、图像处理和各种数据采集应用中具有重要作用。

它强调了适当选择采样频率的重要性，以避免信息丢失和混叠问题，确保准确的信号重建。

因此，合理的采样频率选择是数字信号处理的基本原则之一。

简述采样定理的基本内容采样定理，也被称为奈奎斯特定理（Nyquist theorem）或香农-奈奎斯特采样定理（Shannon-Nyquist sampling theorem），是在信号处理领域中至关重要的一条基本原理。

它对数字信号处理、通信系统以及采样率等方面具有重要的指导意义。

1. 采样定理的基本内容采样定理表明，如果要正确恢复连续时间信号的完整信息，就需要以至少两倍于信号最高频率的采样频率对信号进行采样。

采样频率应该大于等于信号最高频率的两倍，即Fs >= 2 * Fmax。

采样定理的原理基于奈奎斯特频率，奈奎斯特频率是指信号频谱中的最高频率成分。

如果采样频率小于奈奎斯特频率的两倍，那么采样信号中将出现混叠现象，即频谱中的不同频率成分相互干扰，导致原信号无法准确恢复。

2. 采样定理的应用采样定理在多个领域都有广泛的应用，以下是几个常见的应用领域：音频处理：在音频信号的数字化处理中，采样定理保证了通过合适的采样率可以准确还原原始音频信号，同时避免了音频信号的混叠现象。

这就是为什么音频 CD 的采样率是44.1kHz，超过人类可听到的最高频率20kHz的两倍。

通信系统：在数字通信系统中，为了正确传输模拟信号，信号需要经过模数转换（采样）和数模转换两个过程。

采样定理确保了在采样时不会丢失信号的信息，同时在接收端通过恢复出原始信号。

这对于保证通信质量和准确传输数据来说非常关键。

图像处理：在数字图像采集中，采样定理用于设置合适的采样率，以避免图片出现信息丢失和混叠现象。

在数字摄影中，也需要根据采样定理来选择适当的像素密度，以保证图像的质量和细节。

3. 采样定理的局限性和改进采样定理的一个重要前提是信号是带限的，即信号的频谱有一个上限，超过这个上限的频率成分可以被忽略。

然而，在实际应用中，许多信号并不是严格带限的，因此采样定理可能无法完全适用。

为了克服采样定理的局限性，一种常见的方法是使用过采样（oversampling）技术。

香农采样定理英文描述Shannon Sampling Theorem or Nyquist Sampling Theorem is an important concept in digital signal processing. Thetheorem states that the sample rate must be equal to or more than twice the highest frequency component of a signal to accurately reconstruct it. In simpler terms, it means that we need to sample a signal at a rate equal to or higher thantwice its maximum frequency.The theorem was proposed by Claude Shannon in 1949, andit has been widely used in many areas of science, engineering, and technology. It has been used for signal processing in music, telecommunications, image processing, and many other fields.The following are the steps involved in understandingand using the Shannon Sampling Theorem:1. Definition of the theorem: The Shannon Sampling Theorem is a mathematical principle that describes the minimum sampling rate required to accurately represent a continuous-time signal in the digital domain.2. Understanding the importance of sampling rate: Sampling rate is the number of times a signal is sampled per unit of time. It is essential to have a higher sampling rateto capture the details of a signal accurately. If thesampling rate is not high enough, the reconstructed signalmay not be accurate.3. Finding the Nyquist frequency: The Nyquist frequencyis half of the sampling rate. It represents the maximum frequency component that can be accurately represented. Ifthe maximum frequency of a signal is above the Nyquist frequency, aliasing occurs, and the signal cannot be accurately reconstructed.4. Applying the theorem: Once the Nyquist frequency is known, it can be used to determine the minimum sample rate required to accurately reproduce the signal. This is done using the formula Fs = 2B, where Fs is the sample rate and B is the bandwidth of the signal.5. Examples: Here is a simple example that demonstrates the importance of the Shannon Sampling Theorem. Suppose we have a signal that has a maximum frequency of 100 Hz. According to the theorem, the minimum sampling rate required to accurately reproduce the signal is 200 Hz (2*100). Any lower sampling rate would lead to aliasing, and the signal cannot be reconstructed accurately.In conclusion, the Shannon Sampling Theorem is a fundamental principle in digital signal processing. It is essential to understand its importance and apply it correctly to avoid errors and inaccuracies in signal processing.。

For the next several lectures we will be discussing the von Neumann entropy and various concepts relating to it.This lecture is intended to introduce the notion of entropy and its connection to compression.7.1Shannon entropyBefore we discuss the von Neumann entropy,we will take a few moments to discuss the Shannon entropy.This is a purely classical notion,but it is appropriate to start here.The Shannon entropy of a probability vector p∈RΣis defined as follows:p(a)log(p(a)).H(p)=−∑a∈Σp(a)>0Here,and always in this course,the base of the logarithm is2.(We will write ln(α)if we wish to refer to the natural logarithm of a real numberα.)It is typical to express the Shannon entropy slightly more concisely asp(a)log(p(a)),H(p)=−∑a∈Σwhich is meaningful if we make the interpretation0log(0)=0.This is sensible given thatαlog(α)=0.limα↓0There is no reason why we cannot extend the definition of the Shannon entropy to arbitrary vectors with nonnegative entries if it is useful to do this—but mostly we will focus on probability vectors.There are standard ways to interpret the Shannon entropy.For instance,the quantity H(p)can be viewed as a measure of the amount of uncertainty in a random experiment described by the probability vector p,or as a measure of the amount of information one gains by learning the value of such an experiment.Indeed,it is possible to start with simple axioms for what a measure of uncertainty or information should satisfy,and to derive from these axioms that such a measure must be equivalent to the Shannon entropy.Something to keep in mind,however,when using these interpretations as a guide,is that the Shannon entropy is usually only a meaningful measure of uncertainty in an asymptotic sense—as the number of experiments becomes large.When a small number of samples from some experi-ment is considered,the Shannon entropy may not conform to your intuition about uncertainty,as the following example is meant to demonstrate.Example7.1.LetΣ={0,1,...,2m2},and define a probability vector p∈RΣas follows:p(a)= 1−1m2−m21≤a≤2m2.56nn∑j=1Y j−E[Y j] ≥ε =0,which is true by the(weak)law of large numbers.uu∗, Ξ⊗1L(Z) (uu∗)represents one way of measuring the quality with whichΦacts trivially on the state uu∗.Now,any purification u∈W⊗Z ofσmust take the formu=vec √k∑j=1√σB 2=k ∑j=1 σ,A j 2.Another expression for the channelfidelity isF channel(Ξ,σ)=ρ⊗n,B j A i。

解密模数转换器（ADC）分辨率和采样率分辨率和采样率是选择（模数转换器）(（ADC）) 时要考虑的两个重要因素。

为了充分理解这些，必须在一定程度上理解量化和奈奎斯特准则等概念。

在选择模数转换器(（AD）C) 的过程中要考虑的两个最重要的特性可能是分辨率和采样率。

在进行任何选择之前，应仔细考虑这两个因素。

它们将影响选择过程中的一切，从价格到所需模数转换器的底层架构。

为了为特定应用正确确定正确的分辨率和正确的采样率，应该对这些特性有一个合理的了解。

下面是与模数转换相关的术语的一些数学描述。

数学很重要，但它所代表的概念更重要。

如果您能忍受数学并理解所介绍的概念，您将能够缩小适合您应用的ADC 的数量，并且选择将变得容易得多。

量化（Quan（ti）sation）模数转换器将连续（信号）（电压或（电流））转换为由离散逻辑电平表示的数字序列。

术语量化是指将大量值转换为较小值集或离散值集的过程。

在数学上，ADC 可以被描述为量化具有大域的函数以产生具有较小域的函数。

上面的等式在数学上描述了模数转换过程。

在这里，我们将输入电压V in描述为一系列位b N-1 ...b 0。

在这个公式中，2 N 代表量化级别的数量。

直观的是，更多的量化级别会导致原始（模拟）信号的更精确的数字表示。

例如，如果我们可以用1024 个量化级别而不是256 个级别来表示信号，我们就提高了ADC 的精度，因为每个量化级别代表一个更小的幅度范围。

（Vr）ef 表示可以成功转换为精确数字表示的最大输入电压。

因此，重要的是V ref 大于或等于V in的最大值。

但是请记住，比V in值大得多的值将导致表示原始信号的量化级别更少。

例如，如果我们知道我们的信号永远不会增加到 2.4 V 以上，那么使用5 V 的电压参考将是低效的，因为超过一半的量化电平将被使用。

量化误差（Quantisation Error）量化误差是一个术语，用于描述原始信号与信号的离散表示之间的差异。

1. 逻辑综合(Logic Synthesis)EDA工具把数字电路的功能描述(或结构描述)转化为电路的结构描述。

实现上述转换的同时要满足用户给定的约束条件，即速度、功耗、成本等方面的要求。

2. 逻辑电路(Logic Circuit)逻辑电路又称数字电路，在没有特别说明的情况下指的是二值逻辑电路。

其电平在某个阈值之上时看作高电平，在该阈值之下时看作低电平。

通常把高电平看作逻辑值1；把低电平看作逻辑值0。

3. 约束(restriction)设计者给EDA工具提出的附加条件，对逻辑综合而言，约束条件一般包括速度、功耗、成本等方面的要求。

4. 真值表(Truth Table)布尔函数的表格描述形式，描述输入变量每一种组合情况下函数的取值。

输入变量组合以最小项形式表示，函数的取值为真或假(1 或0)。

5. 卡诺图(Karnaugh Map)布尔函数的图形描述形式，图中最小方格和最小项对应，两个相邻的最小方格所对应的最小项只有一个变量的取值不同。

卡诺图适合于用观察法化简布尔函数，但是当变量的个数大于4时，卡诺图的绘制和观察都变得很困难。

6. 单输出函数(Single-output Function)一个布尔函数的单独描述。

7. 多输出函数(Multiple-output Function)输入变量相同的多个布尔函数的统一描述。

8. 最小项(Minterm)设a1,a2,…a i,…a n是n个布尔变量，p为n个因子的乘积。

若是在p中每一变量都以原变量a i或反变量的形式作为因子出现一次且仅出现一次，则称p 为n个变量的一个最小项。

最小项在卡诺图中对应于最小的方格；在立方体表示中对应于极点。

9. 蕴涵项(Implicant)布尔函数f的"与-或"表达式中的每一乘积项都叫作f的蕴涵项。

例如：f=+中的乘积项和都是函数f的蕴涵项。

蕴涵项对应于立方体表示法中的立方体。

10.质蕴涵项(Prime Implicant，PI)设函数f有多个蕴涵项，若某个蕴涵项i所包含的最小项集合不是任何别的蕴涵项所包含的最小项集合的子集的话，则称i为函数f的质蕴涵项。

本答案是英文原版的配套答案，与翻译的中文版课本题序不太一样但内容一样。

翻译的中文版增加了题量。

2.2、Entropy of functions. Let X be a random variable taking on a finite number of values. What is the (general) inequality relationship of ()H X and ()H Y if(a) 2X Y =?(b) cos Y X =?Solution: Let ()y g x =. Then():()()x y g x p y p x ==∑.Consider any set of x ’s that map onto a single y . For this set()():():()()log ()log ()log ()x y g x x y g x p x p x p x p y p y p y ==≤=∑∑,Since log is a monotone increasing function and ():()()()x y g x p x p x p y =≤=∑.Extending this argument to the entire range of X (and Y ), we obtain():()()log ()()log ()x y x g x H X p x p x p x p x =-=-∑∑∑()log ()()yp y p y H Y ≥-=∑,with equality iff g if one-to-one with probability one.(a) 2X Y = is one-to-one and hence the entropy, which is just a function of the probabilities does not change, i.e., ()()H X H Y =.(b) cos Y X =is not necessarily one-to-one. Hence all that we can say is that ()()H X H Y ≥, which equality if cosine is one-to-one on the range of X .2.16. Example of joint entropy. Let (,)p x y be given byFind(a) ()H X ,()H Y .(b) (|)H X Y ,(|)H Y X . (c) (,)H X Y(d) ()(|)H Y H Y X -.(e) (;)I X Y(f) Draw a Venn diagram for the quantities in (a) through (e).Solution:Fig. 1 Venn diagram(a) 231()log log30.918 bits=()323H X H Y =+=.(b) 12(|)(|0)(|1)0.667 bits (/)33H X Y H X Y H X Y H Y X ==+===((,)(|)()p x y p x y p y =)((|)(,)()H X Y H X Y H Y =-)(c) 1(,)3log3 1.585 bits 3H X Y =⨯=(d) ()(|)0.251 bits H Y H Y X -=(e)(;)()(|)0.251 bits=-=I X Y H Y H Y X(f)See Figure 1.2.29 Inequalities. Let X,Y and Z be joint random variables. Prove the following inequalities and find conditions for equality.(a) )ZHYXH≥X(Z()|,|(b) )ZIYXI≥X((Z);,;(c) )XYXHZ≤Z-H-XYH),(,)(((X,,)H(d) )XYIZIZII+-XZY≥Y(););(|;;(Z|)(XSolution:(a)Using the chain rule for conditional entropy,HZYXHZXH+XH≥XYZ=),(|(Z)(||,()|)With equality iff 0YH,that is, when Y is a function of X andXZ,|(=)Z.(b)Using the chain rule for mutual information,ZIXXIZYX+=,I≥IYZ(|;)X);)(,;;(Z)(With equality iff 0ZYI, that is, when Y and Z areX)|;(=conditionally independent given X.(c)Using first the chain rule for entropy and then definition of conditionalmutual information,XZHYHIXZYX==-H-XHYYZ)()(;Z)|,|),|(X(,,)(XHXZH-Z≤,=,)()()(X|HWith equality iff 0ZYI, that is, when Y and Z areX(=|;)conditionally independent given X .(d) Using the chain rule for mutual information,);()|;();,();()|;(Z X I X Y Z I Z Y X I Y Z I Y Z X I +==+And therefore this inequality is actually an equality in all cases.4.5 Entropy rates of Markov chains.(a) Find the entropy rate of the two-state Markov chain with transition matrix⎥⎦⎤⎢⎣⎡--=1010010111p p p p P (b) What values of 01p ,10p maximize the rate of part (a)?(c) Find the entropy rate of the two-state Markov chain with transition matrix⎥⎦⎤⎢⎣⎡-=0 1 1p p P(d) Find the maximum value of the entropy rate of the Markov chain of part (c). We expect that the maximizing value of p should be less than 2/1, since the 0 state permits more information to be generated than the 1 state.Solution:(a) T he stationary distribution is easily calculated.10010*********,p p p p p p +=+=ππ Therefore the entropy rate is10011001011010101012)()()()()|(p p p H p p H p p H p H X X H ++=+=ππ(b) T he entropy rate is at most 1 bit because the process has only two states. This rate can be achieved if( and only if) 2/11001==p p , in which case the process is actually i.i.d. with2/1)1Pr()0Pr(====i i X X .(c) A s a special case of the general two-state Markov chain, the entropy rate is1)()1()()|(1012+=+=p p H H p H X X H ππ.(d) B y straightforward calculus, we find that the maximum value of)(χH of part (c) occurs for 382.02/)53(=-=p . The maximum value isbits 694.0)215()1()(=-=-=H p H p H (wrong!)5.4 Huffman coding. Consider the random variable⎪⎪⎭⎫ ⎝⎛=0.02 0.03 0.04 0.04 0.12 0.26 49.0 7654321x x x x x x x X (a) Find a binary Huffman code for X .(b) Find the expected codelength for this encoding.(c) Find a ternary Huffman code for X .Solution:(a) The Huffman tree for this distribution is(b)The expected length of the codewords for the binary Huffman code is 2.02 bits.( ∑⨯=)()(i p l X E )(c) The ternary Huffman tree is5.9 Optimal code lengths that require one bit above entropy. The source coding theorem shows that the optimal code for a random variable X has an expected length less than 1)(+X H . Given an example of a random variable for which the expected length of the optimal code is close to 1)(+X H , i.e., for any 0>ε, construct a distribution for which the optimal code has ε-+>1)(X H L .Solution: there is a trivial example that requires almost 1 bit above its entropy. Let X be a binary random variable with probability of 1=X close to 1. Then entropy of X is close to 0, but the length of its optimal code is 1 bit, which is almost 1 bit above its entropy.5.25 Shannon code. Consider the following method for generating a code for a random variable X which takes on m values {}m ,,2,1 with probabilities m p p p ,,21. Assume that the probabilities are ordered so thatm p p p ≥≥≥ 21. Define ∑-==11i k i i p F , the sum of the probabilities of allsymbols less than i . Then the codeword for i is the number ]1,0[∈i Frounded off to i l bits, where ⎥⎥⎤⎢⎢⎡=i i p l 1log . (a) Show that the code constructed by this process is prefix-free and the average length satisfies 1)()(+<≤X H L X H .(b) Construct the code for the probability distribution (0.5, 0.25, 0.125, 0.125).Solution:(a) Since ⎥⎥⎤⎢⎢⎡=i i p l 1log , we have 11log 1log +<≤i i i p l pWhich implies that 1)()(+<=≤∑X H l p L X H i i .By the choice of i l , we have )1(22---<≤ii l i l p . Thus j F , i j > differs from j F by at least il -2, and will therefore differ from i F is at least one place in the first i l bits of the binary expansion of i F . Thus thecodeword for j F , i j >, which has length i j l l ≥, differs from thecodeword for i F at least once in the first i l places. Thus no codewordis a prefix of any other codeword.(b) We build the following table3.5 AEP. Let ,,21X X be independent identically distributed random variables drawn according to theprobability mass function {}m x x p ,2,1),(∈. Thus ∏==n i i n x p x x x p 121)(),,,( . We know that)(),,,(log 121X H X X X p n n →- in probability. Let ∏==n i i n x q x x x q 121)(),,,( , where q is another probability mass function on {}m ,2,1.(a) Evaluate ),,,(log 1lim 21n X X X q n-, where ,,21X X are i.i.d. ~ )(x p . Solution: Since the n X X X ,,,21 are i.i.d., so are )(1X q ,)(2X q ,…，)(n X q ，and hence we can apply the strong law of large numbers to obtain∑-=-)(log 1lim ),,,(log 1lim 21i n X q n X X X q n 1..))((log p w X q E -=∑-=)(log )(x q x p∑∑-=)(log )()()(log )(x p x p x q x p x p )()||(p H q p D +=8.1 Preprocessing the output. One is given a communication channel withtransition probabilities )|(x y p and channel capacity );(max )(Y X I C x p =.A helpful statistician preprocesses the output by forming )(_Y g Y =. He claims that this will strictly improve the capacity.(a) Show that he is wrong.(b) Under what condition does he not strictly decrease the capacity? Solution:(a) The statistician calculates )(_Y g Y =. Since _Y Y X →→ forms a Markov chain, we can apply the data processing inequality. Hence for every distribution on x ,);();(_Y X I Y X I ≥. Let )(_x p be the distribution on x that maximizes );(_Y X I . Then__)()()(_)()()();(max );();();(max __C Y X I Y X I Y X I Y X I C x p x p x p x p x p x p ==≥≥===.Thus, the statistician is wrong and processing the output does not increase capacity.(b) We have equality in the above sequence of inequalities only if we have equality in data processing inequality, i.e., for the distribution that maximizes );(_Y X I , we have Y Y X →→_forming a Markov chain.8.3 An addition noise channel. Find the channel capacity of the following discrete memoryless channel:Where {}{}21Pr 0Pr ====a Z Z . The alphabet for x is {}1,0=X . Assume that Z is independent of X . Observe that the channel capacity depends on the value of a . Solution: A sum channel.Z X Y += {}1,0∈X , {}a Z ,0∈We have to distinguish various cases depending on the values of a .0=a In this case, X Y =,and 1);(max =Y X I . Hence the capacity is 1 bitper transmission.1,0≠≠a In this case, Y has four possible values a a +1,,1,0. KnowingY ,we know the X which was sent, and hence 0)|(=Y X H . Hence thecapacity is also 1 bit per transmission.1=a In this case Y has three possible output values, 0,1,2, the channel isidentical to the binary erasure channel, with 21=f . The capacity of this channel is 211=-f bit per transmission.1-=a This is similar to the case when 1=a and the capacity is also 1/2 bit per transmission.8.5 Channel capacity. Consider the discrete memoryless channel)11 (mod Z X Y +=, where ⎪⎪⎭⎫ ⎝⎛=1/3 1/3, 1/3,3 2,,1Z and {}10,,1,0 ∈X . Assume thatZ is independent of X .(a) Find the capacity.(b) What is the maximizing )(*x p ?Solution: The capacity of the channel is );(max )(Y X I C x p =)()()|()()|()();(Z H Y H X Z H Y H X Y H Y H Y X I -=-=-=bits 311log)(log );(=-≤Z H y Y X I , which is obtained when Y has an uniform distribution, which occurs when X has an uniform distribution.(a)The capacity of the channel is bits 311log /transmission.(b) The capacity is achieved by an uniform distribution on the inputs.10,,1,0for 111)( ===i i X p 8.12 Time-varying channels. Consider a time-varying discrete memoryless channel. Let n Y Y Y ,,21 be conditionally independent givenn X X X ,,21 , with conditional distribution given by ∏==ni i i i x y p x y p 1)|()|(.Let ),,(21n X X X X =, ),,(21n Y Y Y Y =. Find );(max )(Y X I x p . Solution:∑∑∑∑∑=====--≤-≤-=-=-=-=ni i n i i i n i i ni i i ni i i n p h X Y H Y H X Y H Y H X Y Y Y H Y H X Y Y Y H Y H X Y H Y H Y X I 111111121))(1()|()()|()(),,|()()|,,()()|()();(With equlity ifnX X X ,,21 is chosen i.i.d. Hence∑=-=ni i x p p h Y X I 1)())(1();(max .10.2 A channel with two independent looks at Y . Let 1Y and 2Y be conditionally independent and conditionally identically distributed givenX .(a) Show );();(2),;(21121Y Y I Y X I Y Y X I -=. (b) Conclude that the capacity of the channelX(Y1,Y2)is less than twice the capacity of the channelXY1Solution:(a) )|,(),(),;(212121X Y Y H Y Y H Y Y X I -=)|()|();()()(212121X Y H X Y H Y Y I Y H Y H ---+=);();(2);();();(2112121Y Y I Y X I Y Y I Y X I Y X I -=-+=(b) The capacity of the single look channel 1Y X → is );(max 1)(1Y X I C x p =.Thecapacityof the channel ),(21Y Y X →is11)(211)(21)(22);(2max );();(2max ),;(max C Y X I Y Y I Y X I Y Y X I C x p x p x p =≤-==10.3 The two-look Gaussian channel. Consider the ordinary Shannon Gaussian channel with two correlated looks at X , i.e., ),(21Y Y Y =, where2211Z X Y Z X Y +=+= with a power constraint P on X , and ),0(~),(221K N Z Z ,where⎥⎦⎤⎢⎣⎡=N N N N K ρρ. Find the capacity C for (a) 1=ρ (b) 0=ρ (c) 1-=ρSolution:It is clear that the two input distribution that maximizes the capacity is),0(~P N X . Evaluating the mutual information for this distribution,),(),()|,(),()|,(),(),;(max 212121212121212Z Z h Y Y h X Z Z h Y Y h X Y Y h Y Y h Y Y X I C -=-=-==Nowsince⎪⎪⎭⎫⎝⎛⎥⎦⎤⎢⎣⎡N N N N N Z Z ,0~),(21ρρ,wehave)1()2log(21)2log(21),(222221ρππ-==N e Kz e Z Z h.Since11Z X Y +=, and22Z X Y +=, wehave ⎪⎪⎭⎫⎝⎛⎥⎦⎤⎢⎣⎡++++N N P N N P N Y Y P P ,0~),(21ρρ, And ))1(2)1(()2log(21)2log(21),(222221ρρππ-+-==PN N e K e Y Y h Y .Hence⎪⎪⎭⎫⎝⎛++=-=)1(21log 21),(),(21212ρN P Z Z h Y Y h C(a) 1=ρ.In this case, ⎪⎭⎫⎝⎛+=N P C 1log 21, which is the capacity of a single look channel.(b) 0=ρ. In this case, ⎪⎭⎫⎝⎛+=N P C 21log 21, which corresponds to using twice the power in a single look. The capacity is the same as the capacity of the channel )(21Y Y X +→.(c) 1-=ρ. In this case, ∞=C , which is not surprising since if we add1Y and 2Y , we can recover X exactly.10.4 Parallel channels and waterfilling. Consider a pair of parallel Gaussianchannels,i.e.,⎪⎪⎭⎫⎝⎛+⎪⎪⎭⎫ ⎝⎛=⎪⎪⎭⎫ ⎝⎛212121Z Z X X Y Y , where⎪⎪⎭⎫ ⎝⎛⎥⎥⎦⎤⎢⎢⎣⎡⎪⎪⎭⎫ ⎝⎛222121 00 ,0~σσN Z Z , And there is a power constraint P X X E 2)(2221≤+. Assume that 2221σσ>. At what power does the channel stop behaving like a single channel with noise variance 22σ, and begin behaving like a pair of channels? Solution: We will put all the signal power into the channel with less noise until the total power of noise+signal in that channel equals the noise power in the other channel. After that, we will split anyadditional power evenly between the two channels. Thus the combined channel begins to behave like a pair of parallel channels when the signal power is equal to the difference of the two noise powers, i.e., when 22212σσ-=P .。

Unit 3Transition to Modern Information ScienceChapter One&Part4 Extensive Reading @Part 1 Notes to Text@Part5Notes to Passage & Part 2 Word Study@Part3 Practice on Text @Part6 Practice on Passage@Part 1 Notes to TextTransition to Modern Information Science1)With the 1950‘s came increasing awareness of the potentialof automatic devices for literature searching and informationstorage and retrieval.随着二十世纪五十年代的来临，人们对用于文献资料搜索、信息储存与检索的自动装置的潜力认识日益增长。

注释：该句是一个完全倒装句。

主语是awareness；介词短语With the 1950‘s是状语，修饰谓语动词came。

2)As these concepts grew in magnitude and potential, so did thevariety of information science interests. 由于这些概念的大量增长，潜移默化，对信息科学研究的各种兴趣也亦如此。

注释：介词短语in magnitude and potential作方式状语，意思是“大量地，潜移默化地”；后面的主句因为so放在句首而倒装。

So指代前文的grew in magnitude and potential。

3) Grateful Med at the National Library of Medicine美国国家医学图书馆数据库注释：Grateful Med是对另一个NLM（国家医学图书馆）基于网络的查询系统的链接。

采样定理的基本内容1. 什么是采样定理采样定理（Sampling Theorem）是数字信号处理中的一个基本理论，也被称为奈奎斯特定理（Nyquist Theorem）或香农定理（Shannon Theorem）。

它描述了如何在连续时间域中对信号进行采样，以便在离散时间域中能够完全还原原始信号。

2. 采样定理的基本原理采样定理的基本原理是：当一个信号的带宽不超过采样频率的一半时，我们可以通过对信号进行采样并以一定的频率进行重建，从而完整地恢复原始信号。

3. 采样定理的数学表达采样定理可以用数学方式表达如下： - 一个信号的最高频率为B，则采样频率Fs 应满足Fs > 2B，即采样频率必须是信号最高频率的2倍以上。

- 采样频率过低会导致混叠现象，也称为折叠现象（Aliasing），即原始信号的高频部分在采样后被混叠到低频部分。

- 采样频率过高不会引起混叠现象，但会浪费存储和计算资源。

4. 采样定理的应用采样定理在数字信号处理中有着广泛的应用，包括但不限于以下几个方面：4.1 通信系统在通信系统中，采样定理保证了信号的完整传输。

发送端将模拟信号进行采样，并通过数字信号处理技术将其转换为数字信号，然后通过传输介质传输到接收端。

接收端将数字信号还原为模拟信号，以便接收者能够恢复原始信息。

4.2 数字音频在数字音频领域，采样定理被广泛应用于音频录制和播放。

音频信号在录制过程中通过模拟转换器（ADC）进行采样，并以数字形式存储。

在播放过程中，数字音频通过数字转换器（DAC）转换为模拟信号，以便音箱或耳机能够播放出声音。

4.3 数字图像在数字图像处理中，采样定理被用于图像的采集和显示。

采样定理保证了图像的细节在数字化过程中不会丢失。

图像传感器将连续的光信号转换为数字图像，然后在显示器上以像素的形式显示出来。

4.4 数据压缩采样定理对数据压缩也有重要意义。

在信号的采样过程中，我们可以通过降低采样频率来减少数据量，从而实现信号的压缩。