76,99 €
Mehr erfahren.
- Herausgeber: John Wiley & Sons
- Kategorie: Wissenschaft und neue Technologien
- Serie: Wiley - IEEE
- Sprache: Englisch
Joint Source-Channel Coding
Consolidating knowledge on Joint Source-Channel Coding (JSCC), this book provides an indispensable resource on a key area of performance enhancement for communications networks
Presenting in one volume the key theories, concepts and important developments in the area of Joint Source-Channel Coding (JSCC), this book provides the fundamental material needed to enhance the performance of digital and wireless communication systems and networks.
It comprehensively introduces JSCC technologies for communications systems, including coding and decoding algorithms, and emerging applications of JSCC in current wireless communications. The book covers the full range of theoretical and technical areas before concluding with a section considering recent applications and emerging designs for JSCC. A methodical reference for academic and industrial researchers, development engineers, system engineers, system architects and software engineers, this book:
- Explains how JSCC leads to high performance in communication systems and networks
- Consolidates key material from multiple disparate sources
- Is an ideal reference for graduate-level courses on digital or wireless communications, as well as courses on information theory
- Targets professionals involved with digital and wireless communications and networking systems
Sie lesen das E-Book in den Legimi-Apps auf:
Seitenzahl: 735
Veröffentlichungsjahr: 2022
Ähnliche
Table of Contents
Cover
Title Page
Copyright
Dedication
Preface
1 Introduction and Background
1.1 Simplified Model for a Communication System
1.2 Entropy and Information
1.3 Introduction to Source Coding
1.4 Channels, Channel Coding, and Capacity
1.5 Layered Model for a Communication System
1.6 Distortion, Quality of Service, and Quality of Experience
1.7 Shannon's Separation Principle and Joint Source–Channel Coding
1.8 Major Classes of Joint Source–Channel Coding Techniques
References
2 Source Coding and Signal Compression
2.1 Types of Sources
2.2 Lossless Compression
2.3 Lossy Compression
2.4 Embedded and Layered Coding
2.5 Coding of Practical Sources
References
3 Channel Coding
3.1 Linear Block Codes
3.2 Convolutional Codes
3.3 Modified Linear Codes (Puncturing, Shortening, Expurgating, Extending, Augmenting, and Lengthening)
3.4 Rate‐Compatible Channel Codes
References
4 Concatenated Joint Source–Channel Coding
4.1 Concatenated JSCC Bit Rate Allocation
4.2 Performance Characterization
4.3 Application Cases
References
5 Unequal Error Protection Source–Channel Coding
5.1 Effect of Channel Errors on Source Encoded Data
5.2 Priority Encoding Transmission Schemes for Unequal Loss Protection
5.3 Dynamic Programming Algorithm for Optimal UEP
5.4 Unequal Error Protection Using Digital Fountain Codes
References
Notes
6 Source–Channel Coding with Feedback
6.1 Joint Source–Channel Coding Formulation for a System with ACK/NACK Feedback
6.2 Packet Combining for Joint Source–Channel ARQ over Memoryless Channels
6.3 Pruned Tree‐Structured Quantization in Noise and Feedback
6.4 Delay‐Constrained JSCC Using Incremental Redundancy with Feedback
References
Note
7 Quantizers Designed for Noisy Channels
7.1 Channel‐Optimized Quantizers
7.2 Scalar Quantizer Design
7.3 Vector Quantizer Design
7.4 Channel Mismatch Considerations
7.5 Structured Vector Quantizers
References
8 Error‐Resilient Source Coding
8.1 Multiple‐Description Coding
8.2 Error‐Resilient Coded Bit Streams
References
9 Analog and Hybrid Digital–Analog JSCC Techniques
9.1 Analog Joint Source–Channel Coding Techniques
9.2 Hybrid Digital–Analog JSCC Techniques
References
10 Joint Source–Channel Decoding
10.1 Source‐Controlled Channel Decoding
10.2 Exploiting Residual Redundancy at the Decoder
10.3 Iterative Source–Channel Decoding
References
11 Recent Applications and Emerging Designs in Source–Channel Coding
11.1 Source–Channel Coding for Wireless Sensor Networks
11.2 Extending Network Capacity Through JSCC
11.3 Source–Channel Coding and Cognitive Radios
11.4 Design of JSCC Schemes Based on Artificial Neural Networks
References
Index
End User License Agreement
List of Tables
Chapter 5
Table 5.1 Fraction of packets needed for segment reception as reported in [...
Chapter 6
Table 6.1 Transition probabilities of the derived discrete channel for diff...
Chapter 7
Table 7.1 Four‐bit natural binary code (NBC), folded binary code (FBC), and...
Table 7.2 Scalar quantizer performance results (SNR in dB) for a zero‐mean ...
Table 7.3 Scalar quantizer performance results (SNR in dB) for a zero‐mean u...
Table 7.4 Scalar quantizer performance results (SNR in dB) for a zero‐mean u...
Table 7.5 The mean squared channel distortion normalized to the BER for the...
Table 7.6 PSNR results for
grayscale Lenna image coded with CPVQs under c...
Chapter 8
Table 8.1 A Hamming weight variable length code and its equivalent reversib...
Table 8.2 Golomb–Rice code and its equivalent reversible variable length co...
Chapter 9
Table 9.1 Correspondences between communication systems components and geom...
List of Illustrations
Chapter 1
Figure 1.1 A block diagram of a general communication system.
Figure 1.2 Illustration of the sampling process.
Figure 1.3 The spectrum of a sampled signal.
Figure 1.4 Quantization of the five left‐most samples in Figure 1.2.
Figure 1.5 Rate‐distortion function for binary sources following a Bernoulli...
Figure 1.6 (a) Rate‐distortion functions and (b) distortion‐rate functions f...
Figure 1.7 (a) The binary symmetric channel and (b) the binary erasure chann...
Figure 1.8 Distilled view of the message communication process.
Figure 1.9 The Internet protocol stack consisting of five layers.
Figure 1.10 5G‐NR User Plane stack as an example of layered architecture.
Figure 1.11 Block diagram for speech MOS estimation using the ITU‐T recommen...
Figure 1.12 Conversion from
R
‐factor into MOS.
Figure 1.13 Distilled view of the message communication process with joint s...
Figure 1.14 The Optimal Performance Theoretically Attainable (OPTA) curve fo...
Chapter 2
Figure 2.1 A speech signal as an example of a continuous source.
Figure 2.2 The probability density function of a speech source sample.
Figure 2.3 Source types.
Figure 2.4 Example design of a Huffman code.
Figure 2.5 Example arithmetic encoding.
Figure 2.6 Predictive coding; the insert illustrates the prediction of one s...
Figure 2.7 Histograms for (a) source output and (b) error signal.
Figure 2.8 A uniform scalar quantizer.
Figure 2.9 Example of the reconstruction levels of a scalar quantizer for tw...
Figure 2.10 A two‐dimensional vector quantizer.
Figure 2.11 Voronoi regions on a two‐dimensional space.
Figure 2.12 Differential encoder and decoder.
Figure 2.13 An example of coding through source sample transformation
Figure 2.14 Ideal subband codec.
Figure 2.15 Two‐level wavelet analysis and the resulting transfer functions ...
Figure 2.16 Transfer functions resulting from the cascaded filters (cascaded...
Figure 2.17 Embedded coding.
Figure 2.18 A three‐layer encoder and decoder.
Figure 2.19 Simplified block diagram showing the main operations in JPEG ima...
Figure 2.20 (a) An 8‐by‐8 pixel image with smooth variations and (b) the cor...
Figure 2.21 (a) An 8‐by‐8 pixel image with sharp variations and (b) the corr...
Figure 2.22 Converting the array of quantized DCT coefficients into a sequen...
Figure 2.23 Two‐level decomposition of an image into wavelet coefficients.
Figure 2.24 Parent–children tree relationships between wavelet coefficients....
Figure 2.25 Three consecutive video frames for the “Foreman” video sequence ...
Figure 2.26 Simplified block diagram for the H.264 video encoder. The blocks...
Figure 2.27 Number of bits used to encode each frame in the
quarter common i
...
Figure 2.28 Simplified block diagram for the MPEG‐4 fine‐grained scalable (F...
Figure 2.29 Mean‐squared error as a function of encoding bits for four frame...
Figure 2.30 Linear predictive coding (LPC) model of speech synthesis.
Figure 2.31 Analysis‐by‐synthesis encoder structure.
Chapter 3
Figure 3.1 An
linear binary block channel encoder.
Figure 3.2 Visual representation of the mapping operation at the encoder of ...
Figure 3.3 Systematic block encoder.
Figure 3.4 Illustration of the use of Reed–Solomon codes to recover from pac...
Figure 3.5 An example memory two convolutional encoder.
Figure 3.6 The state transition diagram corresponding to the encoder in Figu...
Figure 3.7 The trellis diagram corresponding to the possible state transitio...
Figure 3.8 Example Viterbi decoding algorithm metrics from the trellis in Fi...
Figure 3.9 Rate‐compatible punctured convolutional (RCPC) channel code examp...
Figure 3.10 Performance of different codes from the same family of RCPC code...
Figure 3.11 Free distance as a function of channel coding rate
for two fam...
Figure 3.12 Number of error events with Hamming weight
as a function of ch...
Chapter 4
Figure 4.1 Block diagram of a tandem source and channel coding scheme.
Figure 4.2 A typical end‐to‐end distortion curve as a function of the channe...
Figure 4.3 The single‐mode D‐SNR curves and the resulting D‐SNR curve.
Figure 4.4 The single‐mode D‐SNR curves and the resulting D‐SNR curve for a ...
Figure 4.5 The single‐mode D‐SNR curves and the resulting D‐SNR curve for a ...
Figure 4.6 Illustration of points from a single‐mode D‐SNR curve most likely...
Figure 4.7 Modeling the D‐SNR curve in systems with different RS (top) and R...
Figure 4.8 The single‐mode D‐SNR curves, the D‐SNR curve, and the characteri...
Figure 4.9 The single‐mode D‐SNR curves, the D‐SNR curve, and the characteri...
Figure 4.10 Comparison among different concatenated JSCC schemes and the min...
Figure 4.11 Comparing
and
for systems using source codecs with different...
Chapter 5
Figure 5.1 The effect of channel errors on a differentially encoded first‐or...
Figure 5.2 The effect of channel errors on a differentially encoded first‐or...
Figure 5.3 Simplified block diagram for the H.264 video encoder, showing the...
Figure 5.4
(a)
Variance of difference frame
as a single error event is intr...
Figure 5.5 Original image to be encoded and decoded using a simplified JPEG ...
Figure 5.6 (Left) Recovered image from Figure 5.5, after encoding and decodi...
Figure 5.7 The recovered image from Figure 5.5, after encoding and decoding ...
Figure 5.8 The recovered image from Figure 5.5, after encoding and decoding ...
Figure 5.9 Main processing blocks in a typical digital communication system,...
Figure 5.10 Priority Encoding Transmission configured for transmission of an...
Figure 5.11 Effect of losing the third packet during transmission of the PET...
Figure 5.12 Trellis for maximizing the performance for arbitrary
.
Figure 5.13 Trellis for maximizing the average useful source coding rate.
Figure 5.14 Progressive transmission with two policies. The shaded area repr...
Figure 5.15 Optimal progressive transmission of five source packets; the num...
Figure 5.16 Average PSNR performance of UEP over memoryless channels for the...
Figure 5.17 The loss of PSNR in EEP schemes and optimal UEP scheme maximizin...
Figure 5.18 Average PSNR performance of UEP over memoryless channels for the...
Figure 5.19 The loss of PSNR in EEP schemes and optimal UEP scheme maximizin...
Figure 5.20 Average PSNR performance of unequal error protection for memoryl...
Figure 5.21 The loss of PSNR in EEP schemes and optimal UEP scheme maximizin...
Figure 5.22 Average PSNR performance of EEP and the optimal UEP scheme for t...
Figure 5.23 Average PSNR gain of the optimal UEP scheme over equal erasure p...
Figure 5.24 The decoding steps of a fountain code with
,
using a belief p...
Figure 5.25 A system using UEP fountain codes for transmitting a multimedia ...
Figure 5.26 Reorganization of source bits from blocks into equal size segmen...
Figure 5.27 Performance of generalized UEP fountain codes with the
Goldhill
...
Figure 5.28 The Goldhill test image. Source: Robin Weaver/Alamy Images.
Chapter 6
Figure 6.1 General JSCC system with ACK/NACK feedback at the
th step in tra...
Figure 6.2 Active encoder at
th step.
Figure 6.3 System with incremental redundancy transmission, e.g. using RCPC ...
Figure 6.4 Passive Encoder or Type I hybrid ARQ transmitter for any step.
Figure 6.5 Code combining or packet combining.
Figure 6.6 Code combining or packet combining with state estimation.
Figure 6.7 Feedback generation with state estimation.
Figure 6.8 Receiver for baseline CRC‐based system.
Figure 6.9 Receiver for CRC‐based system with pseudo‐MMSE decoding.
Figure 6.10 Receiver for CRC‐based system with list decoding.
Figure 6.11 Discrete two‐input three‐output channel is obtained as BPSK over...
Figure 6.12 Performance (total SNR vs. transmission rate) of various schemes...
Figure 6.13 Performance (total SNR vs. transmission rate) of various schemes...
Figure 6.14 Performance (total SNR vs. transmission rate) of high‐rate CRC‐b...
Figure 6.15 Channel Distortion for Various Schemes, IID Gaussian source, dim...
Figure 6.16 Performance comparison with zero‐redundancy BER‐based schemes. G...
Figure 6.17 Performance comparison with zero‐redundancy BER‐based schemes. G...
Figure 6.18 TSVQ and Pruned TSVQ
Figure 6.19 Feedback generation rule over a full TSVQ and equivalent pruned ...
Figure 6.20 Block diagram of the mobile terminal for scheme implementing del...
Figure 6.21 Example transmission sequence for the JSCC protocol using increm...
Figure 6.22 Markov chain of codetypes.
Figure 6.23 Normalized distortion vs. channel SNR for AWGN channels with and...
Figure 6.24 Comparison of robustness under channel mismatch conditions.
Figure 6.25 Smoothness of performance with respect to data rate/ packet size...
Figure 6.26 Outage probability in a CDMA system with no FER constraint.
Figure 6.27 Outage probability in a CDMA system with residual FER constraint...
Chapter 7
Figure 7.1 Block diagram of the system to study scalar quantizer design for ...
Figure 7.2 (a) Sorted segments for the encoding partition and their upper an...
Figure 7.3 Encoder and decoder design for a range
of the crossover probabi...
Figure 7.4 Work flow for all the components of the vector quantizer design f...
Figure 7.5 Communication system with channel‐optimized vector quantizer and ...
Figure 7.6 Quality (measured as PSNR in dB) for the Lena image coded with a ...
Chapter 8
Figure 8.1 A dual‐description encoder and decoder.
Figure 8.2 The distortion associated with a symmetric dual‐description codec...
Figure 8.3 The design of a multiple‐description scalar quantizer: (a) a four...
Figure 8.4 An alternative multiple‐description scalar quantization design wi...
Figure 8.5 Example of a sequence of frames configured to implement video red...
Figure 8.6 Multistate stereo‐multiple‐description coding.
Figure 8.7 Spatial scaling stereo‐multiple‐description coding.
Figure 8.8 Block diagram to implement multiple‐description coding from corre...
Figure 8.9 (a) Relation between coding redundancy
and correlation angle
....
Figure 8.10 Side decoder distortion as a function of the coding redundancy
Figure 8.11 The transcoding operation in the multiple‐description source cod...
Figure 8.12 Illustration of the use of resynchronization markers and reversi...
Figure 8.13 Example of the operation of the EREC bit allocation algorithm.
Chapter 9
Figure 9.1 A mapping from a single‐dimensional source space to a two‐dimensi...
Figure 9.2 OPTA curves for different channel to source bandwidth ratios,
....
Figure 9.3 The probability density function for a two‐dimensional circularly...
Figure 9.4 Double Archimedes' spiral used for a 2 : 1 dimensionality reducti...
Figure 9.5 Exact distance along a branch of the double Archimedes' spiral an...
Figure 9.6 A fully connected feed‐forward artificial neural network with two...
Figure 9.7 An analog joint source–channel coder implemented with an autoenco...
Figure 9.8 Hybrid digital–analog JSCC through signal decomposition and mappi...
Figure 9.9 Hybrid digital–analog joint source–channel encoder for images.
Figure 9.10 Mapping
, from two components with nine possible values each to...
Figure 9.11 A hybrid digital–analog joint source–channel coder implemented w...
Chapter 10
Figure 10.1 Main processing blocks in a typical digital communication system...
Figure 10.2 The log‐likelihood of a binary random variable as a function of ...
Figure 10.3 Approximate bit error rate performance for a memory 4, rate 1/2,...
Figure 10.4 The convolutional encoder with memory 4 used in the results show...
Figure 10.5 Trellis for the decoding of a rate 1/2 convolutional code, showi...
Figure 10.6 Value of the most significant bit for the DCT DC coefficient in ...
Figure 10.7 Value of the most significant bit for the DCT coefficient with c...
Figure 10.8 Approximation of the log‐likelihood function, using logarithms i...
Figure 10.9 Peak signal‐to‐noise ratio (PSNR) for the image “Lena” when usin...
Figure 10.10 Concatenation of two convolutional encoders in (a) parallel and...
Figure 10.11 Block diagram for a system doing iterative JSCD. A comparison w...
Figure 10.12 Block diagram for an iterative joint source–channel decoder.
Chapter 11
Figure 11.1 Overview of a wireless sensor network.
Figure 11.2 Overview of a distributed source coding (DSC) scheme.
Figure 11.3 DSC using syndrome calculation.
Figure 11.4 A Wyner–Ziv codec.
Figure 11.5 Block diagram of the system that relies on a concatenated source...
Figure 11.6 Distortion‐rate performance for different video frames of the vi...
Figure 11.7 Distortion‐rate performance for different video frames of the vi...
Figure 11.8 Total optimal equivalent bandwidth
as a function of
.
Figure 11.9 Markov chain representation of the
traffic model.
Figure 11.10 Comparison of three CDMA systems supporting video calls.
Figure 11.11 A frame from the sequence “Foreman” when the network operates w...
Figure 11.12 A frame from the sequence “Foreman” when the network operates w...
Figure 11.13 A frame from the sequence “Foreman” when the network operates w...
Figure 11.14 A frame from the sequence “'Foreman” when the network operates ...
Figure 11.15 A frame from the sequence “Foreman” when the network operates w...
Figure 11.16 A frame from the sequence “Foreman” when the network operates w...
Figure 11.17 Expected distortion (PSNR in dB) as a function of the offered l...
Figure 11.18 Probability of pseudo‐congestion as a function of the offered l...
Figure 11.19 Expected distortion (PSNR in dB) as a function of probability o...
Figure 11.20 Blocking probability as a function of the offered load.
Figure 11.21 A typical system architecture for a wireless device, including ...
Figure 11.22 A
‐table as a function of action space indexes for a given sta...
Figure 11.23 Average distortion as a function of the number of SUs in the ne...
Figure 11.24 Congestion rate as a function of the number of SUs in the netwo...
Figure 11.25 Average number of cycles to achieve convergence.
Figure 11.26 Block diagram abstracting the JSCC system with the model for th...
Guide
Cover
Table of Contents
Title Page
Copyright
Dedication
Preface
Begin Reading
Index
End User License Agreement
Pages
iii
iv
v
xi
xii
xiii
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
221
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
381
382
383
384
385
386
Joint Source‐Channel Coding
Andres KwasinskiRochester Institute of TechnologyRochester, NY, USA
Vinay ChandeQualcomm Technologies Inc.San Diego, CA, USA
This edition first published 2023© 2023 John Wiley & Sons Ltd
All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, except as permitted by law. Advice on how to obtain permission to reuse material from this title is available at http://www.wiley.com/go/permissions.
The right of Andres Kwasinski and Vinay Chande to be identified as the authors of this work has been asserted in accordance with law.
Registered OfficesJohn Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, USAJohn Wiley & Sons Ltd, The Atrium, Southern Gate, Chichester, West Sussex, PO19 8SQ, UK
Editorial OfficeThe Atrium, Southern Gate, Chichester, West Sussex, PO19 8SQ, UK
For details of our global editorial offices, customer services, and more information about Wiley products visit us at www.wiley.com.
Wiley also publishes its books in a variety of electronic formats and by print‐on‐demand. Some content that appears in standard print versions of this book may not be available in other formats.
Trademarks: Wiley and the Wiley logo are trademarks or registered trademarks of John Wiley & Sons, Inc. and/or its affiliates in the United States and other countries and may not be used without written permission. All other trademarks are the property of their respective owners. John Wiley & Sons, Inc. is not associated with any product or vendor mentioned in this book.
Limit of Liability/Disclaimer of WarrantyWhile the publisher and authors have used their best efforts in preparing this work, they make no representations or warranties with respect to the accuracy or completeness of the contents of this work and specifically disclaim all warranties, including without limitation any implied warranties of merchantability or fitness for a particular purpose. No warranty may be created or extended by sales representatives, written sales materials or promotional statements for this work. The fact that an organization, website, or product is referred to in this work as a citation and/or potential source of further information does not mean that the publisher and authors endorse the information or services the organization, website, or product may provide or recommendations it may make. This work is sold with the understanding that the publisher is not engaged in rendering professional services. The advice and strategies contained herein may not be suitable for your situation. You should consult with a specialist where appropriate. Further, readers should be aware that websites listed in this work may have changed or disappeared between when this work was written and when it is read. Neither the publisher nor authors shall be liable for any loss of profit or any other commercial damages, including but not limited to special, incidental, consequential, or other damages.
Library of Congress Cataloging‐in‐Publication Data
Names: Kwasinski, Andres, author. | Chande, Vinay, 1972- author.Title: Joint source-channel coding / Andres Kwasinski, Rochester Institute of Technology, Rochester, NY, USA, Vinay Chande, Qualcomm Technologies Inc., San Diego, CA, USA.Description: First edition. | Hoboken, NJ, USA : Wiley-IEEE Press, 2023. | Series: IEEE Press | Includes bibliographical references and index.Identifiers: LCCN 2022036505 (print) | LCCN 2022036506 (ebook) | ISBN 9781119978527 (hardback) | ISBN 9781118693773 (adobe pdf) | ISBN 9781118693797 (epub)Subjects: LCSH: Combined source channel coding.Classification: LCC TK5102.93 .K89 2023 (print) | LCC TK5102.93 (ebook) | DDC 621.382–dc23/eng/20220930LC record available at https://lccn.loc.gov/2022036505LC ebook record available at https://lccn.loc.gov/2022036506
Cover Design: WileyCover Images: © Andrea Danti/Shutterstock, Tatiana Popova/Shutterstock
To our mentors, role models, and alma maters – our Gurujans and Gurukuls.—AK and VC
Preface
Claude Shannon's seminal work in 1948 – “A Mathematical Theory of Communication” – gave birth to the field of Information Theory and introduced foundational deep ideas that allowed to model and understand the process of communicating information reliably between a transmitter and a receiver. The most fundamental ideas introduced by Shannon were “coding” of an information source to obtain compact representations (“source encoding”) and coding for transmission in a form tailored to the communication channel characteristics (“channel encoding”). An astonishing set of results established that, for an information source, the information content can be measured as the surprise/uncertainty that is associated with the generated message. Further, communication over noisy channels with increasing reliability need not involve a diminishing rate of information transfer ‐ that is, asymptotically error‐free communication is possible over the channel using source and channel coding, provided the rate of information transfer does not exceed the “capacity” of the channel. The notion of capacity, along with the characteristics of the medium, also captures resource constraints, for example power and bandwidth.
Shannon's ideas were revolutionary in sparking the digital age that we live in (along with Shannon's Information Theory, his master's thesis, “A Symbolic Analysis of Relay and Switching Circuits,” also played a large part there). Since Shannon's breakthrough work, multiple generations of communication engineers have learned these important ideas and technologies via textbooks on information theory, source coding and data compression, signal processing, and error control coding. In his paper, Shannon drew a diagram for an end‐to‐end communication system envisioned as a cascade of information source, source encoding, and channel encoding at the transmitter side leading to a communication channel. At the receiver side, the processing blocks followed the stages of channel decoding, source decoding, and finally source reproduction. These building blocks essentially continue to make up the modern communication systems. Modern‐day end‐to‐end communication systems are intricate, multilayered architectures, with many devices, entities, and software and hardware components operating over diverse physical media, with heterogeneous information sources and sinks as termination points within them. For tractability, the system designers have preferred to take a modular approach where the design considerations for source coding may, physically and logically, be far removed from the aspects of channel coding.
As a consequence, it has been natural that source coding and channel coding got treated separately in dedicated textbooks or in textbooks of Information Theory. In the textbooks on communications systems, they may get treated as components of an end‐to‐end system, but with an understanding that each component needs to be designed and optimized independently. This approach is further rooted in Shannon's “separation theorem” which stated that in many cases of interest, coding can be done in separate source coding and channel coding stages, independently, without loss of optimality in rate. This optimality, though, is established in an asymptotic sense of increasing delays and coding/decoding complexity. However, it is recognized that blind and separate designs of various components, including source and channel coding, may have large inefficiencies in practice, especially under practical non‐asymptotic limitations for delay and complexity. These inefficiencies can be reduced by paying attention to both components jointly. This recognition is not new though. Over the years, considerable work has been done by researchers that treated the source and the channel coding stages jointly at the transmitter as well as at the receiver. This field of investigation is collectively known as joint source–channel coding (JSCC). The works range from information theoretic studies of the validity of separation theorems to highly concrete techniques designed and applied to systems of various levels of abstractions and practicality. Yet, even as the JSCC approach keeps growing in relevance and seeing increased attention, to this day, there is not a textbook that is dedicated to presenting JSCC in a comprehensive way, leaving those willing to work on JSCC with the only option to learn by searching and reading through dozens of technical papers that have been published in the span of decades. The main motivation for this textbook is to fill this gap. Here, our aim is threefold. Firstly, we aim at presenting a fairly rich introduction to the topic, with key refreshers needed to initiate research. Secondly, we wish to provide the reader with a compendium of different classical and modern variants and approaches to JSCC. Given the vast amount of published literature over decades, this is a tall aim and we do not claim to conquer it even remotely. Yet, although the book borrows heavily from the research work of the authors, we believe that it will serve to provide broad coverage and treatment of JSCC. Thirdly, we see the book as a recipe book or procedure book, where ideas and formulations used in one context can be cross‐pollinated with others or used in new contexts by the reader. As such, we intend that for newcomers in the area of JSCC, this book will serve them not only for their initiation but also to prepare them to make new contributions right away. For engineers that are already working on JSCC, we aim to provide a holistic presentation that, we hope, will spark ideas for new techniques.
To serve these objectives, the book starts with Chapters 1 through 3 that are dedicated to discussing background in communication systems, Information Theory, source coding, and channel coding, as well as to explaining the reason to use JSCC and to present the main classes of JSCC techniques to be covered later in the book. Following this introductory part, Chapter 4 delves into concatenated joint source–channel coding, as a first and basic JSCC technique. Chapter 4 is followed by six chapters where each one is dedicated to an specific approach to JSCC: Chapter 5 covers unequal error protection source–channel coding, Chapter 6 focuses on source–channel coding with feedback, Chapter 7 is dedicated to the study of the quantizer design for noisy channels, Chapter 8 covers error‐resilient source coding, Chapter 9 focuses on analog and hybrid digital–analog JSCC techniques, and Chapter 10 discusses joint source–channel decoding. The book concludes in Chapter 11 with a presentation of some recent applications of JSCC, connecting the solutions with techniques seen in earlier chapters, and discussing the emerging and promising design approach for JSCC based on artificial neural networks.
This book has been written over many years of effort. We are grateful to all the staff at Wiley that has supported us in this long road. The book, as you can see it now in a finished form, reflects the investment of the most valuable of resources: time. Acknowledging that time is a finite resource that cannot be regenerated or stretched, we would like to express our deepest gratitude to colleagues, friends, and family. Without their support this book would not have been possible.
2022
Andres Kwasinski
Rochester Institute of Technology
Rochester, NY, USA
Vinay Chande
Qualcomm Technologies Inc
.
San Diego, CA, USA
1Introduction and Background
This textbook is about jointly performing source and channel coding for an information source. The idea of coding is arguably the most significant contribution from Claude Shannon in his mathematical theory of communication [1] that gave birth to information theory. Up until Shannon's work, communication engineers believed that to improve reliability in transmitting a message, all they could do was to increase the transmit power or repeat the transmission many times until it was received free of errors. Instead, Shannon postulated that a message could be transmitted free of errors when not exceeding a capacity for the communication channel. All that was required to achieve this feat was to use a suitable coding mechanism. With coding, the message from the source is mapped into a signal that is matched to the channel characteristics. The ideal coding operation is able to both represent the message from the source in the most efficient form (removing redundant information) and add redundant information in a controlled way to enable the receiver to combat errors introduced in the channel. The efficient representation of the source message is called source coding, and the controlled introduction of redundant information to combat communication errors is called channel coding.
In [1], Shannon showed that under some ideal conditions, for example, for point‐to‐point “memoryless” channels, asymptotically in the codeword length, there is no performance difference whether the coding operation is performed as a single block or as a concatenation of a source coding operation and a separate channel coding operation. This is typically described as a separation theorem. The separation theorems are a class of asymptotic results of profound importance. It may not be an exaggeration to say that those forms of results motivated the development of today's digital revolution, where every form of information – media, gaming, text, video, multidimensional graphics, and data of all forms – can be encoded without worrying about the medium or channel where it will be shared, and shared without the concern for how it was encoded. Nevertheless, there are multiple practical scenarios where the ideal conditions given by Shannon do not hold. In these cases, we pay a performance penalty for doing the source and channel coding as two separate operations instead of jointly doing source and channel coding. This book focuses on some of these scenarios and helps the reader study the theory and techniques associated with performing source and channel coding as a single operation. Before delving into these topics, in this chapter, we provide an introduction and needed background for the rest of this book.
1.1 Simplified Model for a Communication System
A communication system enables transmission of information so that a message that originates at one place is reproduced exactly, or approximately, at another location. To develop his mathematical theory of communication [1, 2], Claude Shannon considered a simplified model for a communication system, as shown in Figure 1.1. In this model, a communication system is formed by five basic elements:
An
information source
, which generates a message that contains some information to be transmitted to the receiving end of the communication chain
A
transmitter
, which converts the message into a signal that can propagate through a communication medium
A
channel
, which constitutes the medium through which the signal propagates between the transmitter and receiver. This propagating signal is subject to different distortions and impairments, such as selective attenuation, delays, erasure, and the addition of noise.
A
receiver
, which attempts to recover the original message (possibly affected by distortion and impairments) by reversing the sequence of operations done at the transmitter while also attempting to correct or compensate for the distortion effects introduced in the channel
A
destination
, which receives the transmitted version of the message and makes sense of it.
The design of a communication system focuses on the transmitter and receiver. By designing the transmitter output to match the channel characteristics and requirements, the transmitter converts, or maps, the source message into a signal appropriate for transmission. This mapping operation is called encoding. The reverse operation, where a source message is estimated from the encoded signal, is called decoding. For reasons that will be seen later, the mapping operation at the encoder is frequently divided into two stages. The first stage, called source encoding, aims to represent the source output in a compact and efficient way that will require as few communication resources as possible while achieving some level of fidelity for the source message recovered after source decoding at the receiver. The compact and efficient representation that results from source encoding is matched to the source but is likely not matched to the channel characteristics and requirements. For example, the representation may be so compact that each of its parts may be critical for the recovery of the source message at the required level of fidelity, so any error introduced during transmission renders the received message completely unrecoverable. Because of this, the second stage of the transmitter, called channel encoding, has the function of converting the compact source representation into a signal matched to the channel. This likely results in a representation that is less compact but more resilient to the effects of channel impairments. On the receiver side, it is natural to think of the structure also separated into a sequence of decoding stages because the goal is to reverse the sequence of encoding operations that were performed at the transmitter.
Figure 1.1 A block diagram of a general communication system.
Designing a communication system entails designing the transmitter and thus the mapper from the source message to the channel signal. Would it be better to design the mapping as a single operation from source directly to the channel? Or, would it be better to design the mapping as a first source encoding stage followed by the channel encoding stage? Is there any performance advantage with either approach? One answer to these questions is given in an asymptotic sense by Shannon's source–channel separation theorem, which states that under some conditions, there is no difference in performance between a transmitter designed as a single mapper and a transmitter designed as the two cascaded source and channel encoding stages. This result is appealing from a designer's perspective. It appears to be a simpler divide‐and‐conquer approach, involving design of the source and channel coder–decoder pairs (codecs) separately. Nevertheless, the tricky element of Shannon's source–channel separation theorem is that the conditions under which it holds are difficult to find in practical scenarios. To discuss this, we first need to establish some key principles from information theory. The next sections provide an overview of important information theory concepts that will be used throughout the rest of this book.
1.2 Entropy and Information
The concept of information as understood in Shannon's information theory originates in Hartley's paper [3] and provides a measure on how much the uncertainty associated with a random phenomenon (mathematically characterized as the outcome of a random variable) is reduced by the observation of a particular outcome. The information provided by the outcome of a discrete random variable is defined as:
where is the probability of the outcome and the logarithm can be of any base in principle, but it is most frequently taken as base 2 (in which case the unit of information is the bit, a condensed merging of the words binary and digit). Intuitively, the rarer an event is, the more information its occurrence provides.
The notion of information as introduced in (1.1) provides a measure that is related to a specific outcome of a random variable. To measure the uncertainty associated with a random variable, Shannon introduced the concept of entropy (in an information theoretic context) as the expectation of the information function associated with a random variable. Then the entropy of a discrete random variable is defined as:
where is, in this context, the probability mass function (PMF). The entropy of a random variable can be interpreted as the mean value of the information provided by all its outcomes.
In the case of a continuous random variable with probability density function (PDF) , the concept of entropy has been extended to become the differential entropy:
For example, it can be shown that differential entropy of a zero‐mean Gaussian random variable with variance is [4].
By using joint probability distributions, one can extend the definition of entropy to calculate the joint entropy between multiple random variables. In the case of two discrete random variables, and , their joint entropy is
where is the joint PMF and and are marginal PMFs. Similarly, the conditional entropy of , is
where is the conditional PMF of given that . Consequently, the conditional entropy can be regarded as the mean value of the information provided by all the outcomes of a random variable () given that the outcome of a second random variable () is known.
The entropy presents a number of useful properties that in the interest of maintaining the introductory nature of this chapter we enumerate next without a detailed proof:
Entropy is nonnegative:
.
if and only if there exists a value
such that
almost surely (that is, the random variable
behaves as a deterministic constant).
Distributions with maximum entropy:
if the discrete random variable
takes with nonzero probability
different values. Equality,
, is achieved if and only if
is uniformly distributed. For a continuous random variable
with variance
, we have the differential entropy
, with equality achieved when
is a Gaussian random variable.
Subadditivity of joint entropy:
with equality if and only if
and
are statistically independent random variables.
.
Conditional entropy is the remaining uncertainty:
. This property states that conditional entropy measures how much uncertainty about a random variable (
) remains after knowing the outcome of a second random variable (
).
Nonnegativity of conditional entropy:
.
Chain rule for Shannon's entropy and conditional entropy:
In a simpler form: .
Conditioning reduces entropy:
.
Communication is inherently a process relating two or more random variables to each other (e.g. the input and output of a channel, an uncompressed and a compressed representation of a signal). A quantity that can measure the closeness of two random variables in terms of information shared between them is the mutual information. For two discrete random variables and , it is defined as:
Following Bayes's theorem (), the mutual information can also be written as
We can also write
Note that the first term in this result is the entropy of the random variable and the second term can be written in terms of the conditional entropy of . Therefore, the mutual information as in (1.8) can now be written as:
This quantity has an intuitive interpretation as follows. If was the measure of uncertainty in random variable and is the remaining uncertainty in on observing (i.e. on learning of the outcome about) , then is the amount of uncertainty reduction about on observing . , therefore, is the information about contained in or shared by . By its definition, it is seen that the quantity is symmetric (i.e. ). Therefore, it is called Mutual Information between two random variables. The analogous quantity of mutual information can be defined for continuous random variables too – by replacing the notion of entropy and conditional entropy by differential entropy and conditional differential entropy, respectively.
Analogous to the introduction of conditional entropy, we can also think of a conditional mutual information between two random variables and given that the outcome of a random variable is known. This conditional mutual information, denoted , is defined as:
The conditional mutual information can be understood as being the average amount of information shared by two random variables after a third random variable is known. In this case, the third random variable is often interpreted as representing side information that has been revealed through some process. The conditional mutual information allows for the definition of a chain rule to calculate the mutual information between a random variable and two other random variables and :
or in a more general case when having random variables ,
1.3 Introduction to Source Coding
The communication process starts with a source generating a message intended for transmission. The message may be of very different types. For example, it may be a text (a sequence of letters and symbols), a file residing in a computer, or somebody speaking. In all cases, the message is just a container of information that needs to be transmitted. Messages of all types usually need to go through a step that efficiently represents the information contained in the message. Further, transmission of a message through a communication system as conceptualized by Shannon requires a few extra steps when the message is formed by a continuous‐time signal. All these steps are collectively called “source coding.” We provide next a brief overview on source coding, leaving for Chapter 2 to cover more details.
1.3.1 Sampling and Quantization of Signals
Today's “digital” age derives its name from the revolution brought about by digital representation of information – which permits unprecedented flexibility in storage, reproduction, computing, and communication. When the source of information is “analog” (that is, continuous‐time and continuous‐amplitude), the analog signals need to be converted to a digital form for transmission or storage. The first steps in this process are sampling and quantization.
Figure 1.2 Illustration of the sampling process.
The conversion of an analog signal into a digital form starts with a sampling operation. If the signal is a function of time, sampling consists in recording samples of the continuous‐time analog signal, usually at constant time intervals. (If the signal is visual, such as an analog photograph, the sampling occurs at regular spatial intervals, but the basic principles explained here remain the same.) The process is illustrated in Figure 1.2. The left plot shows a speech signal lasting about 3 s. The plots in the middle and right show a progressive zooming in to a portion of the signal. The signals in the first two plots can be regarded as continuous time; the signal has an infinite number of values no matter how small a time interval we examine. In the third plot, the signal is made discrete time by taking samples (the circles) of the infinite number of values in the continuous‐time signal (the dotted line). For example, the first five samples in the portion of the discrete‐time signal shown in Figure 1.2 are . The samples are taken at a constant time interval, which is called the sampling period, denoted in Figure 1.2. The inverse of the sampling period is called the sampling frequency, .
An idealized version of the sampling operation can be modeled as a multiplication of a continuous‐time signal and an impulse train of infinite duration, , resulting in a still continuous‐time signal that represents the sampled signal after sampling:
The final conversion into a discrete‐time sequence of samples from is achieved by passing the signal through an ideal processing block that selects the values of the signal at the instants when the impulses occur, resulting in a sequence with values at discrete times .
The continuous‐time signal is an idealization that is useful for understanding the process of sampling and the relationship between samples and the sampled signal. An initial question of interest is how to choose the sampling period or the sampling frequency. The theory of Fourier analysis suggests that the spectrum of the signal is an addition of an infinite number of shifted replicas of the spectrum of the original signal (see Figure 1.3). This arises from the properties that (i) the Fourier transform of an infinite train of impulses in time domain is another train of impulses and (ii) multiplication in the time domain implies a convolution in the frequency domain. Now if the original signal is band‐limited, i.e. its spectrum has nonzero power components at frequencies restricted to a bounded frequency band , then it is possible to recover or reconstruct it fully – in continuous time – without distortion from the samples , provided the sampling frequency is larger than twice the maximum frequency component in the spectrum of :
This result, known as the Nyquist–Shannon sampling theorem, determines the conditions for the sampling frequency under ideal assumptions.
In practice, there are multiple factors of non‐idealness built into the sampling process. First, it is practically impossible to generate an infinite train of impulses that form . Second, no real world signal is perfectly band‐limited. Third, there is always noise present that prevents the samples from being perfect instantaneous representations of the input signal .
In a practical setting, the choice of sampling frequency needs to account for the effect of noise in the sampled signal, the performance of the filter to limit the sampled signal maximum frequency, the goals for sampled signal quality, and a number of other factors. Therefore, the sampling frequency is frequently set to a value much larger than . Also note that the Nyquist–Shannon theorem specifies a bound on the sampling frequency based on preventing the overlap of copies of the spectrum of . There are applications that allow the use of sampling at frequencies lower than , but this subject is out of the scope of this chapter. More information about sampling can be found in digital signal processing textbooks such as [5–7].
Figure 1.3 The spectrum of a sampled signal.
After sampling, the second operation in the conversion of an analog signal into a digital signal is quantization. While sampling converts a continuous‐time signal into a discrete‐time sequence of samples, quantization converts the continuous amplitudes of the samples into values from a discrete set. Figure 1.4 illustrates this operation using the five left‐most samples in Figure 1.2. The values of these samples before quantization are real numbers. During quantization, these values are approximated by integer multiples of some value (in Figure 1.4, it was chosen to be 0.01). This operation divides the range of values taken by the input signal into equal‐sized (0.01 in this case) intervals, known as quantization intervals. For the example in Figure 1.4, using prior knowledge of the signal, the quantizer was designed under the assumption that the samples to be quantized were bounded between (note that Figure 1.2 shows a small portion of the full signal range). Consequently, the decision to have a quantization interval equal to 0.01 divides the complete range of sample values into quantization intervals of equal size. A quantizer with equal‐sized intervals is called a uniform quantizer. During quantization, the approximation of the real‐valued samples to a multiple of the quantization interval size is accomplished by a simple operation of rounding to the closest value. This operation is shown in Figure 1.4 with the original real‐valued samples represented as circles and their approximation to the closest interval values as x's. The last step in quantization consists of assigning a label that identifies the quantization interval. In the case of Figure 1.4
