Non-Gaussian Statistical Communication Theory E-Book

David Middleton

• Herausgeber: John Wiley & Sons
• Kategorie: Wissenschaft und neue Technologien
• Serie: IEEE Press Series on Digital & Mobile Communication
• Sprache: Englisch

Seitenzahl: 979

Veröffentlichungsjahr: 2012

Beschreibung

The book is based on the observation that communication is the central operation of discovery in all the sciences. In its "active mode" we use it to "interrogate" the physical world, sending appropriate "signals" and receiving nature's "reply". In the "passive mode" we receive nature's signals directly. Since we never know a prioriwhat particular return signal will be forthcoming, we must necessarily adopt a probabilistic model of communication. This has developed over the approximately seventy years since it's beginning, into a Statistical Communication Theory (or SCT). Here it is the set or ensemble of possible results which is meaningful. From this ensemble we attempt to construct in the appropriate model format, based on our understanding of the observed physical data and on the associated statistical mechanism, analytically represented by suitable probability measures. Since its inception in the late '30's of the last century, and in particular subsequent to World War II, SCT has grown into a major field of study. As we have noted above, SCT is applicable to all branches of science. The latter itself is inherently and ultimately probabilistic at all levels. Moreover, in the natural world there is always a random background "noise" as well as an inherent a priori uncertainty in the presentation of deterministic observations, i.e. those which are specifically obtained, a posteriori. The purpose of the book is to introduce Non-Gaussian statistical communication theory and demonstrate how the theory improves probabilistic model. The book was originally planed to include 24 chapters as seen in the table of preface. Dr. Middleton completed first 10 chapters prior to his passing in 2008. Bibliography which represents remaining chapters are put together by the author's close colleagues; Drs. Vincent Poor, Leon Cohen and John Anderson. email href="mailto:[email protected]">[email protected] to request Ch.10

Leseprobe

IEEE Press

445 Hoes Lane

Piscataway, NJ 08854

IEEE Press Editorial Board

Lajos Hanzo, Editor in Chief

Kenneth Moore, Director of IEEE Book and Information Services (BIS)

IEEE Information Theory Society, Sponsor

Copyright © 2012 by the Institute of Electrical and Electronics Engineers, Inc.

Published by John Wiley & Sons, Inc., Hoboken, New Jersey. All rights reserved.

Published simultaneously in Canada

No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning, or otherwise, except as permitted under Section 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the Publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, [phone], fax [phone], or on the web at www.copyright.com. Requests to the Publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, [phone], fax [phone], or online at http://www.wiley.com/go/permission.

Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts in preparing this book, they make no representations or warranties with respect to the accuracy or completeness of the contents of this book and specifically disclaim any implied warranties of merchantability or fitness for a particular purpose. No warranty may be created or extended by sales representatives or written sales materials. The advice and strategies contained herein may not be suitable for your situation. You should consult with a professional where appropriate. Neither the publisher nor author shall be liable for any loss of profit or any other commercial damages, including but not limited to special, incidental, consequential, or other damages.

For general information on our other products and services or for technical support, please contact our Customer Care Department within the United States at [phone], outside the United States at [phone] or fax [phone].

Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may not be available in electronic formats. For more information about Wiley products, visit our web site at www.wiley.com.

Library of Congress Cataloging-in-Publication Data:

Print ISBN: 9780470948477

Foreword

“The statistical theory of communication has proven to be a powerful methodology for the design, analysis, and understanding of practical systems for electronic communications and related applications. From its origins in the 1940's to the present day, this theory has remained remarkably vibrant and useful, while to evolve as new modes of communication emerge. The publication in 1960 of Introduction to Statistical Communication Theory (ISCT) was a landmark for the field of statistical communication theory – random processes, modulation and detection, signal extraction, information theory – we combined into a single, unified treatment at a level of depth and degree of completeness which have not been matched in any subsequent comprehensive work. Moreover, ICST introduced a further interdisciplinary feature, in which relevant physical characteristics of communication channels were incorporated into many topics.”

I wrote these words in my foreword to the 1996 IEEE Press reissue of David Middleton's classic book Introduction to Statistical Communication Theory (ISCT). In the decades after he wrote ISCT and well into the 2000s, Dr. Middleton pursued a very active research program motivated by the above last-mentioned feature of his book, namely, the incorporation of physics into communication theory. Among other considerations, this work notably brought in two features that were not evident in ISCT: the spatial dimension and the fact that many of the noise phenomena arising in applications are not well modeled as Gaussian random processes. Through this research, he greatly expanded our understanding of the physical aspects of communication, and this book was conceived as a synthesis of that work.

David Middleton was a giant in the field of statistical communication theory, and the physicist's perspective he brought to the field was somewhat rare. Sadly, he did not live to complete this book, his full vision for which is described in the introductory chapter. However, the material that he did complete is nevertheless of considerable value in exposing the insights that he gained over more than six decades in the field.

H. Vincent PoorPrinceton, New Jersey

Visualizing the Invisible

Hanging on my apartment wall is a framed line drawing entitled Cat with Mouse from 1959. The abstract profile of tail and rear haunches connects in swift sweeping lines across the paper to form pointed ears, whiskers, and culminates in a half hidden mouse enveloped in its paws, its fate uncertain. It has a Picasso-like quality, curvilinear strokes of the oil crayon made with verve, remarkable for the empty space surrounding the lines as for the seductive feline form. The artist was David Blakeslee Middleton, my father, and judging from the date on the drawing he probably made this sketch one evening after a day in his study proofreading the galleys for his soon to be published An Introduction to Statistical Communication Theory. Drawing on a big manila pad was one of his ways to unwind, perhaps to counterbalance the intense analytical left brain activity with some right brain visual and emotional release through imaginative sketches. Oftentimes this was also a clever way to entertain his young children before bedtime. I can vividly recall various creatures—cats in particular were a favorite—and other-worldly landscapes magically taking shape on the blank paper. Witnessing these forms unfolding from my father's pen, to an impressionable child it seemed that science and drawing were naturally linked thus:

In this Preface I'd like to share a few recollections about how the arts, and drawing in particular, played an integral part in David Blakeslee Middleton's creative process. And since I am an architect, the reader will hopefully forgive “artistic license” to speculate that his ability to visualize an otherwise invisible world of signals and noise through drawing contributed to the evolution of his research in communication theory over a sixty year period, culminating in this book.

My father was gifted with a marvelous drawing hand and a free imagination. No doubt being raised in a family of creative artists fostered this talent. His mother played violin. His father was a poet, one uncle a playwright, another uncle the Austrian émigré painter Joseph Amadeus Fleck, and one aunt was the author of well known western novels including My Friend Flicka. Dad was married twice, to women who were both gifted artists. On top of this arsenal of artistic influences, my father was a seasoned classical pianist. While never quite ready for Carnegie Hall, he would practice regularly and with evident pleasure into the late evening hours the sonatas of Beethoven, Schubert and Chopin. He accrued over many years a large library of history, biography, and many books on painting and sculpture.

The inclination—possibly the compulsion—to draw was, I think, a natural part of his life and heritage. It manifested itself on an endless stream of drawings on small white note or oversized manila artist pads, on gift cards for friends and family, doodles with authorial inscriptions in his books, and diagrams and marginalia surrounding his manuscripts on yellow lined writing tablets. He drew animals—phantasmagorical or real—incessantly, and when clearing out his papers after his death in 2008, my siblings and I discovered a trove of these drawings. One can discern the evolution of his drawing style from an early reliance on ruled lines and graphs, along with somewhat rigid Cubist-like animal profiles and caricatures, to some three decades later drawings full of squiggles and agitated contours, where straight lines are only rarely evident. In one image of a dragon (another favorite subject) the profile is so diffused and fuzzy it seems like the poor creature has been intercepted by intense noise signals and is being jostled to the point of being truly indeterminate. Over time, this tendency to greater abstraction of figure, of fluctuating line weight, and verve of the freehand line may also be indicative of his later research encompassing quantum theory and its attendant complexity and fluidity. My father's technical diagrams and illustrations, usually drawn by hand and directly transcribed into his first two books and a host of his published research papers, appear to become more visually dynamic and complex over time. This may due to the increasing complexity of the science involved, or to better explain the concept to the reader, or both.

I doubt a day of work went by that my father didn't draw a graph or figure while working his physics. I would stop in his study upstairs to say hi and there would be invariably a handful of colored pens, always fine point, scattered on his crowded desk or stowed in his shirt pocket (I never saw a pocket liner—but often an ink stain ruining another button-down Oxford). A typical draft manuscript would have at least several squiggles and amoeboid-like drawings, with darts of straight lines, attended by numeric or alphabetical punctuation, piercing through or oscillating around the perimeters of these figures. Parabolic arcs, waves both regular and asymmetrical, conic sections and planar intersections, vast irregular rhomboids and tori: these would all be lovingly drawn, in a fine line, exacting proportion, and precise annotation. They strike the uninitiated (at least this non-scientist) as abstract volumes and forms in a state of visual tension, yet somehow poised, not unlike figures in a Miro or early Kandinsky painting. As a young adult, struggling in calculus and marginally apprehending the finer points of Newton's Three Laws, these drawings made physics more accessible, more “real.” To this day I remain baffled as to the actual science, but they gave graphic shape and substance to equations describing a part of the electromagnetic spectrum. These diagrams remain for me a rich formal vocabulary intriguing in their geometry and multi-dimensions, making visible the world of detecting signals in a cosmos of noise.

My father's working method was really like that of a visual artist or even a composer. If one looks at any of his early drafts of his abstracts or papers it resembles a score by Beethoven, full of scratches and ink blotches and re-scribed measures. One would never perceive them as “Mozartian”, immaculately and precisely inscribed the first time set to paper. Indeed, some “final” drafts of his abstracts or even the chapters for this book look like collages. They are full of rub outs, white-outs (liquid or tape!), edits scribbled vertically in the margins in multiple ink colors, here and there whole sections cut out with a razor and replaced with revised equations and text. Few pages in a final draft would escape this sort of surgery. And almost everything was done long hand; typewriters just weren't capable of constructing the architecture of these dense and complex equations. While the technology of computational devices evolved rapidly in his lifetime, my father rarely if ever availed himself of a computer, even when they became a household appliance. Trained on a slide rule, for many years his only concession to modern hardware was a 1987 Hewlett Packard hand held calculator with Reverse Polish Notation only occasionally dusted off to verify a certain summation. Otherwise the entire math for his abstracts and various research papers was done in his head, augmented with a set of yellow or white lined paper pads to work through some of the calculations. I was astonished to notice that over all the decades he almost always wrote his manuscripts on ink and paper. Pencil drafts are rarely evident. And despite the time it required, I can discern an almost joyful immersion in the tactility of “cut and paste”. I can not attest to the science, but the manuscript process provided one heck of show: It was “messy vitality” at its finest!

“Creation,” as the architect Le Corbusier once said, “is a patient search.” I grew up with my father modeling this dictum every day. Dad's efforts had all the dead ends and small victories familiar to any creative artist—or scientist: many hours of quiet, methodical sorting and testing, punctuated by intermittent sturm und drang of frustration, occasional late nights of revisions to remove some small theoretical imperfection, or re-drafting to get it “just right”. Science is a rigorous, un-sentimental, and empirical pursuit. This effort was one of mental exertion, but it seemed to me often one of physical endurance for my father. Sometimes in the early evening when he came downstairs from his study and said “it was a hard day at the office,” he surely meant it.

It wasn't until his last days in the hospital, still proofing and re-writing sections of the chapters of this book while cancer gnawed away at him, that I began to appreciate the implication of his intention to include “a space-time treatment” in the title of this monograph—and to apprehend how much visualization was a key to his particular type of research. Not unlike his drawing style, I believe his research evolved from a cool “classical” foundation in statistics to the rarefied and “romantic” world of quantum theory. He didn't say much about this compulsion to draw, but didn't have to: one can easily discern the delight and care he took to making the diagrams illuminate, punctuate, and indeed more richly describe what his summations and differentials express (to my untrained eye) in their inscrutable syntax. Only later in life did I come to realize that not all scientists had this gift, and that for many their “patient search” was less tactile and literally more cerebral.

I'm convinced my father's research relied on intensely visualizing the natural environment—imagining the shape and texture of a stochastic universe—in order to tease out the abstract equations that would accurately account for the sometimes predictable but more likely random events and features of a world full of physical uncertainty. It is a world of turbulence, under the sea or in the air, waves of all shapes and sizes and curves, chaotic and sublime. Today this world is super-saturated with signals, from radar and sonar and radio, from ELF to microwave broadcasts, cell phones and Blue-Tooth and E-Z Pass readers, all seeking a receptor of one sort or another, always under threat of being scattered. As I reflect on how David Blakeslee Middleton's work sought to find order and predictability in chaos, I am awed at how fantastically rich the physics of something that can not be seen readily has had such an impact on our modern life, and the role my father played in enhancing the connective-ness we today take for granted.

David Blakeslee Middleton

Acknowledgments

Publication of David Blakeslee Middleton's opus finale could not have been accomplished without the wonderful support and efforts of his colleagues, collaborators, and editors. These include Dawn M. Ernst-Smith and Nancy Russell, who had the acuity of vision and the patience of saints in transcribing my father's handwriting into cogent and editable form. Those who provided my father encouragement and careful editing over the many years of gestation of this book include Dr. Leon Cohen, Dr. Joseph Goodman, and Dr. Julius Bussgang, who provided guidance and support for the purpose of this publication, and most especially Dean Vincent Poor of Princeton University's Department of Electrical Engineering, who gave valuable advice and encouragement throughout the later stages of this work, and who organized a symposium in honor of my father at Princeton in 2008. My family is especially indebted to: Dr. John Anderson of Lund University, who has painstakingly proofread the manuscript, and has gracefully provided a Editor's Note describing the context of this work; Sanchari Sil at Thomson Digital for meeting the exacting challenges of typesetting and composition; and the Editors and Publishers at John Wiley & Sons – most especially Danielle LaCourciere, Senior Production Editor and Taisuke Soda, Senior Editor – who collectively had the diligence and vision to get this work to press despite some formidable obstacles. The Middleton family expresses our heartfelt gratitude to all involved for bringing to completion a task my father had hoped to finish himself.

David Blakeslee Middleton FAIANew York City, 2011

About the Author

Physicist and pioneer of modern Statistical Communication Theory, David Middleton devoted his career to the study of how information is transferred from one point in space-time to another, with ground-breaking applications in radar, underwater listening devices, satellite technology, and wireless communications. Born in 1920 in New York City, he attended Deerfield Academy and graduated from Harvard College in 1942. During WW II, as a Special Research Associate at Harvard's Radio Research Laboratory, he refined the analysis and subsequent effectiveness of aluminum “chaff” used to jam enemy radar, protecting American aircraft from detection. He received his M.A in 1945 and Ph.D. from Harvard in 1947. From 1949–54 he was an Assistant Professor of Applied Physics at Harvard.

In 1960, Dr. Middleton published “An Introduction to Statistical Communication Theory”. Widely translated and reprinted, it remains the seminal text for the field. In 1965 he published “Topics in Communication Theory”. For over fifty years he taught, inspired and occasionally confounded his graduate and doctoral students at Harvard, RPI, Columbia, Johns Hopkins, Texas, Rice, and University of Rhode Island. From 1954 on he was a Consulting Physicist to various government agencies, advisory boards, private industry and served for seven years on the U.S. Naval Research Advisory Committee. During his lifetime Dr. Middleton published over 160 papers and abstracts.

Dr. Middleton was a Life Fellow of the Institute of Electrical and Electronic Engineers, and Fellow of the American Physical Society, the Acoustical Society of America, and American Association for the Advancement of Science. He died in the city of his birth on 16 November 2008.

Editor's Note

When IEEE Press and its Series on Digital and Mobile Communication signed a contract with David Middleton in 1996, we could not expect that the book would take 16 years and run to 24 chapters. By the time of his passing in 2008, David had completed versions of 10 chapters, and more than one version of several. How should one prepare such a manuscript for publication? Unlike F.X. Suessmayr, the young assistant who completed Mozart's Requiem, I have not made up new text and whole chapters. The Requiem became a beloved piece of music, even with its Suessmayr chapters, but after much deliberation we have decided to limit David's book to nine chapters. Successive chapters are more rough in form and contain newer material, and it becomes steadily more risky to guess Middleton's intentions. With Chapter 10, we could find no practical way to create a reliable text. Those who would like to see for themselves–and perhaps attempt a revision–can view this part of the manuscript on the Wiley book Web site.

Thus, we present a book of nine chapters. David Middleton's original “Introduction” is reproduced as he wrote it, and the reader can see there the magnificent opus that he had in mind. In the rest of the book, we have removed all mention of the missing parts and we have aimed to make the book a coherent work in nine chapters. Exercises existed for Chapters 4 and 6 only, and these are included. There were to be many appendices, but the text for only two exists; these are included.

Middleton's style employed a multitude of equation and section references, and these presented a special problem because they often referred to early versions now lost. Usually they could be re-established, but where there was a serious risk of error, they were deleted. References to conclusions reached in Chapters 11–24 were softened or deleted, since in many cases it was doubtful that they exist. Many typed manuscript passages were not proofread by the author, and contained obvious errors and misspelled foreign names that had to be corrected. With thousands of corrections, it is certain there are errors and misguesses in this editorial process. We regret this, and ask for the indulgence of the reader.

What then is this book about? As the title suggests, it is about detection and estimation when statistics are neither Gaussian nor homogeneous. For example, in Chapter 2, Middleton extends Wiener–Khintchine theory to this case. But Middleton believes that the essence of the problem is transmission medium, and he devotes the later of the nine chapters to complicated multilayer inhomogeneous media, whose transmission is as much by diffusion as by electromagnetic waves. Another recurring theme in the book, as pointed out by Vincent Poor, is processing in four-dimensional space–time. A perhaps more subtle theme is sonar and signaling in the ocean. His hope seems to have been that later chapters would tie together these frightfully complex media with traditional detection and estimation. He had a vision for how to do this. Could he have carried it out as he wished? Can anyone carry it out, or is a practical and understandable solution beyond our comprehension? We will not find the answer this time around.

The attentive reader can find more than these technical matters. David Middleton's book is a window to a past now nearly forgotten: to mid-twentieth century pioneers in detection, estimation, and signal processing; to organizations that changed the world; to a Cold War with doomsday submarines; and to a much smaller research community, sometimes employed by opposing armies, who nonetheless knew each other's work. At the end of Chapter 3, readers can find an interesting history of the matched filter, which Middleton helped discover. An historical oddity is that Middleton was a coauthor of the first paper, Vol. 1, No. 1, p. 1, published by the prestigious IEEE Transactions on Information Theory (Ref. 1 in Chapter 4). It is also interesting that Middleton's day-to-day research world was almost free of computers. He mentions them from time to time, but a researcher in this field today would base his or her thinking much more on algorithms and what they could and could not do, and verify the work every step of the way with computations.

We wish you happy reading!John B. Anderson

Introduction

In his Introduction, David Middleton refers to the parts of his book that were not completed; we have left them in place so that the reader may see the original plan for the book — Editors.

This Introduction explains my purpose of writing this book and its earlier companion [1]. It is based on the observation that communication is the central operation of discovery in all the sciences. In its “active mode,” we use it to “interrogate” the physical world, sending appropriate “signals” and receiving nature's “reply.” In the “passive mode,” we receive nature's signals directly. Since we never know a priori what particular return signal will be forthcoming, we must necessarily adopt a probabilistic model of communication. This has developed over approximately 70 years since its beginning into a statistical communication theory (SCT). Here, it is the set or ensemble of possible results that is meaningful. From this ensemble, we attempt to construct an appropriate model format, based on our understanding of the observed physical data and on the associated statistical mechanism, analytically represented by suitable probability measures.

Since its inception in the late 1930s, and in particular subsequent to World War II, SCT has grown into a major field of study. As we have noted above, SCT is applicable to all branches of science. The latter itself is inherently and ultimately probabilistic at all levels. Moreover, in the natural world, there is always a random background “noise” as well as an inherent a priori uncertainty in the presentation of deterministic observations, that is, those that are specifically obtained, a posteriori.

The Book's Title

Let me now begin with a brief explanation of the title of the book.

Elements of Non-Gaussian Space–Time Statistical Communication Theory, Propagation, Noise, and Signal Processing in the Canonical Channel

My choice of “elements” is intended to signify a connected development of fundamental material, but with an exposition that is inevitably incomplete, with many important topics necessarily omitted, primarily for reasons of space. “Elements,” however, includes the propagation physics of the channel, the role of spatial coupling (e.g., apertures and arrays), and noise models, both physically founded. The analyses also treat deterministic and random scatter, Doppler effects, and, of course, four-dimensional (i.e., space and time) signal processing, with particular attention to threshold reception in arbitrary noise environments. Non-Gaussian noise receives special analysis, since it is a phenomenon of increasing practical importance. Moreover, it is a topic that presents much greater complexities than the familiar Gaussian noise model, which has dominated so much of recent as well as earlier studies.

In addition, the class of signals considered here is entirely general or “canonical,”1 so that the coding results of parallel studies in Information Theory2 [2] can be readily applied in specific cases. This book (Book 2) may also be considered an extension of Book 1 (An Introduction to Statistical Communication Theory, [1]). Book 1 considers primarily random processes and continuously sampled noise and signals. Here, on the other hand, Book 2 deals with many earlier features of Book 1. These that require a four-dimensional space–time formulation now involve random fields. Particular attention is also given here to the physics of propagation. In this context, another portion of this book is then devoted to physical problems of signal detection and extraction in a Bayesian formulation, with particular attention to threshold (or weak signal) operation.

Finally, both homogeneous and inhomogeneous media are considered here. Such media are linear provided their equations of propagation are themselves linear where the requirement, of course, is that the superposition principle holds: If α is a typical field in such media, we have, symbolically for two fields . On the other hand, for nonlinear media : superposition is violated. Furthermore, stationarity or nonstationarity itself does not invalidate linearity or nonlinearity. Note, however, that the presence of a (finite) boundary is itself an inhomogeneity of the medium, and thus is a component of nonlinearity by the above definition. The presence of scattering elements (inhomogeneities) is also a major topic of interest as are the probability distributions generated by such scattering elements.