3D Visual Communications E-Book

Guan-Ming Su

• Herausgeber: John Wiley & Sons
• Kategorie: Wissenschaft und neue Technologien
• Sprache: Englisch

Seitenzahl: 697

Veröffentlichungsjahr: 2012

Beschreibung

Provides coverage of the major theories and technologies involved in the lifecycle of 3D video content delivery

Presenting the technologies used in end-to-end 3D video communication systems, this reference covers 3D graphics and video coding, content creation and display, and communications and networking. It covers the full range of key areas from the fundamentals of 3D visual representation to the latest 3D video coding techniques, relevant communication infrastructure and networks to the 3D quality of experience.

The book is structured to logically lead readers through the topic, starting with generic and fundamental information, continuing with a detailed section of different visualisation techniques before concluding with an extensive view of 3D mobile communication systems and trends. The authors give most focus to four important areas: 3D video coding and communications; 3D graphics/gaming and mobile communications; end-to-end 3D ecosystem (including 3D display, 3D player, networking facility and 3D quality issues), and future communications and networks advances for emerging 3D experience.

• Presents the theory and key concepts behind the latest 3D visual coding framework, standards, and corresponding quality assessment
• Provides fundamental material which forms the basis for future research on enhancing the performance of 3D visual communications over current and future wireless networks
• Covers important topics including: 3D video coding and communications; 3D graphics/gaming and mobile communications; end-to-end 3D ecosystem; and future communications and networks advances for emerging 3D experience

Essential reading for engineers involved in the research, design and development of 3D visual coding and 3D visual transmission systems and technologies, as well as academic and industrial researchers.

Leseprobe

Table of Contents

Title Page

Copyright

Preface

Organization of the book

Acknowledgements

About the Authors

Chapter 1: Introduction

1.1 Why 3D Communications?

1.2 End-to-End 3D Visual Ecosystem

1.3 3D Visual Communications

1.4 Challenges and Opportunities

Chapter 2: 3D Graphics and Rendering

2.1 3DTV Content Processing Procedure

2.2 3D Scene Representation with Explicit Geometry—Geometry Based Representation

2.3 3D Scene Representation without Geometry—Image-Based Representation

2.4 3D Scene Representation with Implicit Geometry—Depth-Image-Based Representation

Chapter 3: 3D Display Systems

3.1 Depth Cues and Applications to 3D Display

3.2 Stereoscopic Display

3.3 Autostereoscopic Display

3.4 Multi-View System

3.5 Recent Advances in Hologram System Study

Chapter 4: 3D Content Creation

4.1 3D Scene Modeling and Creation

4.2 3D Content Capturing

4.3 2D-to-3D Video Conversion

4.4 3D Multi-View Generation

Chapter 5: 3D Video Coding and Standards

5.1 Fundamentals of Video Coding

5.2 Two-View Stereo Video Coding

5.3 Frame-Compatible Stereo Coding

5.4 Video Plus Depth Coding

5.5 Multiple View Coding

5.6 Multi-View Video Plus Depth (MVD) Video

5.7 Layered Depth Video (LDV)

5.8 MPEG-4 BIFS and AFX

5.9 Free-View Point Video

Chapter 6: Communication Networks

6.1 IP Networks

6.2 Wireless Communications

6.3 Wireless Networking

6.4 4G Standards and Systems

Chapter 7: Quality of Experience

7.1 3D Artifacts

7.2 QoE Measurement

7.3 QoE Oriented System Design

Chapter 8: 3D Video over Networks

8.1 Transmission-Induced Error

8.2 Error Resilience

8.3 Error Concealment

8.4 Unequal Error Protection

8.5 Multiple Description Coding

8.6 Cross-Layer Design

Chapter 9: 3D Applications

9.1 Glass-Less Two-View Systems

9.2 3D Capture and Display Systems

9.3 Two-View Gaming Systems

9.4 3D Mobile

9.5 Augmented Reality

Chapter 10: Advanced 3D Video Streaming Applications

10.1 Rate Control in Adaptive Streaming

10.2 Multi-View Video View Switching

10.3 Peer-to-Peer 3D Video Streaming

10.4 3D Video Broadcasting

10.5 3D Video over 4G Networks

Index

This edition first published 2013

© 2013 John Wiley and Sons Ltd

Registered office

John Wiley & Sons Ltd, The Atrium, Southern Gate, Chichester, West Sussex, PO19 8SQ, United Kingdom

For details of our global editorial offices, for customer services and for information about how to apply for permission to reuse the copyright material in this book please see our website at www.wiley.com.

The right of the author to be identified as the author of this work has been asserted in accordance with the Copyright, Designs and Patents Act 1988.

All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, except as permitted by the UK Copyright, Designs and Patents Act 1988, without the prior permission of the publisher.

Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may not be available in electronic books.

Designations used by companies to distinguish their products are often claimed as trademarks. All brand names and product names used in this book are trade names, service marks, trademarks or registered trademarks of their respective owners. The publisher is not associated with any product or vendor mentioned in this book. This publication is designed to provide accurate and authoritative information in regard to the subject matter covered. It is sold on the understanding that the publisher is not engaged in rendering professional services. If professional advice or other expert assistance is required, the services of a competent professional should be sought.

Library of Congress Cataloging-in-Publication Data

Su, Guan-Ming.

3D visual communications / Guan-Ming Su, Yu-Chi Lai, Andres Kwasinski, Haohong Wang.

pages cm

Includes bibliographical references and index.

ISBN 978-1-119-96070-6 (cloth)

1. Multimedia communications. 2. Three-dimensional display systems. I.Lai, Yu-Chi. II.Kwasinski, Andres.

III.Wang, Haohong, 1973- IV.Title. V.Title: Three dimensional visual communications.

TK5105.15.S83 2013

006.7– dc23

2012031377

A catalogue record for this book is available from the British Library.

ISBN: 978-1-119-96070-6

Preface

As the Avatar 3D movie experience swept the world in 2010, 3D visual content has become the most eye-catching spot in the consumer electronics products. This 3D visual wave has spread to 3DTV, Blu-ray, PC, mobile, and gaming industries, as the 3D visual system provides sufficient depth cues for end users to acquire better understanding of the geometric structure of the captured scenes, and nonverbal signals and cues in visual conversation. In addition, 3D visual systems enable observers to recognize the physical layout and location for each object with immersive viewing experiences and natural user interaction, which also makes it an important topic for both academic and industrial researchers.

Living in an era of widespread mobility and networking, where almost all consumer electronic devices are endpoints of the wireless/wired networks, the deployment of 3D visual representation will significantly challenge the network bandwidth as well as the computational capability of terminal points. In other words, the data volume received in an endpoint required to generate 3D views will be many times that of a single view in a 2D system, and hence the new view generation process sets a higher requirement for the endpoint's computational capability. Emerging 4G communication systems fit very well into the timing of 3D visual communications by significantly improving the bandwidth as well as introducing many new features designed specifically for high-volume data communications.

In this book, we aim to provide comprehensive coverage of major theories and practices involved in the lifecycle of a 3D visual content delivery system. The book presents technologies used in an end-to-end 3D visual communication system, including the fundamentals of 3D visual representation, the latest 3D video coding techniques, communication infrastructure and networks in 3D communications, and 3D quality of experience.

This book targets professionals involved in the research, design, and development of 3D visual coding and 3D visual transmission systems and technologies. It provides essential reading for students, engineers, and academic and industrial researchers. This book is a comprehensive reference for learning all aspects of 3D graphics and video coding, content creation and display, and communications and networking.

Organization of the book

This book is organized as three parts:

principles of 3D visual systems: 3D graphics and rending, 3D display, and 3D content creation are all well covered

visual communication: fundamental technologies used in 3D video coding and communication system, and the quality of experience. There are discussions on various 3D video coding formats and different communication systems, to evaluate the advantages of each system

advances and applications of 3D visual communication

Chapter 1 overviews the whole end-to-end 3D video ecosystem, in which we cover key components in the pipeline: the 3D source coding, pre-processing, communication system, post-processing, and system-level design. We highlight the challenges and opportunities for 3D visual communication systems to give readers a big picture of the 3D visual content deployment technology, and point out which specific chapters relate to the listed advanced application scenarios.

3D scene representations are the bridging technology for the entire 3D visual pipeline from creation to visualization. Different 3D scene representations exhibit different characteristics and the selections should be chosen according to the requirement of the targeted applications. Various techniques can be categorized according to the amount of geometric information used in the 3D representation spectrum; at one extreme is the simplest form via rendering without referring to any geometry, and the other end uses geometrical description. Both extremes of the technology have their own advantages and disadvantages. Therefore, hybrid methods, rendering with implicit geometries, are proposed to combine the advantages and disadvantages of both ends of the technology spectrum to better support the needs of stereoscopic applications. In Chapter 2, a detailed discussion about three main categories for 3D scene representations is given.

In Chapter 3, we introduce the display technologies that allow the end users to perceive 3D objects. 3D displays are the direct interfaces between the virtual world and human eyes and these play an important role in reconstructing 3D scenes. We first describe the fundamentals of the human visual system (HVS) and discuss depth cues. Having this background, we introduce the simplest scenario to support stereoscopic technologies (two-view only) with aided glasses. Then, the common stereoscopic technologies without aided glasses are presented. Display technologies to support multiple views simultaneously are addressed to cover the head-tracking-enabled multi-view display, occlusion-based and reflection-based multi-view system. At the end of this chapter, we will briefly discuss the holographic system.

In Chapter 4, we look at 3D content creation methods, from 3D modeling and representation, capturing, 2D to 3D conversion and, to 3D multi-view generation. We showcase three practical examples that are adopted in industrial 3D creation process to provide a clear picture of how things work together in a real 3D creation system.

It has been observed that 3D content has significantly higher storage requirements compared to their 2D counterparts. Introducing compression technologies to reduce the required storage size and alleviate transmission bandwidth is very important for deploying 3D applications. In Chapter 5, we introduce 3D video coding and related standards. We will first cover the fundamental concepts and methods used in conventional 2D video codecs, especially the state-of-the-art H.264 compression method and the recent development of next generation video codec standards. With common coding knowledge, we first introduce two-view video coding methods which have been exploited in the past decade. Several methods, including individual two-view coding, simple inter-view prediction stereo video coding, and the latest efforts on frame-compatible stereo coding, are presented. Research on the depth information to reconstruct the 3D scene has brought some improvements and the 3D video coding can benefit from introducing depth information into the coded bit stream. We describe how to utilize and compress the depth information in the video-plus-depth coding system. Supporting multi-view video sequence compression is an important topic as multi-view systems provide a more immersive viewing experience. We will introduce the H.264 multiple view coding (MVC) for this particular application. More advanced technologies to further reduce the bit rate for multi-view systems, such as the multi-view video plus depth coding and layered depth video coding system, are introduced. At the end of this chapter, the efforts on the 3D representation in MPEG-4, such as binary format for scenes (BIFS) and animation framework extension (AFX), are presented. The ultimate goal for 3D video system, namely, the free viewpoint system, is also briefly discussed.

In Chapter 6, we present a review of the most important topics in communication networks that are relevant to the subject matter of this book. We start by describing the main architecture of packet networks with a focus on those based on the Internet protocol (IP) networks. Here we describe the layered organization of network protocols. After this, we turn our focus to wireless communications, describing the main components of digital wireless communications systems followed by a presentation of modulation techniques, the characteristics of the wireless channels, and adaptive modulation and coding. These topics are then applied in the description of wireless networks and we conclude with a study of fourth generation (4G) cellular wireless standards and systems.

To make 3D viewing systems more competitive relative to 2D systems, the quality of experience (QoE) shown from 3D systems should provide better performance than from 2D systems. Among different 3D systems, it is also important to have a systematic way to compare and summarize the advances and assess the disadvantages. In Chapter 7, we discuss the quality of experience in 3D systems. We first present the 3D artifacts which may be induced throughout the whole content life cycle: content capture, content creation, content compression, content delivery, and content display. In the second part, we address how to measure the quality of experience for 3D systems subjectively and objectively. With those requirements in mind, we discuss the important factors to design a comfortable and high-quality 3D system.

Chapter 8 addresses the main issue encountered when transmitting 3D video over a channel: that of dealing with errors introduced during the communication process. The chapter starts by presenting the effects of transmission-induced errors following by a discussion of techniques to counter these errors, such as the error resilience, error concealment, unequal error protection, and multiple description coding. The chapter concludes with a discussion of cross-layer approaches.

Developing 3D stereoscopic applications has become really popular in the software industry. 3D stereoscopic research and applications are advancing rapidly due to the commercial need and the popularity of 3D stereoscopic products. Therefore, Chapter 9 gives a short discussion of commercially available products and technologies for application development. The discussed topics include commercially available glass-less two-view systems, depth adaptation capturing and displaying systems, two-view gaming systems, mobile 3D systems and perception, and 3D augmented reality systems.

In the final chapter, we introduce the state-of-the-art technologies for delivering compressed 3D content over communication channels. Subject to limited bandwidth constraints in the existing communication infrastructure, the bit rate of the compressed video data needs to be controlled to fit in the allowed bandwidth. Consequently, the coding parameters in the video codec need to be adjusted to achieve the required bit rate. In this chapter, we first review different popular 2D video rate control methods, and then discuss how to extend the rate control methods to different 3D video streaming scenarios. For the multi-view system, changing the viewing angle from one point to another point to observe a 3D scene (view switching) is a key feature to enable the immersive viewing experience. We address the challenges and the corresponding solutions for 3D view switching. In the third part of this chapter, we discuss the peer-to-peer 3D video streaming services. As the required bandwidth for 3D visual communication service poses a heavy bandwidth requirement on centralized streaming systems, the peer-to-peer paradigm shows great potential for penetrating the 3D video streaming market. After this, we cover 3D video broadcasting and 3D video communication over 4G cellular networks.

Acknowledgements

We would like to thank a few of the great many people whose contributions were instrumental in taking this book from an initial suggestion to a final product. First, we would like to express our gratitude to Dr. Chi-Yuan Yao for his help on collecting and sketching the content in Sections 9.1 and 9.2 and help with finishing Chapter 9 in time. We also thank him for his input on scene representation because of his deep domain knowledge in the field of computer geometry. We would like to thank Dr. Peng Yin and Dr. Taoran Lu for their help in enriching the introduction of HEVC. We also thank Mr. Dobromir Todorov for help in rendering figures used in Chapters 2 and 9. Finally, the authors appreciate the many contributions and sacrifices that our families have made to this effort. Guan-Ming Su would like to thanks his wife Jing-Wen's unlimited support and understanding during the writing process; and also would like to dedicate this book to his parents. Yu-Chi Lai would like to thank his family for their support of his work. Andres Kwasinski would like to thank his wife Mariela and daughters Victoria and Emma for their support, without which this work would not have been possible. Andres would also like to thank all the members of the Department of Computer Engineering at the Rochester Institute of Technology. Haohong Wang would like to thank his wife Xin Lu, son Nicholas and daughter Isabelle for their kind supports as always, especially for those weekends and nights that he had to be separated from them to work on this book at the office. The dedication of this book to our families is a sincere but inadequate recognition of all their contributions to our work.

About the Authors

Guan-Ming Su received the BSE degree in Electrical Engineering from National Taiwan University, Taipei, Taiwan, in 1996 and the MS and PhD degrees in Electrical Engineering from the University of Maryland, College Park, U.S.A., in 2001 and 2006, respectively. He is currently with Dolby Labs, Sunnyvale, CA. Prior to this he has been with the R&D Department, Qualcomm, Inc., San Diego, CA; ESS Technology, Fremont, CA; and Marvell Semiconductor, Inc., Santa Clara, CA. His research interests are multimedia communications and multimedia signal processing. He is the inventor of 15 U.S. patents and pending applications. Dr Su is an associate editor of Journal of Communications; guest editor in Journal of Communications special issue on Multimedia Communications, Networking, and Applications; and Director of review board and R-Letter in IEEE Multimedia Communications Technical Committee. He serves as the Publicity Co-Chair of IEEE GLOBECOM 2010, International Liaison Chair in IEEE ICME 2011, Technical Program Track Co-Chair in ICCCN 2011, and TPC Co-Chair in ICNC 2013. He is a Senior member of IEEE.

Yu-Chi Lai received the B.S. from National Taiwan University, Taipei, R.O.C., in 1996 in Electrical Engineering Department. He received his M.S. and Ph.D. degrees from University of Wisconsin–Madison in 2003 and 2009 respectively in Electrical and Computer Engineering. He received his M.S. and Ph.D. degrees from University of Wisconsin–Madison in 2004 and 2010 respectively in Computer Science. He is currently an assistant professor in NTUST. His research focus is on the area of computer graphics, computer vision, multimedia, and human-computer interaction. Due to his personal interesting, he is interested in industrial projects and he currently also cooperates with IGS to develop useful and interesting computer game technologies and NMA to develop animation technologies.

Andres Kwasinski received in 1992 his diploma in Electrical Engineering from the Buenos Aires Institute of Technology, Buenos Aires, Argentina, and, in 2000 and 2004 respectively, the M.S. and Ph.D. degrees in Electrical and Computer Engineering from the University of Maryland, College Park, Maryland. He is currently an Assistant Professor at the Department of Computer Engineering, Rochester Institute of Technology, Rochester, New York. Prior to this, he was with the Wireless Infrastructure group at Texas Instruments Inc., working on WiMAX and LTE technology, and with the University of Maryland, where he was a postdoctoral Research Associate. Dr. Kwasinski is a Senior Member of the IEEE, an Area Editor for the IEEE Signal Processing Magazine and Editor for the IEEE Transactions on Wireless Communications. He has been in the Organizing Committee for the 2010 IEEE GLOBECOM, 2011 and 2012 IEEE ICCCN, 2012 ICNC and 2013 IEEE ICME conferences. Between 2010 and 2012 he chaired the Interest Group on Distributed and Sensor Networks for Mobile Media Computing and Applications within the IEEE Multimedia Communications Technical Committee. His research interests are in the area of multimedia wireless communications and networking, cross layer designs, cognitive and cooperative networking, digital signal processing and speech, image and video processing for signal compression and communication, and signal processing for non-intrusive forensic analysis of speech communication systems.

Haohong Wang received the B.S. degree in computer science and the M.Eng. degree in computer applications both from Nanjing University, China, the M.S. degree in computer science from University of New Mexico, and the Ph.D. degree in Electrical and computer engineering from Northwestern University, Evanston, USA. He is currently the General Manager of TCL Research America, TCL Corporation, at Santa Clara, California, in charge of the overall corporate research activities in North America with research teams located at fourplaces. Prior to that he held various technical and management positions at AT&T, Catapult Communications, Qualcomm, Marvell, TTE and Cisco. Dr. Wang's research involves the areas of multimedia processing and communications, mobile sensing and data mining. He has published more than 50 articles in peer-reviewed journals and International conferences. He is the inventor of more than 40 U.S. patents and pending applications. He is the co-author of 4G Wireless Video Communications (John Wiley & Sons, 2009), and Computer Graphics (1997).

Dr. Wang is the Editor-in-Chief of the Journal of Communications, a member of the Steering Committee of IEEE Transactions on Multimedia, and an editor of IEEE Communications Surveys & Tutorials. He has been serving as an editor or guest editor for many IEEE and ACM journals and magazines. He chairs the IEEE Technical Committee on Human Perception in Vision, Graphics and Multimedia, and was the Chair of the IEEE Multimedia Communications Technical Committee. He is an elected member of the IEEE Visual Signal Processing and Communications Technical Committee, and IEEE Multimedia and Systems Applications Technical Committee. Dr. Wang has chaired more than dozen of International conferences, which includes the IEEE GLOBECOM 2010 (Miami) as the Technical Program Chair, and IEEE ICME 2011 (Barcelona) and IEEE ICCCN 2011 (Maui) as the General Chair.

Chapter 1

Introduction

1.1 Why 3D Communications?

Thanks to the great advancement of hardware, software, and algorithms in the past decade, our daily life has become a major digital content producer. Nowadays, people can easily share their own pieces of artwork on the network with each other. Furthermore, with the latest development in 3D capturing, signal processing technologies, and display devices, as well as the emergence of 4G wireless networks with very high bandwidth, coverage, and capacity, and many advanced features such as quality of service (QoS), low latency, and high mobility, 3D communication has become an extremely popular topic. It seems that the current trend is closely aligned with the expected roadmap for reality video over wireless, estimated by Japanese wireless industry peers in 2005 (as shown in Figure 1.1), according to which the expected deployment timing of stereo/multi-view/hologram video is around the same time as the 4G wireless networks deployment. Among those 3D video representation formats, the stereoscopic and multi-view 3D videos are more mature and the coding approaches have been standardized in Moving Picture Experts Group (MPEG) as “video-plus-depth” (V+D) and the Joint Video Team (JVT) Multi-view Video Coding (MVC) standard, respectively. The coding efficiency study shows that coded V+D video only takes about 1.2 times bit rate compared to the monoscopic video (i.e., the traditional 2D video). Clearly, the higher reality requirements would require larger volumes of data to be delivered over the network, and more services and usage scenarios to challenge the wireless network infrastructures and protocols.

Figure 1.1 Estimated reality video over wireless development roadmap.

From a 3D point of view, reconstructing a scene remotely and/or reproducibly as being presented face-to-face has always been a dream through human history. The desire for such technologies has been pictured in many movies, such as Star Trek's Holodeck, Star Wars' Jedi council meeting, The Matrix's matrix, and Avatar's Pandora. The key technologies to enable such a system involve many complex components, such as a capture system to describe and record the scene, a content distribution system to store/transmit the recorded scene, and a scene reproduction system to show the captured scenes to end users. Over the past several decades, we have witnessed the success of many applications, such as television broadcasting systems in analog (e.g., NTSC, PAL) and digital (e.g., ATSC, DVB) format, and home entertainment system in VHS, DVD, and Blu-ray format. Although those systems have served for many years and advanced in many respects to give better viewing experiences, end users still feel that the scene reconstruction has its major limitation: the scene presentation is on a 2D plane, which significantly differs from the familiar three-dimensional view of our daily life. In a real 3D world, humans can observe objects and scenes from different angles to acquire a better understanding of the geometry of the watched scenes, and nonverbal signals and cues in visual conversation. Besides, humans can perceive the depth of different objects in a 3D environment so as to recognize the physical layout and location for each object. Furthermore, 3D visual systems can provide immersive viewing experience and higher interaction. Unfortunately, the existing traditional 2D visual systems cannot provide those enriched viewing experiences.

The earliest attempt to construct a 3D image was via the anaglyph stereo approach which was demonstrated by W. Rollmann in 1853 and J. C. D'Almeida in 1858 and patented in 1891 by Louis Ducos du Hauron. In 1922, the earliest confirmed 3D film was premiered at the Ambassador Hotel Theater in Los Angeles and was also projected in the red/green anaglyph format. In 1936, Edwin H. Land invented the polarizing sheet and demonstrated 3D photography using polarizing sheet at the Waldorf-Astoria Hotel. The first 3D golden era was between 1952 and 1955, owing to the introduction of color stereoscopy. Several golden eras have been seen since then. However, there are many factors affecting the popularity and success of 3D visual systems, including the 3D visual and content distribution technologies, the viewing experience, the end-to-end ecosystem, and competition from improved 2D systems. Recently, 3D scene reconstruction algorithms have achieved great improvement, which enables us to reconstruct a 3D scene from a 2D one and from stereoscope images, and the corresponding hardware can support the heavy computation at a reasonable cost, and the underlying communication systems have advanced to provide sufficient bandwidth to distribute the 3D content. Therefore, 3D visual communication systems have again drawn considerable attention from both academia and industry.