High-Dimensional Covariance Estimation E-Book

Mohsen Pourahmadi

• Herausgeber: John Wiley & Sons
• Kategorie: Wissenschaft und neue Technologien
• Serie: Wiley Series in Probability and Statistics
• Sprache: Englisch

Seitenzahl: 307

Veröffentlichungsjahr: 2013

Beschreibung

Methods for estimating sparse and large covariance matrices Covariance and correlation matrices play fundamental roles in every aspect of the analysis of multivariate data collected from a variety of fields including business and economics, health care, engineering, and environmental and physical sciences. High-Dimensional Covariance Estimation provides accessible and comprehensive coverage of the classical and modern approaches for estimating covariance matrices as well as their applications to the rapidly developing areas lying at the intersection of statistics and machine learning. Recently, the classical sample covariance methodologies have been modified and improved upon to meet the needs of statisticians and researchers dealing with large correlated datasets. High-Dimensional Covariance Estimation focuses on the methodologies based on shrinkage, thresholding, and penalized likelihood with applications to Gaussian graphical models, prediction, and mean-variance portfolio management. The book relies heavily on regression-based ideas and interpretations to connect and unify many existing methods and algorithms for the task. High-Dimensional Covariance Estimation features chapters on: * Data, Sparsity, and Regularization * Regularizing the Eigenstructure * Banding, Tapering, and Thresholding * Covariance Matrices * Sparse Gaussian Graphical Models * Multivariate Regression The book is an ideal resource for researchers in statistics, mathematics, business and economics, computer sciences, and engineering, as well as a useful text or supplement for graduate-level courses in multivariate analysis, covariance estimation, statistical learning, and high-dimensional data analysis.

Leseprobe

CONTENTS

Cover

Series

Title Page

Copyright

Preface

Part I: Motivation and the Basics

Chapter 1: Introduction

1.1 Least Squares and Regularized Regression

1.2 Lasso: Survival of The Bigger

1.3 Thresholding The Sample Covariance Matrix

1.4 Sparse PCA and Regression

1.5 Graphical Models: Nodewise Regression

1.6 Cholesky Decomposition and Regression

1.7 The Bigger Picture: Latent Factor Models

1.8 Further Reading

Chapter 2: Data, Sparsity, and Regularization

2.1 Data Matrix: Examples

2.2 Shrinking The Sample Covariance Matrix

2.3 Distribution of The Sample Eigenvalues

2.4 Regularizing Covariances Like a Mean

2.5 The Lasso Regression

2.6 Lasso: Variable Selection and Prediction

2.7 Lasso: Degrees of Freedom and Bic

2.8 Some Alternatives to The Lasso Penalty

Chapter 3: Covariance Matrices

3.1 Definition and Basic Properties

3.2 The Spectral Decomposition

3.3 Structured Covariance Matrices

3.4 Functions of a Covariance Matrix

3.5 PCA: The Maximum Variance Property

3.6 Modified Cholesky Decomposition

3.7 Latent Factor Models

3.8 GLM for Covariance Matrices

3.9 GLM via the Cholesky Decomposition

3.10 GLM for Incomplete Longitudinal Data

3.11 A Data Example: Fruit Fly Mortality Rate

3.12 Simulating Random Correlation Matrices

3.13 Bayesian Analysis of Covariance Matrices

Part II: Covariance Estimation: Regularization

Chapter 4: Regularizing the Eigenstructure

4.1 Shrinking The Eigenvalues

4.2 Regularizing The Eigenvectors

4.3 A Duality Between PCA and SVD

4.4 Implementing Sparse PCA: A Data Example

4.5 Sparse Singular Value Decomposition (SSVD)

4.6 Consistency of PCA

4.7 Principal Subspace Estimation

4.8 Further Reading

Chapter 5: Sparse Gaussian Graphical Models

5.1 Covariance Selection Models: Two Examples

5.2 Regression Interpretation of Entries of Σ−1

5.3 Penalized Likelihood and Graphical Lasso

5.4 Penalized Quasi-Likelihood Formulation

5.5 Penalizing The Cholesky Factor

5.6 Consistency and Sparsistency

5.7 Joint Graphical Models

5.8 Further Reading

Chapter 6: Banding, Tapering, and Thresholding

6.1 Banding The Sample Covariance Matrix

6.2 Tapering The Sample Covariance Matrix

6.3 Thresholding The Sample Covariance Matrix

6.4 Low-Rank Plus Sparse Covariance Matrices

6.5 Further Reading

Chapter 7: Multivariate Regression: Accounting for Correlation

7.1 Multivariate Regression and LS Estimators

7.2 Reduced Rank Regressions (RRR)

7.3 Regularized Estimation of B

7.4 Joint Regularization of (B, Ω)

7.5 Implementing MRCE: Data Examples

7.6 Further Reading

Bibliography

Index

Wiley Series in Probability and Statistics

WILEY SERIES IN PROBABILITY AND STATISTICS

Established by WALTER A. SHEWHART and SAMUEL S. WILKS

Editors: David J. Balding, Noel A. C. Cressie, Garrett M. Fitzmaurice, Harvey Goldstein, Iain M. Johnstone, Geert Molenberghs, David W. Scott, Adrian F. M. Smith, Ruey S. Tsay, Sanford Weisberg Editors Emeriti: Vic Barnett, J. Stuart Hunter, Joseph B. Kadane, Jozef L. Teugels

A complete list of the titles in this series appears at the end of this volume.

Copyright © 2013 by John Wiley & Sons, Inc. All rights reserved.

Published by John Wiley & Sons, Inc., Hoboken, New Jersey. Published simultaneously in Canada.

No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning, or otherwise, except as permitted under Section 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the Publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, [phone], fax [phone], or on the web at www.copyright.com. Requests to the Publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, [phone], fax [phone].

Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts in preparing this book, they make no representations or warranties with respect to the accuracy or completeness of the contents of this book and specifically disclaim any implied warranties of merchantability or fitness for a particular purpose. No warranty may be created or extended by sales representatives or written sales materials. The advice and strategies contained herein may not be suitable for your situation. You should consult with a professional where appropriate. Neither the publisher nor author shall be liable for any loss of profit or any other commercial damages, including but not limited to special, incidental, consequential, or other damages.

For general information on our other products and services please contact our Customer Care Department with the U.S. at 877-762-2974, outside the U.S. at 317-572-3993 or fax [phone].

Wiley also publishes its books in a variety of electronic formats. Some content that appears in print, however, may not be available in electronic format.

Library of Congress Cataloging-in-Publication Data:

Pourahmadi, Mohsen.  Modern methods to covariance estimation / Mohsen Pourahmadi, Department of Statistics, Texas A&M University, College Station, TX.   pages cm  Includes bibliographical references and index.  ISBN 978-1-118-03429-3 (hardback)  1. Analysis of covariance.  2. Multivariate analysis.  I. Title.  QA279.P68 2013  519.5′38–dc23  2013000326

PREFACE

The aim of this book is to bring together and present some of the most important recent ideas and methods in high-dimensional covariance estimation. It provides computationally feasible methods and their conceptual underpinnings for sparse estimation of large covariance matrices. The major unifying theme is to reduce sparse covariance estimation to that of estimating suitable regression models using penalized least squares. The framework has the great advantage of reducing the unintuitive and challenging task of covariance estimation to that of modeling a sequence of regressions. The book is intended to serve the needs of researchers and graduate students in statistics and various areas of science, engineering, economics and finance. Coverage is at an intermediate level, familiarity with the basics of regression analysis, multivariate analysis, and matrix algebra is expected.

A covariance matrix, the simplest summary measure of dependence of several variables, plays prominent roles in almost every aspect of multivariate data analysis. In the last two decades due to technological advancements and availability of high-dimensional data in areas like microarray, e-commerce, information retrieval, fMRI, business, and economy, there has been a growing interest and great progress in developing computationally fast methods that can handle data with as many as thousand variables collected from only a few subjects. This situation is certainly not suited for the classical multivariate statistics, but rather calls for a sort of “fast and sparse multivariate methodology.”

The two major obstacles in modeling covariance matrices are high-dimensionality (HD) and positive-definiteness (PD). The HD problem is familiar from regression analysis with a large number of covariates where the penalized least squares with the Lasso penalty is commonly used to obtain computationally feasible solutions. However, the PD problem is germane to covariances where one hopes to remove it by infusing regression-based ideas into principal component analysis (PCA), Cholesky decomposition, and Gaussian graphical models (inverse covariance matrices), etc.

The primary focus of current research in high-dimensional data analysis and hence covariance estimation has been on developing feasible algorithms to compute the estimators. There has been less focus on inference and the effort is mostly devoted to establishing consistency of estimators when both the sample size and the number of variables go to infinity in certain manners depending on the nature of sparsity of the model and the data. At present, there appears to be a sort of disconnection between the theory and practice where further research is hoped to bridge the gap. Our coverage follows mostly the recent pattern of research in the HD data literature by focusing more on the algorithmic aspects of the high-dimensional covariance estimation. This is a rapidly growing area of statistics and machine learning, less than a decade old, but has seen tremendous growth in such a short time. Deciding what to include in the first book of its kind is not easy as one does not have the luxury of choosing results that have passed the test of time. My selection of topics has been guided by the promise of lasting merit of some of the existing and freshly minted results, and personal preferences.

The book is divided into two parts. Part I, consisting the first three chapters, deals with the more basic concepts and results on linear regression models, high-dimensional data, regularization, and various models/estimation methods for covariance matrices. Chapter 1 provides an overview of various regression-based methods for covariance estimation, Chapter 2 introduces several examples of high-dimensional data and illustrates the poor performance of the sample covariance matrix and the need for its regularization. A fairly comprehensive review of mathematical and statistical properties of the covariance matrices along with classical covariance estimation results is provided in Chapter 3. Part II is concerned with the modern high-dimensional covariance estimation. It covers shrinkage estimation of covariance matrices, sparse PCA, Gaussian graphical models, and penalized likelihood estimation of inverse covariance matrices. Chapter 6 deals with banding, tapering, and thresholding of the sample covariance matrix or its componentwise penalization. The focus of Chapter 7 is on applications of covariance estimation and singular value decomposition (SVD), to multivariate regression models for high-dimensional data.

The genesis of the book can be traced to teaching a topic course on covariance estimation in the Department of Statistics at the University of Chicago, during a sabbatical in 2001–2002 academic year. I have had the benefits of discussing various topics and issues with many colleagues and students including Anindya Bahdra, Lianfu Chen, Michael Daniels, Nader Ebrahimi, Tanya Garcia, Shuva Gupta, Jianhua Huang, Priya Kohli, Soumen Lahiri, Mehdi Madoliat, Ranye Sun, Adam Rothman, Wei Biao Wu, Dale Zimmerman, and Joel Zinn. Financial support from the NSF in the last decade has contributed greatly to the book project. The editorial staff at John Wiley & Sons and Steve Quigley were generous with their assistance and timely reminders.

MOHSEN POURAHMADI

College Station, TexasApril, 2013

PART I

MOTIVATION AND THE BASICS

CHAPTER 1

INTRODUCTION

Is it possible to estimate a covariance matrix using the regression methodology? If so, then one may bring the vast machinery of regression analysis (regularized estimation, parametric and nonparametric methods, Bayesian analysis,  …) developed in the last two centuries to the service of covariance modeling.

In this chapter, through several examples, we show that sparse estimation of high-dimensional covariance matrices can be reduced to solving a series of regularized regression problems. The examples include sparse principal component analysis (PCA), Gaussian graphical models, and the modified Cholesky decomposition of covariance matrices. The roles of sparsity, the least absolute shrinkage and smooth operator (Lasso) and particularly the soft-thresholding operator in estimating the parameters of linear regression models with a large number of predictors and large covariance matrices are reviewed briefly.

Nowadays, high-dimensional data are collected routinely in genomics, biomedical imaging, functional magnetic resonance imaging (fMRI), tomography, and finance. Let X be an n × p data matrix where n is the sample size and p is the number of variables. By the high-dimensional data usually it is meant that p is bigger than n. Analysis of high-dimensional data often poses challenges which calls for new statistical methodologies and theories (Donoho, 2000). For example, least-squares fitting of linear models and classical multivariate statistical methods cannot handle high-dimensional X since both rely on the inverse of X′X which could be singular or not well-conditioned. It should be noted that increasing n and p each has very different and opposite effects on the statistical results. In general, the focus of multivariate analysis is to make statistical inference about the dependence among variables so that increasing n has the effect of improving the precision and certainty of inference, whereas increasing p has the opposite effect of reducing the precision and certainty. Therefore the level of detail that can be inferred about correlations among variables improves with increasing but it deteriorates with increasing .