Large-Scale Computing Techniques for Complex System Simulations E-Book

Werner Dubitzky

• Herausgeber: John Wiley & Sons
• Kategorie: Wissenschaft und neue Technologien
• Serie: Wiley Series on Parallel and Distributed Computing
• Sprache: Englisch

Seitenzahl: 365

Veröffentlichungsjahr: 2012

Beschreibung

Complex systems modeling and simulation approaches are being adopted in a growing number of sectors, including finance, economics, biology, astronomy, and many more. Technologies ranging from distributed computing to specialized hardware are explored and developed to address the computational requirements arising in complex systems simulations. The aim of this book is to present a representative overview of contemporary large-scale computing technologies in the context of complex systems simulations applications. The intention is to identify new research directions in this field and to provide a communications platform facilitating an exchange of concepts, ideas and needs between the scientists and technologist and complex system modelers. On the application side, the book focuses on modeling and simulation of natural and man-made complex systems. On the computing technology side, emphasis is placed on the distributed computing approaches, but supercomputing and other novel technologies are also considered.

Leseprobe

Table of Contents

Cover

WILEY SERIES ON PARALLEL AND DISTRIBUTED COMPUTING

Title page

Copyright page

Foreword

Preface

Contributors

Chapter 1 State-of-the-Art Technologies for Large-Scale Computing

1.1 INTRODUCTION

1.2 GRID COMPUTING

1.3 VIRTUALIZATION

1.4 CLOUD COMPUTING

1.5 GRID AND CLOUD: TWO COMPLEMENTARY TECHNOLOGIES

1.6 MODELING AND SIMULATION OF GRID AND CLOUD COMPUTING

1.7 SUMMARY AND OUTLOOK

Chapter 2 The e-Infrastructure Ecosystem: Providing Local Support to Global Science

2.1 THE WORLDWIDE E-INFRASTRUCTURE LANDSCAPE

2.2 BALTICGRID: A REGIONAL E-INFRASTRUCTURE, LEVERAGING ON THE GLOBAL “MOTHERSHIP” EGEE

2.3 THE EGEE INFRASTRUCTURE

2.4 INDUSTRY AND E-INFRASTRUCTURES: THE BALTIC EXAMPLE

2.5 THE FUTURE OF EUROPEAN E-INFRASTRUCTURES: THE EUROPEAN GRID INITIATIVE (EGI) AND THE PARTNERSHIP FOR ADVANCED COMPUTING IN EUROPE (PRACE) INFRASTRUCTURES

2.6 SUMMARY

ACKNOWLEDGMENTS

Chapter 3 Accelerated Many-Core GPU Computing for Physics and Astrophysics on Three Continents

3.1 INTRODUCTION

3.2 ASTROPHYSICAL APPLICATION FOR STAR CLUSTERS AND GALACTIC NUCLEI

3.3 HARDWARE

3.4 SOFTWARE

3.5 RESULTS OF BENCHMARKS

3.6 ADAPTIVE MESH REFINEMENT HYDROSIMULATIONS

3.7 PHYSICAL MULTISCALE DISCRETE SIMULATION AT IPE

3.8 DISCUSSION AND CONCLUSIONS

ACKNOWLEDGMENTS

Chapter 4 An Overview of the SimWorld Agent-Based Grid Experimentation System

4.1 INTRODUCTION

4.2 SYSTEM ARCHITECTURE

4.3 SYSTEM IMPLEMENTATION

4.4 A SWAGES CASE STUDY

4.5 DISCUSSION

4.6 CONCLUSIONS

Chapter 5 Repast HPC: A Platform for Large-Scale Agent-Based Modeling

5.1 INTRODUCTION

5.2 AGENT SIMULATION

5.3 MOTIVATION AND RELATED WORK

5.4 FROM REPAST S TO REPAST HPC

5.5 PARALLELISM

5.6 IMPLEMENTATION

5.7 EXAMPLE APPLICATION: RUMOR SPREADING

5.8 SUMMARY AND FUTURE WORK

Chapter 6 Building and Running Collaborative Distributed Multiscale Applications

6.1 INTRODUCTION

6.2 REQUIREMENTS OF MULTISCALE SIMULATIONS

6.3 AVAILABLE TECHNOLOGIES

6.4 AN ENVIRONMENT SUPPORTING THE HLA COMPONENT MODEL

6.5 CASE STUDY WITH THE MUSE APPLICATION

6.6 SUMMARY AND FUTURE WORK

ACKNOWLEDGMENTS

Chapter 7 Large-Scale Data-Intensive Computing

7.1 DIGITAL DATA: CHALLENGE AND OPPORTUNITY

7.2 DATA-INTENSIVE COMPUTERS

7.3 ADVANCED SOFTWARE TOOLS AND TECHNIQUES

7.4 CONCLUSION

ACKNOWLEDGMENTS

Chapter 8 A Topology-Aware Evolutionary Algorithm for Reverse-Engineering Gene Regulatory Networks

8.1 INTRODUCTION

8.2 METHODOLOGY

8.3 RESULTS AND DISCUSSION

8.4 CONCLUSIONS

ACKNOWLEDGMENTS

Chapter 9 QosCosGrid e-Science Infrastructure for Large-Scale Complex System Simulations

9.1 INTRODUCTION

9.2 DISTRIBUTED AND PARALLEL SIMULATIONS

9.3 PROGRAMMING AND EXECUTION ENVIRONMENTS

9.4 QCG MIDDLEWARE

9.5 ADDITIONAL QCG TOOLS

9.6 QOSCOSGRID SCIENCE GATEWAYS

9.7 DISCUSSION AND RELATED WORK

Glossary

Index

WILEY SERIES ON PARALLEL AND DISTRIBUTED COMPUTING

Series Editor: Albert Y. Zomaya

Parallel and Distributed Simulation Systems / Richard Fujimoto

Mobile Processing in Distributed and Open Environments / Peter Sapaty

Introduction to Parallel Algorithms / C. Xavier and S. S. lyengar

Solutions to Parallel and Distributed Computing Problems: Lessons from Biological Sciences / Albert Y. Zomaya, Fikret Ercal, and Stephan Olariu (Editors)

Parallel and Distributed Computing: A Survey of Models, Paradigms, and Approaches / Claudia Leopold

Fundamentals of Distributed Object Systems: A CORBA Perspective / Zahir Tari and Omran Bukhres

Pipelined Processor Farms: Structured Design for Embedded Parallel Systems / Martin Fleury and Andrew Downton

Handbook of Wireless Networks and Mobile Computing / Ivan Stojmenovi (Editor)

Internet-Based Workflow Management: Toward a Semantic Web / Dan C. Marinescu

Parallel Computing on Heterogeneous Networks / Alexey L. Lastovetsky

Performance Evaluation and Characteization of Parallel and Distributed Computing Tools / Salim Hariri and Manish Parashar

Distributed Computing: Fundamentals, Simulations and Advanced Topics,Second Edition / Hagit Attiya and Jennifer Welch

Smart Environments: Technology, Protocols, and Applications / Diane Cook and Sajal Das

Fundamentals of Computer Organization and Architecture / Mostafa Abd-El-Barr and Hesham El-Rewini

Advanced Computer Architecture and Parallel Processing / Hesham El-Rewini and Mostafa Abd-El-Barr

UPC: Distributed Shared Memory Programming / Tarek El-Ghazawi, William Carlson, Thomas Sterling, and Katherine Yelick

Handbook of Sensor Networks: Algorithms and Architectures / Ivan Stojmenovi (Editor)

Parallel Metaheuristics: A New Class of Algorithms / Enrique Alba (Editor)

Design and Analysis of Distributed Algorithms / Nicola Santoro

Task Scheduling for Parallel Systems / Oliver Sinnen

Computing for Numerical Methods Using Visual C++ / Shaharuddin Salleh, Albert Y. Zomaya, and Sakhinah A. Bakar

Architecture-Independent Programming for Wireless Sensor Networks / Amol B. Bakshi and Viktor K. Prasanna

High-Performance Parallel Database Processing and Grid Databases / David Taniar, Clement Leung, Wenny Rahayu, and Sushant Goel

Algorithms and Protocols for Wireless and Mobile Ad Hoc Networks / Azzedine Boukerche (Editor)

Algorithms and Protocols for Wireless Sensor Networks / Azzedine Boukerche (Editor)

Optimization Techniques for Solving Complex Problems / Enrique Alba, Christian Blum, Pedro Isasi, Coromoto León, and Juan Antonio Gómez (Editors)

Emerging Wireless LANs, Wireless PANs, and Wireless MANs: IEEE 802.11, IEEE 802.15, IEEE 802.16 Wireless Standard Family / Yang Xiao and Yi Pan (Editors)

High-Performance Heterogeneous Computing / Alexey L. Lastovetsky and Jack Dongarra

Mobile Intelligence / Laurence T. Yang, Augustinus Borgy Waluyo, Jianhua Ma, Ling Tan, and Bala Srinivasan (Editors)

Advanced Computational Infrastructures for Parallel and Distributed Adaptive Applicatons / Manish Parashar and Xiaolin Li (Editors)

Market-Oriented Grid and Utility Computing / Rajkumar Buyya and Kris Bubendorfer (Editors)

Cloud Computing Principles and Paradigms / Rajkumar Buyya, James Broberg, and Andrzej Goscinski

Copyright © 2012 by John Wiley & Sons, Inc. All rights reserved.

Published by John Wiley & Sons, Inc., Hoboken, New Jersey.

Published simultaneously in Canada.

No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning, or otherwise, except as permitted under Section 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the Publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, [phone], fax [phone], or on the web at www.copyright.com. Requests to the Publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, [phone], fax [phone], or online at http://www.wiley.com/go/permissions.

Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts in preparing this book, they make no representations or warranties with respect to the accuracy or completeness of the contents of this book and specifically disclaim any implied warranties of merchantability or fitness for a particular purpose. No warranty may be created or extended by sales representatives or written sales materials. The advice and strategies contained herein may not be suitable for your situation. You should consult with a professional where appropriate. Neither the publisher nor author shall be liable for any loss of profit or any other commercial damages, including but not limited to special, incidental, consequential, or other damages.

For general information on our other products and services or for technical support, please contact our Customer Care Department within the United States at [phone], outside the United States at [phone] or fax [phone].

Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may not be available in electronic formats. For more information about Wiley products, visit our web site at www.wiley.com.

Library of Congress Cataloging-in-Publication Data:

Large-scale computing techniques for complex system simulations / [edited by] Werner Dubitzky, Krzysztof Kurowski, Bernhard Schott.

p. cm.—(Wiley series on parallel and distributed computing ; 80)

 Includes bibliographical references and index.

 ISBN 978-0-470-59244-1 (hardback)

 1. Computer simulation. I. Dubitzky, Werner, 1958– II. Kurowski, Krzysztof, 1977– III. Schott, Bernhard, 1962–

 QA76.9.C65L37 2012

 003'.3–dc23

2011016571

eISBN 9781118130476

oISBN 9781118130506

ePub 9781118130490

MOBI 9781118130483

Foreword

The human desire to understand things is insatiable and perpetually drives the advance of society, industry, and our quality of living. Scientific inquiry and research are at the core of this desire to understand, and, over the years, scientists have developed a number of formal means of doing that research. The third means of scientific research, developed in the past 100 years after theory and experimentation, is modeling and simulation. The sharpest tool invented thus far for modeling and simulation has been the computer. It is fascinating to observe that, within the short span of a few decades, humans have developed dramatically larger and larger scale computing systems, increasingly based on massive replication of commodity elements to model and simulate complex phenomena in increasing details, thus delivering greater insights. At this juncture in computing, it looks as if we have managed to turn everything physical into its virtual equivalent, letting a virtual world precede the physical one it reflects so that we can predict what is going to happen before it happens. Better yet, we want to use modeling and simulation to tell us how to change what is going to happen. The thought is almost scary as we are not supposed to do God’s work.

We all know the early studies of ballistic trajectories and code breaking, which stimulated the development of the first computers. From those beginnings, all kinds of physical objects and natural phenomena have been captured and reflected in computer models, ranging from particles in a “simple” atom to the creation of the universe, with modeling the earth’s climate in between. Computer-based modeling systems are not just limited to natural systems, but increasingly, man-made objects are being modeled, as well. One could say that, without computers, there would not have been the modern information, communication, and entertainment industries because the heart of these industries’ products, the electronic chips, must be extensively simulated and tested before they are manufactured.

Even much larger physical products, like cars and airplanes, are also modeled by computers before they go into manufacturing. An airbus virtually assembles all of its planes’ graphically rendered parts every night to make sure they fit together and work, just like a software development project has all its pieces of code compiled and regression tested every night so that the developers can get reports on what they did wrong the day before to fix their problems. Lately, modelers have advanced to simulating even the human body itself, as well as the organizations we create: How do drugs interact with the proteins in our bodies? How can a business operate more efficiently to generate more revenue and profits by optimizing its business processes? The hottest area of enterprise computing applications nowadays is business analytics.

Tough problems require sharp tools, and imaginative models require innovative computing systems. In computing, the rallying cry from the beginning has been “larger is better”: larger computing power, larger memory, larger storage, larger everything. Our desire for more computing capacity has been insatiable. To solve the problems that computer simulations address, computer scientists must be both dreamers and greedy for more computing power at the same time. And it is all for the better: It is valuable to be able to simulate in detail a car with a pregnant woman sitting in it and to model how the side and front air bags will function when surrounded by all the car parts and involved in all kinds of abnormal turns. More accurate results require using more computing power, more data storage space, and more interconnect bandwidth to link them all together. In computing, greed is good—if you can afford it.

That is where this book will start: How can we construct infinitely powerful computing systems with just the right applications to support modeling and simulating problems at a very low cost so they can be accessible to everybody? There is no simple answer, but there are a million attempts. Of course, not all of them lead to real progress, and not all of them can be described between the two covers of a book. This book carefully selected eight projects and enlisted their thinkers to show and tell what they did and what they learned from those experiences. The reality of computing is that it is still just as much an art as it is science: it is the cleverness of how to put together the silicon and the software programs to come up with a system that has the lowest cost while also being the most powerful and easiest to use. These thinkers, like generations of inventors before them, are smart, creative people on a mission.

There have certainly been many innovations in computer simulation systems, but three in particular stand out: commodity replication, virtualization, and cloud computing. Each of these will also be explored in this book, although none of these trends have been unique to simulation.

Before commodity replication, the computing industry had a good run of complex proprietary systems, such as the vector supercomputers. But when it evolved to 30 layers in a printed circuit board and a cost that could break a regional bank, it had gone too far. After that, the power was in the replication of commodity components like the x86 chips used in PCs, employing a million of them while driving a big volume discount, then voilà, you get a large-scale system at a low(er) cost! That is what drove computing clusters and grids.

The complexity animal was then tamed via software through virtualization, which abstracts away the low-level details of the system components to present a general systems environment supporting a wide array of parallel programming environments and specialized simulations. Virtualization allows innovation in computer system components without changing applications to fit the systems. Finally, cloud computing may allow users not to have to actually own large simulation computers anymore but rather to only use computing resources as needed, sharing such a system with other users. Or even better, cloud resources can be rented from a service provider and users only pay for usage. That is an innovation all right, but it feels like we are back to the mainframe service bureau days. In this book, you will learn about many variations of innovations around these three themes as applied to simulation systems.

So, welcome to a fascinating journey into the virtual world of simulation and computing. Be prepared to be perplexed before possibly being enlightened. After all, simulation is supposed to be the state of the art when it comes to complex and large-scale computing. As they say, the rewards make the journey worthwhile.

Songnian Zhou

Toronto, Canada

March 2011

Preface

Complex systems are defined as systems with many interdependent parts that give rise to nonlinear and emergent properties determining their high-level functioning and behavior. Due to the interdependence of their constituent elements and other characteristics of complex systems, it is difficult to predict system behavior based on the “sum of their parts” alone. Examples of complex systems include human economies and societies, nervous systems, molecular interaction networks, cells and other living things, such as bees and their hives, and ecosystems, as well as modern energy and telecommunication infrastructures. Arguably, one of the most striking properties of complex systems is that conventional experimental and engineering approaches are inadequate to capture and predict the behavior of such systems. A relatively recent and more holistic approach employs computational techniques to model and simulate complex natural phenomena and complex man-made artifacts. Complex system simulations typically require considerable computing and storage resources (processing units and primary and secondary memory) as well as high-speed communication links. Supercomputers are the technology of choice to satisfy these requirements. Because supercomputers are expensive to acquire and maintain, there has been a trend to exploit distributed computing and other large-scale computing technologies to facilitate complex system simulations. Grid computing, service-oriented architectures, programmable logic arrays, and graphic processors are examples of such technologies.

The purpose of this volume is to present a representative overview of contemporary large-scale computing technologies in the context of complex system simulation applications. The book is intended to serve simultaneously as design blueprint, user guide, research agenda, and communication platform. As a design blueprint, the book is intended for researchers and technology and application developers, managers, and other professionals who are tasked with the development or deployment of large-scale computer technology to facilitate complex system applications. As a user guide, the volume addresses the requirements of modelers and scientists to gain an overview and a basic understanding of key concepts, methodologies, technologies, and tools. For this audience, we seek to explain the key concepts and assumptions of the various large-scale computer techniques and technologies, their conceptual and computational merits and limitations. We aim at providing the users with a clear understanding and practical know-how of the relevant technologies in the context of complex system modeling and simulation and the large-scale computing technologies employed to meet the requirements of such applications. As research agenda, the book is intended for computer and complex systems students, teachers, and researchers who seek to understand the state of the art of the large-scale computing technologies involved as well as their limitations and emerging and future developments. As a communication platform, the book is intended to bridge the cultural, conceptual, and technological gap among the key disciplines of complex system modeling and simulation and large-scale computing. To support this goal, we have asked the contributors to adopt an approach that appeals to audiences from different backgrounds.

Clearly, we cannot expect to do full justice to all of these goals in a single book. However, we do believe that this book has the potential to go a long way in fostering the understanding, development, and deployment of large-scale computer technology and its application to the modeling and simulation of complex systems. Thus, we hope this volume will contribute to increased communication and collaboration across various modeling, simulation, and computer science disciplines and will help to improve the complex natural and engineering systems.

This volume comprises nine chapters, which introduce the key concepts and challenges and the lessons learned from developing and deploying large-scale computing technologies in the context of complex system applications. Next, we briefly summarize the contents of the nine chapters.

Chapter 1 is concerned with an overview of some large-scale computing technologies. It discusses how in the last three decades the demand for computer-aided simulation of processes and systems has increased. In the same time period, simulation models have become increasingly complex in order to capture the details of the systems and processes being modeled. Both trends have instigated the development of new concepts aimed at a more efficient sharing of computational resources. Typically, grid and cloud computing techniques are employed to meet the computing and storage demands of complex applications in research, development, and other areas. This chapter provides an overview of grid and cloud computing, which are key elements of many modern large-scale computing environments.

Chapter 2 adopts the view of an e-infrastructure ecosystem. It focuses on scientific collaborations and how these are increasingly relying on the capability of combining computational and data resources supplied by several resource providers into seamless e-infrastructures. This chapter presents the rationale for building an e-infrastructure ecosystem that comprises national, regional, and international e-infrastructures. It discusses operational and usage models and highlights how e-infrastructures can be used in building complex applications.

Chapter 3 presents multiscale physical and astrophysical simulations on new many-core accelerator hardware. The chosen algorithms are deployed on parallel clusters using a large number of graphical processing units (GPUs) on the petaflop scale. The applications are particle-based astrophysical many-body simulations with self-gravity, as well as particle and mesh-based simulations on fluid flows, from astrophysics and physics. Strong and soft scaling are demonstrated using some of the fastest GPU clusters in China and hardware resources of cooperating teams in Germany and the United States.

Chapter 4 presents an overview of the SimWorld Agent-Based Grid Experimentation System (SWAGES). SWAGES has been used extensively for various kinds of agent-based modeling and is designed to scale to very large and complex grid environments while maintaining a very simple user interface for integrating models with the system. This chapter focuses on SWAGES’ unique features for parallel simulation experimentation (such as novel spatial scheduling algorithms) and on its methodologies for utilizing large-scale computational resources (such as the distributed server architecture designed to offset the ever-growing computational demands of administering large simulation experiments).

Chapter 5 revolves around agent-based modeling and simulation (ABMS) technologies. In the last decade, ABMS has been successfully applied to a variety of domains, demonstrating the potential of this approach to advance science, engineering, and other domains. However, realizing the full potential of ABMS to generate breakthrough research results requires far greater computing capability than is available through current ABMS tools. The Repast for High Performance Computing (Repast HPC) project addresses this need by developing a next-generation ABMS system explicitly focusing on larger-scale distributed computing platforms. This chapter’s contribution is its detailed presentation of the implementation of Repast HPC, a complete ABMS platform developed explicitly for large-scale distributed computing systems.

Chapter 6 presents an environment for the development and execution of multiscale simulations composed from high-level architecture (HLA) components. Advanced HLA mechanisms are particularly useful for multiscale simulations as they provide, among others, time management functions that enable the construction of integrated simulations from modules with different individual timescales. Using the proposed solution simplifies the use of HLA services and allows components to be steered by users; this is not possible in raw HLA. This solution separates the roles of simulation module developers from those of users and enables collaborative work. The environment is accessible via a scripting API, which enables the steering of distributed components using straightforward source code.

Chapter 7 is concerned with the data dimensions of large-scale computing. Data-intensive computing is the study of the tools and techniques required to manage and explore digital data. This chapter briefly discusses the many issues arising from the huge increase in stored digital data that we are now confronted with globally. In order to make sense of this data and to transform it into useful information that can inform our knowledge of the world around us, many new techniques in data handling, data exploration, and information creation are needed. The Advanced Data Mining and Integration Research for Europe (ADMIRE) project, which this chapter discusses in some detail, is studying how some of these challenges can be addressed through the creation of advanced, automated data mining techniques that can be applied to large-scale distributed data sets.

Chapter 8 describes a topology-aware evolutionary algorithm that is able to automatically adapt itself to different configurations of distributed computing resources. An important component of the algorithm is the use of QosCosGrid-OpenMPI, which enables the algorithm to run across computing resources hundreds of kilometers distant from one another. The authors use the evolutionary algorithm to compare models of a biological gene regulatory network which have been reverse engineered using three different systems of mathematical equations.

Chapter 9 presents a number of technologies that have been successfully integrated into a supercomputing-like e-science infrastructure called QosCosGrid (QCG). The solutions provide services for simulations such as complex systems, multiphysics, hybrid models, and parallel applications. The key aim in providing these solutions was to support the dynamic and guaranteed use of distributed computational clusters and supercomputers managed efficiently by a hierarchical scheduling structure involving a metascheduler layer and underlying local queuing or batch systems.

Werner Dubitzky

Krzysztof Kurowski

Bernhard Schott

Coleraine, Frankfurt, Poznan

May 2011

Contributors

PETER BERCZIK, National Astronomical Observatories of China, Chinese Academy of Sciences, China, and Astronomisches Rechen-Institut, Zentrum für Astronomie, University of Heidelberg, Germany; Email: [email protected]

INGO BERENTZEN, Zentrum für Astronomie, University of Heidelberg, Heidelberg, Germany; Email: [email protected]

BARTOSZ BOSAK, Poznan Supercomputing and Networking Center, Poznan, Poland; Email: [email protected]

MARIAN BUBAK, AGH University of Science and Technology, Krakow, Poland, and Institute for Informatics, University of Amsterdam, The Netherlands; Email: