Oracle GoldenGate 11g Implementer's guide - John P Jeffries - E-Book

Oracle GoldenGate 11g Implementer's guide E-Book

John P Jeffries

0,0
44,39 €

-100%
Sammeln Sie Punkte in unserem Gutscheinprogramm und kaufen Sie E-Books und Hörbücher mit bis zu 100% Rabatt.

Mehr erfahren.
Beschreibung

Data replication is an important part of any database system that is growing due to today's demand for real-time reporting and regulatory requirements. GoldenGate has recently become Oracle's strategic real-time data replication solution. Until now, very little has been written about how to implement GoldenGate in a production enterprise environment where performance, scalability, and data integrity are paramount.

Your days of dismay over the lack of documentation over Oracle GoldenGate are over.

Welcome to Oracle GoldenGate 11g Implementer's guide – a comprehensive practical book, which will deliver answers to your questions in a clear, concise style, allowing you to progress effectively in a timeline-driven environment. Based on the author's own experience, this long awaited GoldenGate administration book has all that is required to install, design, configure, and tune data replication solutions suited to every environment. Be the first to master GoldenGate's power and flexibility by reading this unique hands-on implementation companion.

Systems need to send data from one system to another in a timely manner to satisfy the ever-increasing need for speed. Regardless of whether you are a novice or an expert – or someone in between – this book will guide you through all the steps necessary to build a high-performance GoldenGate solution on Oracle11gR1. Expert users can dive into key topic areas such as performance tuning or troubleshooting, while novice users can step through the early installation and configuration chapters, later progressing to the advanced chapters.

This book is more than an implementation guide. It offers detailed real-life examples, encouraging additional thought and discussion by going beyond the manual.

With Oracle GoldenGate 11g Implementer's guide in hand, you'll be designing, installing, and configuring high-performance solutions using GoldenGate in less time than you can say "replicate"

Das E-Book können Sie in Legimi-Apps oder einer beliebigen App lesen, die das folgende Format unterstützen:

EPUB
MOBI

Seitenzahl: 307

Veröffentlichungsjahr: 2011

Bewertungen
0,0
0
0
0
0
0
Mehr Informationen
Mehr Informationen
Legimi prüft nicht, ob Rezensionen von Nutzern stammen, die den betreffenden Titel tatsächlich gekauft oder gelesen/gehört haben. Wir entfernen aber gefälschte Rezensionen.



Table of Contents

Oracle GoldenGate 11g Implementer's guide
Credits
Foreword
About the Author
About the Reviewers
www.PacktPub.com
Support files, eBooks, discount offers and more
Why Subscribe?
Free Access for Packt account holders
Instant Updates on New Packt Books
Preface
What this book covers
What you need for this book
Who this book is for
Conventions
Reader feedback
Customer support
Errata
Piracy
Questions
1. Getting Started
What is GoldenGate?
Oracle GoldenGate evolution
Oracle GoldenGate solutions
Oracle GoldenGate technology overview
The capture process (Extract)
Trail files
Data Pump
Server Collector
The Apply process (Replicat)
The Manager process
GGSCI
Process data flow
Oracle GoldenGate architecture
One-to-One
One-to-Many
Many-to-One
Cascading
Bi-directional (Active-Active)
Bi-directional (Active-Passive)
Supported platforms and databases
Oracle Goldengate topology
Process topology
The rules
Position
Statistics
Design considerations
Choosing a solution
Network
Database schema
What to Replicate?
Object mapping and data selection
Initial Load
CSN co-ordination
Trail file format
Summary
2. Installing and Preparing GoldenGate
Prerequisites
Downloading the software
Software requirements
Hardware requirements
Memory
CPU
Network
Disk
Software installation
Installing GoldenGate
The subdirectories
dirchk
dirdat
dirdef
dirpcs
dirprm
dirrpt
dirsql
dirtmp
Preparing the environment
Preparing the database for data replication
Enabling supplemental logging
Preparing the operating system
Creating the initial configuration
Creating the GoldenGate administrator
The Manager process
The Extract process
Creating and starting an Extract process
The Data Pump process
The Replicat process
Creating and starting a Replicat process
Configuration summary
Uninstalling GoldenGate from Linux/UNIX
Summary
3. Design Considerations
Replication methods
Active-active
Active-passive
Cascading
Physical Standby
Networking
Surviving network outages
Redundant networks
NIC teaming
Non-functional requirements (NFRs)
Latency
Availability
Backup and recovery
Hardware considerations
Computing architectures
Grid computing
Single server
Clusters
Machines
The x86-64 Linux Server
The Database Machine
Scaling up and out
Changed data management
Point in Time Recovery (PITR)
Oracle Recovery Manager (RMAN)
Flashback
SAN Snapshot
Summary
4. Configuring Oracle GoldenGate
GoldenGate parameters
Configuring the Initial Load
Choosing a method
File to Replicat
Extract parameters
Replicat parameters
File to database utility
Extract parameters
Replicat parameters
Direct Load
Extract parameters
Replicat parameters
Direct Bulk Load
Extract parameters
Replicat parameters
Performing the Initial Load
Example architecture
File to Replicat method
Configuring Initial Data Capture
Configuring Initial Data Delivery
Configuring Change Data Capture
Configuring Change Delivery
Testing Change Data Capture and Delivery
Stopping GoldenGate processes
More about trail files
The trail
Trail file purging
Configuring the Manager process
Summary
5. Configuration Options
Using BATCHSQL
SQL cache
Exceptions
When to use BATCHSQL
Data compression
Compressing the Data Stream
The COMPRESS option
Oracle table compression
Security features
Data encryption
Password encryption
Default method
Named method
Event Actions
Event Records
Bi-directional configuration options
Loop detection
Conflict detection
Conflict resolution
Oracle sequences
Oracle triggers
Heterogeneous environments
Specific options
Microsoft SQL Server
IBM DB2
The DEFGEN utility
DDL replication
The DDL parameter
Filtering
Mapping options
Initial setup
Known issues
Using DUMPDDL
DDL OPTIONS
Summary
6. Configuring GoldenGate for HA
GoldenGate on RAC
Shared storage
Automatic Storage Management Cluster File System (ACFS)
Database File System (DBFS)
Configuring Clusterware for GoldenGate
The Virtual IP
Creating a GoldenGate application
Increasing system resilience
GoldenGate on Exadata
Configuration
Creating the Extract process
Creating the Replicat process
Failover
Automatic failover
Manual failover
Relocating a service
Relocating a VIP
Summary
7. Advanced Configuration
Mapping your data
Column mapping
Using the COLMAP option
The COLMATCH option
Using the DEFGEN utility
Data selection and filtering
The WHERE clause
The FILTER clause
Loop detection
Active-active
Cascade
Data transformation
Truncation and extraction
Concatenation
Substitution
Case changing
Numeric conversions
Date conversions
DDL support
The SQLEXEC parameter
Data lookups
Executing stored procedures
Executing SQL
Executing DML
Handling errors
Scheduling jobs
Using and defining macros
User tokens
User Exits
Calling C routines
Sample User Exits
Source files explained
Using logic in data replication
Licensing
Summary
8. Managing Oracle GoldenGate
Command level security
The CMDSEC file
Trail file management
Managing process startup
Managing TCP/IP errors
The tcperrs file
Reporting and statistics
Monitoring errors
Monitoring latency
A bespoke solution to monitoring and reporting
Graphing the results using Microsoft Excel
Measuring throughput
Data throughput
Operation throughput
Summary
9. Performance Tuning
Before tuning GoldenGate
Online redo
Large objects (LOBs)
Base lining
Balancing the load across parallel process groups
Considerations for using parallel process groups
Splitting large tables into row ranges across process groups
The RANGE function
Adding Replicats with @RANGE function
Configuring multiple parallel process groups
Source system configuration
Parallel process parameter files
Creating extract parallel process groups
Target system configuration
Parallel process parameter files
Creating Replicat parallel process groups
Improving Replicat throughput
New releases
DBFS enhancements
New redo log OCI API
Tuning the network
Linux TCP tuning
Configuring a Bequeath connection
Summary
10. Troubleshooting GoldenGate
Troubleshooting tips
Replication not working?
The CHECKPARAMS parameter
Adjusting the start point
Altering Extract processes in RAC environments
Checking process checkpoints
Investigating network issues
TCP/IP
SQL*Net
Investigating Oracle errors
Exception handling
Creating an Exceptions handler
Viewing Exceptions
Before and after images
Handling Oracle Sequences
Using LOGDUMP
Opening files
Viewing the header record
Viewing the transaction record
Miscellaneous commands
Filtering records
Upgrading GoldenGate
Summary
A. GGSCI Commands
B. GoldenGate Installed Components
C. The Future of Oracle GoldenGate
Index

Oracle GoldenGate 11g Implementer's guide

Oracle GoldenGate 11g Implementer's guide

Copyright © 2011 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

First published: February 2011

Production Reference: 1150211

Published by Packt Publishing Ltd.

32 Lincoln Road

Olton

Birmingham, B27 6PA, UK.

ISBN 978-1-849682-00-8

www.packtpub.com

Cover Image by David Guettirrez (<[email protected]>)

Credits

Author

John P. Jeffries

Reviewers

ShuXuan Nie

Anirudh Pucha

Gavin Soorma

Development Editor

Maitreya Bhakal

Technical Editor

Neha Damle

Indexer

Rekha Nair

Editorial Team Leader

Vinodhan Nair

Project Team Leader

Lata Basantani

Project Coordinator

Vishal Bodwani

Proofreader

Aaron Nash

Graphics

Geetanjali Sawant

Production Coordinator

Alwin Roy

Cover Work

Alwin Roy

Foreword

Oracle GoldenGate is a product that covers many use cases in the business of data replication. It is not only useful for classical distributed databases, it is also useful for High Availability Architectures and especially for Data Warehouse and Decision Support Systems. Thus, the variety of techniques and methods spreads from unidirectional environments for query offloading/reporting to bidirectional or Peer-to-Peer architectures in an active-active fashion. Besides this, Oracle GoldenGate is not only available for heterogeneous Oracle-to-Oracle databases, but it also scales in heterogeneous non Oracle database environments which is also one topic in the Data Warehouse business. The flexibility of the product is amazing and the functionality manages a wide spectrum such as filtering, performing transformations, event handling, and many other options.

I am happy that John Jeffries summarized the main topics of Oracle Golden Gate in this book. Both parts, configuration & implementation as well as monitoring and performance tuning/troubleshooting, is described for developers and administrators. The book does start from a common architectural overview, shows up standard implementation steps, and explains how to manage the distributed environment.

John Jeffries has been an expert of Oracle databases for years. Besides the knowledge of distributed databases, his skills cover a wide area of the Oracle core technology. So, he is not only able to discuss the topics individually, but he also puts the individual pieces together that rounds up the overall common architecture. This work is very valuable as it is raised from a fundamental practical experience.

Volker Kuhr

Oracle Advanced Customer Support (ACS)

Team Leader for Distributed Databases

About the Author

John P.Jeffries has lived in the southeast of England for most of his life. Although predominantly based in London, he has worked in many countries around the world. He enjoys travel and is the Director of his own consultancy business: www.spirotek.co.uk.

Originally from a development background, he has worked for a number of global software companies including Oracle Corporation and Siebel Systems. His time at Siebel was spent designing and developing ETL solutions for Data Warehouses, ultimately enhancing the eBilling and Billing Analytics products with an Oracle Warehouse Builder based API. He spent six years working in Oracle Consulting in EMEA and Oracle Advanced Customer Services in the UK as a Senior Principal Consultant, earning the internal title of "The UK's Data Replication Expert". Now a freelance consultant, the author can be found onsite in many of the world's most respected financial institutions in London, consulting on Oracle GoldenGate, Streams, and Active Data Guard. With over 15 years of Oracle experience, and an OCP since Oracle 8i, the author has extensive knowledge of Oracle databases, including Enterprise Linux and RAC, coupled with the ability to design and build high performance distributed database systems. He has trained internal and external clients in Data Warehousing and Data Replication techniques, and continues to share his knowledge and experience through his own website: www.oracle11ggotchas.com.

Thank you for purchasing my book, which would not have been possible without the help and support from a number of key individuals and organizations.

Firstly I wish to thank my wife, Wendy, for both her drive and encouragement. Also the sacrifice that she has endured, putting aside other commitments to support me in my accomplishment.

Secondly, I wish to thank my friend and colleague, Paul Vale for his depth of knowledge of Oracle Streams. I worked closely with Paul for two years on an extensive data replication project where our shared knowledge and experience inspired me to write a book on Oracle GoldenGate.

Finally, I wish to thank Packt Publishing for agreeing to work with me and publish this book. Even now, I am astounded by the lack of available technical material on GoldenGate; this being another reason for writing. I trust you will find this book both interesting and informative, helping you to successfully implement an Oracle 11g GoldenGate environment.

About the Reviewers

ShuXuan Nie is a software engineer specializing in SOA and Java technologies.

ShuXuan has more than nine years of experience in IT industry that includes SOA technologies such as BPEL, ESB, SOAP, XML, and Enterprise Java technologies, Eclipse plug-ins, and other areas such as C++ cross-platform development.

Since July 2010, Shuxuan has been working in Rubiconred and focusing on helping customers solve their middleware problems.

Before joining Rubiconred, ShuXuan had been working in the Oracle Global Customer Support team and focused on helping customers solve their SOA integration problems.

Before joining Oracle, ShuXuan had been working in IBM China Software Development Lab for four years as a staff software engineer, participating in several complex products on IBM Lotus Workplace, Webshpere, and Eclipse platform; and then joined the Australia Bureau of Meteorology Research Center, responsible for implementation of Automated Thunderstorm Interactive Forecast System for Aviation and Defence.

ShuXuan holds a MS in Computer Science from Beijing University of Aeronautics and Astronautics.

Anirudh Pucha is a SOA and Cloud Computing Strategist working on Fusion Middleware Integration products in Oracle Corporation. He has been working in the SOA Integration space for over five years focusing on .NET, J2EE, Oracle SOA Suite, BPEL, ESB, Adapters, BAM, Business Rules, BPM. He is one of the top contributors in the Oracle SOA and BPEL Technology Forums. He is a certified webMethods Developer and Oracle Certified Professional. He is an Ex-Microsoft Student Ambassador and Platform Evangelist appointed by Microsoft Corporation. He is the Asst.Manager of DNUGHS (DotNet Users Group of Hyd, supported by Microsoft Corp). He is a INETA (International .NET Association) – APAC Volunteer, The SPOKE member, Culminis member and a member of several renowned NGOs. He is a speaker at various conferences, bar-camps, and a guest lecturer for several universities and colleges. He is a native of Hyderabad (India), certified Latin American Dancer, professional singer, dubbing artist, cricketer, cartoonist, sculpturist, and a fashion model. He can be contacted on LinkedIn at http://in.linkedin.com/in/anirudhpucha and his website at http://www.anirudhpucha.tk/.

I am thankful to GOD, my mother – Smt. Pucha Annapurna, father – Shri. Pucha Hanumanth Rao, cute nephew – Karthik Achintya and niece – Pallavi Shriya.

Gavin Soorma is an Oracle Certified Master with over 17 years of experience. He also is an Oracle Certified Professional (versions 7.3, 8i, 9i, 10g, and 11g) as well as an Oracle Certified Expert in 10g RAC.

Gavin is a regular presenter at various Oracle conferences and seminars having presented several papers at the IOUG, South African Oracle User's Group, Oracle Open World, and the Australian Oracle User Group. Recently, at this year's AUSOUG held in Melbourne and Perth he presented a paper on Oracle GoldenGate titled 'Real Time Access to Real Time Information'.

He is currently employed as a Senior Principal Consultant for an Oracle solution provider, OnCall DBA based in Perth, Western Australia. Prior to this, he held the position of Senior Oracle DBA and Team Lead with Bank West in Perth. Before migrating to Australia, Gavin worked for Emirates Airline Group IT in Dubai for over 15 years where he held the position of Technical Team Manager, Databases.

He has also written a number of tutorials and notes on Oracle GoldenGate which can be accessed via his personal blog website http://gavinsoorma.com.

www.PacktPub.com

Support files, eBooks, discount offers and more

You might want to visit www.PacktPub.com for support files and downloads related to your book.

Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.PacktPub.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at <[email protected]> for more details.

At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a range of free newsletters and receive exclusive discounts and offers on Packt books and eBooks.

http://PacktLib.PacktPub.com

Do you need instant solutions to your IT questions? PacktLib is Packt's online digital book library. Here, you can access, read and search across Packt's entire library of books. 

Why Subscribe?

Fully searchable across every book published by PacktCopy & paste, print and bookmark contentOn demand and accessible via web browser

Free Access for Packt account holders

If you have an account with Packt at www.PacktPub.com, you can use this to access PacktLib today and view nine entirely free books. Simply use your login credentials for immediate access.

Instant Updates on New Packt Books

Get notified! Find out when new books are published by following @PacktEnterprise on Twitter, or the Packt Enterprise Facebook page.

Preface

Data replication is an important part of any database system that is growing due to today's demand for real-time reporting and regulatory requirements. GoldenGate has recently become Oracle's strategic real-time data replication solution. Until now, very little has been written about how to implement GoldenGate in a production enterprise environment where performance, scalability, and data integrity are paramount.

Your days of dismay over the lack of documentation over Oracle GoldenGate are over.

Welcome to Oracle GoldenGate 11g Implementer's guide— a comprehensive, practical book that will deliver answers to your questions in a clear, concise style, allowing you to progress effectively in a timeline-driven environment. Based on the author's own experience, this long awaited GoldenGate administration book has all that is required to install, design, configure, and tune data replication solutions suited to every environment. Be the first to master GoldenGate's power and flexibility by reading this unique hands-on implementation companion.

Computers need to send data from one system to another in a timely manner to satisfy the ever-increasing need for speed. Regardless of whether you are a novice or an expert – or someone in between – this book will guide you through all the steps necessary to build a high-performance GoldenGate solution on Oracle11gR1. Expert users can dive into key topic areas such as performance tuning or troubleshooting, while novice users can step through the early installation and configuration chapters, later progressing to the advanced chapters.

This book is more than an implementation guide. It offers detailed real-life examples, encouraging additional thought and discussion by going beyond the manual.

With Oracle GoldenGate 11g Implementer's guide in hand, you'll be designing, installing, and configuring high-performance solutions using GoldenGate within minutes.

What this book covers

Chapter 1, Getting Started, provides an introduction to Oracle GoldenGate, inspiring thought by describing the key components, processes, and considerations required to build and implement a GoldenGate solution.

The topics covered the evolution of GoldenGate Software, including the architecture behind the technology followed by the solutions GoldenGate can provide, coupled with effective design.

Chapter 2, Installing and Preparing GoldenGate, walks you through the individual tasks needed to complete an end-to-end GoldenGate installation, including the download of the software through to creating a simple data replication environment.

Chapter 3, Design Considerations, addresses some of the issues that influence the decision making process when designing a GoldenGate solution. These include design considerations for both performance and high availability. Here you can choose the appropriate hardware and topology to deliver a fast, robust and scalable solution.

Chapter 4, Configuring Oracle GoldenGate, initially discusses the main GoldenGate parameters, and provides a methodical approach to the configuration process, stepping through each task to give the depth of information necessary to successfully implement GoldenGate on Oracle 11g. By providing the building blocks, this chapter forms the basis for more complex configurations.

Chapter 5, Configuration Options, looks at the available options that allow your configuration to extend in functionality and performance. We start with performance enhancements, later exploring the security features, including data compression and encryption, and finally discussing the options available to implement DDL replication.

Chapter 6, Configuring GoldenGatefor HA teaches you how to configure Oracle GoldenGate in a RAC environment and explores the various components that effectively enable HA for data replication and integration.

Chapter 7, Advanced Configuration, gives you a deeper understanding of GoldenGate configuration. By the time you are done with this chapter, you will be able to explore and realize each parameter specification and further develop your GoldenGate configuration.

Chapter 8, Managing Oracle GoldenGate, focuses on the management features already built in to the GoldenGate Command Interpreter (GGSCI). We discuss and implement a number of utilities, including tips and tricks that allow you to manage your GoldenGate environment effectively at no extra cost. This includes a Perl script that will format the text output from the GGSCI "INFO ALL" command so that the data can be loaded into MS Excel for graphing and trend analysis.

Chapter 9, Performance Tuning, focuses on the main areas that lend themselves to tuning, especially parallel processing and load balancing, enabling high data throughput and very low latency. Performance enhancing new features that are available in the next version of GoldenGate are also discussed and evaluated.

Chapter 10, Troubleshooting GoldenGate, provides a troubleshooting guide for Oracle GoldenGate. We address some of the common issues faced by the implementer, followed by a discussion of the tools and utilities available to help resolve them. This includes the creation of an Exception Handler that automatically logs the details of a failed transaction for further analysis and resolution.

This book has three appendices; their purpose is to provide a "quick reference" as well as address any subjects and terminology not addressed in the previous chapters.

Appendix A, GGSCI Commands, provides a quick reference guide to all the available GoldenGate Software Command Interface (GGSCI) commands.

Appendix B, GoldenGate Installed Components, lists the GoldenGate installed components and their description.

Appendix C, The Future of Oracle GoldenGate, discusses Oracle's strategic approach to data replication and integration. The latest release of GoldenGate including its sister products are described along with links to official documentation and forums on the Oracle website.

What you need for this book

As a minimum you will require the following elements to run the code featured in this book. Starting with the hardware, you can configure a simple GoldenGate solution on just one personal computer acting as a database server, running Oracle or Redhat Enterprise Linux x86 version 5.2. This can be native Linux or Linux virtual machine hosted on MS Windows XP.

Ideally, another PC is required as a client that supports terminal emulation software for logging onto the database server, plus MS Excel for graphing and analysis of statistics.

The following list provides the minimum hardware specification for each PC:

1 x 2.5 GHz Dual Core Intel based 32 bit architecture CPU2 GB of physical memory250 GB hard disk drive for the database server150 GB hard disk drive for the MS Windows client1 x 100Mb/s Ethernet adapter

To implement a simple point to point data replication solution, you will need two Oracle 11g Release 1 databases. Both source and target databases can reside on the same machine. Please refer to the pre-installation requirements found on the Oracle Website at the following link: http://download.oracle.com/docs/cd/B28359_01/install.111/b32002/pre_install.htm

The following list provides the software requirements:

Oracle or Redhat Enterprise Linux Server release 5.2 - 32-bitOracle Server 11.1.0.6 on Oracle Enterprise Linux 5.0 32-bit, patched to 11.1.0.7 (this is required for the two Oracle databases, source & target)Oracle GoldenGate V10.4.0.x for Oracle 11g on Oracle Enterprise Linux 5.0 32-bitMS Windows XP 32-bit Professional Edition, Service Pack 3 with MS Excel 2007 installedPerl v5.8.8 (included with OEL/RHEL 5.2)PuTTY 0.56 for MS Windows XP (terminal emulation software that supports SSH protocol)WinSCP Version 3.8.2 for MS Windows XP (file transfer software that supports SFTP protocol)

Who this book is for

This book is for Solution Architects and Database Architects who wish to learn about the functionality and efforts required in implementing a data replication, migration or integration solution using GoldenGate. It is also designed for System Administrators and Database Administrators who want to implement, or those who have already implemented, GoldenGate and who want to explore its advanced features. An intermediate understanding of Oracle database technology is assumed.

Conventions

In this book, you will find a number of styles of text that distinguish between different kinds of information. Here are some examples of these styles, and an explanation of their meaning.

Code words in text are shown as follows: "We can include other contexts through the use of the include directive."

A block of code is set as follows:

EXTRACT EPMP01 PASSTHRU RMTHOST dbserver2, MGRPORT 7809 RMTTRAIL ./dirdat/ta TABLE SRC.DEPT; TABLE SRC.EMP;

When we wish to draw your attention to a particular part of a code block, the relevant lines or items are set in bold:

EXTRACT EPMP01 PASSTHRU RMTHOST dbserver2, MGRPORT 7809 RMTTRAIL ./dirdat/ta TABLE SRC.DEPT; TABLE SRC.EMP;

Any command-line input or output is written as follows:

cd /home/oracle/ggs

New terms and important words are shown in bold. Words that you see on the screen, in menus or dialog boxes for example, appear in the text like this: "clicking the Next button moves you to the next screen".

Note

Warnings or important notes appear in a box like this.

Tip

Tips and tricks appear like this.

Reader feedback

Feedback from our readers is always welcome. Let us know what you think about this book—what you liked or may have disliked. Reader feedback is important for us to develop titles that you really get the most out of.

To send us general feedback, simply send an e-mail to <[email protected]>, and mention the book title via the subject of your message.

If there is a book that you need and would like to see us publish, please send us a note in the SUGGEST A TITLE form on www.packtpub.com or e-mail <[email protected]>.

If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, see our author guide on www.packtpub.com/authors.

Customer support

Now that you are the proud owner of a Packt book, we have a number of things to help you to get the most from your purchase.

Tip

Downloading the example code for this book

You can download the example code files for all Packt books you have purchased from your account at http://www.PacktPub.com. If you purchased this book elsewhere, you can visit http://www.PacktPub.com/support and register to have the files e-mailed directly to you.

Errata

Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you find a mistake in one of our books—maybe a mistake in the text or the code—we would be grateful if you would report this to us. By doing so, you can save other readers from frustration and help us improve subsequent versions of this book. If you find any errata, please report them by visiting http://www.packtpub.com/support, selecting your book, clicking on the errata submission form link, and entering the details of your errata. Once your errata are verified, your submission will be accepted and the errata will be uploaded on our website, or added to any list of existing errata, under the Errata section of that title. Any existing errata can be viewed by selecting your title from http://www.packtpub.com/support.

Piracy

Piracy of copyright material on the Internet is an ongoing problem across all media. At Packt, we take the protection of our copyright and licenses very seriously. If you come across any illegal copies of our works, in any form, on the Internet, please provide us with the location address or website name immediately so that we can pursue a remedy.

Please contact us at <[email protected]> with a link to the suspected pirated material.

We appreciate your help in protecting our authors, and our ability to bring you valuable content.

Questions

You can contact us at <[email protected]> if you are having a problem with any aspect of the book, and we will do our best to address it.

Chapter 1. Getting Started

The objective of this chapter is to get you started using Oracle GoldenGate 10.4. We will discuss the history and evolution of GoldenGate Software, its success in the market and ultimate Oracle acquisition. You will become accustomed with the concepts of data replication and how GoldenGate provides enterprise-wide solutions to address the business requirements.

Although an introduction, this chapter is designed to inspire thought by drilling into the key components, processes, and considerations required to build and implement a GoldenGate solution.

In this chapter, we will discuss the following points surrounding GoldenGate:

The evolution of GoldenGate softwareThe technology and architectureThe solutions offered by GoldenGateThe architecture and topology of GoldenGate, plus design considerationsThe supported platform and database versions

Let's begin by learning what GoldenGate is and what you can expect from this book.

What is GoldenGate?

Oracle GoldenGate is Oracle's strategic solution for real time data integration. GoldenGate software enables mission critical systems to have continuous availability and access to real-time data. It offers a fast and robust solution for replicating transactional data between operational and analytical systems.

Oracle GoldenGate captures, filters, routes, verifies, transforms, and delivers transactional data in real-time, across Oracle and heterogeneous environments with very low impact and preserved transaction integrity. The transaction data management provides read consistency, maintaining referential integrity between source and target systems.

This book aims to illustrate through example, providing the reader with solid information and tips for implementing GoldenGate software in a production environment.

In this book, we will not be making direct comparisons between Oracle GoldenGate and Oracle Streams. At the time of writing, Oracle is leveraging the advantages of GoldenGate by enhancing the product whilst continuing to fully support Streams.

As a competitor to Oracle GoldenGate, data replication products and solutions exist from other software companies and vendors. These are mainly storage replication solutions that provide fast point in time data restoration. The following is a list of the most common solutions available today:

EMC SRDF and EMC RecoverPointIBM PPRC and Global Mirror (known together as IBM Copy Services)Hitachi TrueCopyHewlett-Packard Continuous Access (HP CA)Symantec Veritas Volume Replicator (VVR)DataCore SANsymphony and SANmelodyFalconStor Replication and MirroringCompellent Remote Instant Replay

Data replication techniques have improved enormously over the past 10 years and have always been a requirement in nearly every IT project in every industry. Whether for Disaster Recovery (DR), High Availability (HA), Business Intelligence (BI), or even regulatory reasons, the requirements and expected performance have also increased, making the implementation of efficient and scalable data replication solutions a welcome challenge.

Oracle GoldenGate evolution

GoldenGate Software Inc was founded in 1995. Originating in San Francisco, the company was named after the famous Golden Gate Bridge by its founders, Eric Fish and Todd Davidson. The tried and tested product that emerged quickly became very popular within the financial industry. Originally designed for the fault tolerant Tandem computers, the resilient and fast data replication solution was in demand. The banks initially used GoldenGate software in their ATM networks for sending transactional data from high street machines to mainframe central computers. The data integrity and guaranteed zero data loss is obviously paramount and plays a key factor. The key architectural properties of the product are as follows:

Data is sent in "real time" with sub-second speed.Supports heterogeneous environments across different database and hardware types. "Transaction aware" —maintaining its read-consistent and referential integrity between source and target systems.High performance with low impact; able to move large volumes of data very efficiently while maintaining very low lag times and latency.Flexible modular architecture.Reliable and extremely resilient to failure and data loss. No single point of failure or dependencies, and easy to recover.

Oracle Corporation acquired GoldenGate Software in September 2009. Today there are more than 500 customers around the world using GoldenGate technology for over 4000 solutions, realizing over $100 million in revenue for Oracle.

Oracle GoldenGate solutions

Oracle GoldenGate provides five data replication solutions:

High Availability
Live Standby for an immediate fail-over solution that can later re-synchronize with your primary source.Active-Active solutions for continuous availability and transaction load distribution between two or more active systems.
Zero-Downtime Upgrades and Migrations
Eliminates downtime for upgrades and migrations.
Live Reporting
Feeding a reporting database so as not to burden the source production systems with BI users or tools.
Operational Business Intelligence (BI)
Real-time data feeds to operational data stores or data warehouses, directly or via ExtractTransformandLoad (ETL) tools.
Transactional Data Integration
Real-time data feeds to messaging systems for business activity monitoring, business process monitoring, and complex event processing.Uses event-driven architecture and service-oriented architecture (SOA).

The following diagram shows the basic architecture for the various solutions available from GoldenGate software:

We have discovered there are many solutions where GoldenGate can be applied. Now we can dive into how GoldenGate works, the individual processes, and the data flow that is adopted for all.

Oracle GoldenGate technology overview

Let's take a look at GoldenGate's fundamental building blocks; the Capture process, Trail files, Data pump, Server collector, and Apply processes. In fact, the order in which the processes are listed depicts the sequence of events for GoldenGate data replication across distributed systems. A Manager process runs on both the source and the target systems that "oversee" the processing and transmission of data.

All the individual processes are modular and can be easily decoupled or combined to provide the best solution to meet the business requirements. It is normal practice to configure multiple Capture and Apply processes to balance the load and enhance performance. You can read more about this in Chapter 9, Performance Tuning.

Filtering and transformation of the data can be done at either the source by the Capture or at the target by the Apply processes. This is achieved through parameter files, and is explained in detail in Chapter 3, Configuring Oracle GoldenGate.

The capture process (Extract)

Oracle GoldenGate's capture process, known as Extract, obtains the necessary data from the databases' transaction logs. For Oracle, these are the online redo logs that contain all the data changes made in the database. GoldenGate does not require access to the source database and only extracts the committed transactions from the online redo logs. It can however, read archived redo logs to extract the data from long running transactions, but more about that later in the book.

The Extract process will regularly checkpoint its read and write position, typically to a file. The checkpoint data insures GoldenGate can recover its processes without data loss in the case of failure.

The Extract process can have one the following statuses:

STOPPEDSTARTINGRUNNINGABENDED

The ABENDED status stems back to the Tandem computer, where processes either stop (end normally) or abend (end abnormally). Abend is short for abnormal end.

Trail files

To replicate transactional data efficiently from one database to another, Oracle GoldenGate converts the captured data into a Canonical Format which is written to trail files, both on the source and the target system. The provision of source and target trail files in the GoldenGates architecture eliminates any single point of failure and ensures data integrity is maintained. A dedicated checkpoint process keeps track of the data being written to the trails on both the source and target for fault tolerance.

It is possible to configure GoldenGate not to use trail files on the source system and write data directly from the database's redo logs to the target server data collector. In this case, the Extract process sends data in large blocks across a TCP/IP network to the target system. However, this configuration is not recommended due to the possibility of data loss occurring during unplanned system or network outages. Best practice states, the use of local trail files would provide a history of transactions and support the recovery of data for retransmission via a Data Pump.

Data Pump

When using trail files on the source system, known as a local trail, GoldenGate requires an additional Extract process called Data Pump that sends data in large blocks across a TCP/IP network to the target system. As previously sated, this is best practice and should be adopted for all Extract configurations.

Server Collector

The Server Collector process runs on the target system and accepts data from the source (Extract/Data Pump). Its job is to reassemble the data and write it to a GoldenGate trail file, known as a remote trail.

The Apply process (Replicat)

The Apply process, known in GoldenGate as Replicat, is the final step in the data delivery. It reads the trail file and applies it to the target database in the form of DML (deletes, updates and inserts) or DDL*. (database structural changes). This can be concurrent with the data capture or performed later.

The Replicat process will regularly checkpoint its read and write position, typically to a file. The checkpoint data ensures that GoldenGate can recover its processes without data loss in the case of failure.

The Replicat process can have one of the following statuses:

STOPPEDSTARTINGRUNNINGABENDED

* DDL is only supported in unidirectional configurations and non-heterogeneous (Oracle to Oracle) environments.

The Manager process

The Manager process runs on both source and target systems. Its job is to control activities such as starting, monitoring, and restarting processes; allocating data storage; and reporting errors and events. The Manager process must exist in any GoldenGate implementation. However, there can be only one Manager process per Changed Data Capture configuration on the source and target.

The Manager process can have either of the following statuses:

STOPPEDRUNNING

GGSCI

In addition to the processes previously described, Oracle GoldenGate 10.4 ships with its own command line interface known as GoldenGateSoftwareCommandInterface (GGSCI). This tool provides the administrator with a comprehensive set of commands to create, configure, and monitor all GoldenGate processes. You will become very familiar with GGSCI as you continue through this book.