28,99 €
Mehr erfahren.
- Herausgeber: Packt Publishing
- Kategorie: Wissenschaft und neue Technologien
- Sprache: Englisch
Control-M is one of the most widely used enterprise class batch workload automation platform. With a strong knowledge of Control-M, you will be able to use the tool to meet ever growing batch needs. There has been no book that can guide you to implement and manage this powerful tool successfully... until now. With this book you will quickly master Control-M and be able to call yourself "a Control-M" specialist!
"BMC Control-M 7: A Journey from Traditional Batch Scheduling to Workload Automation" will lead you into the world of Control-M and guide you to implement and maintain a Control-M environment successfully. By mastering this workload automation tool, you will see new opportunities opening up before you.
With this book you will be able to take away and put into practice knowledge from every aspect of Control-M ñ implementation, administration, design and management of Control-M job flows, and more importantly how to move into workload automation and let batch processing utilize the cloud.
You will start off with batch processing and workload automation, and then get an understanding of how Control-M meets these needs. Then we will look more in depth at the technical details of Control-M, and finally look at how to work with it to meet critical business needs. Throughout the book, you will learn important concepts and features, as well as learn from the Author's experience, accumulated over many years. By the end of the book you will be set up to work efficiently with this tool and also understand how to utilize the latest features of Control-M.
Das E-Book können Sie in Legimi-Apps oder einer beliebigen App lesen, die das folgende Format unterstützen:
Seitenzahl: 678
Veröffentlichungsjahr: 2012
Ähnliche
Table of Contents
BMC Control-M 7: A Journey from Traditional Batch Scheduling to Workload Automation
BMC Control-M 7: A Journey from Traditional Batch Scheduling to Workload Automation
Copyright © 2012 Packt Publishing
All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.
Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book.
Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.
First published: October 2012
Production Reference: 1041012
Published by Packt Publishing Ltd.
Livery Place
35 Livery Street
Birmingham B3 2PB, UK.
ISBN 978-1-84968-256-5
www.packtpub.com
Cover Image by Anvar Khodzhaev (<[email protected]>)
Credits
Author
Qiang Ding
Reviewers
Bentze Perlmutter
Robert Stinnett
Acquisition Editor
Dhwani Devater
Technical Editor
Lubna Shaikh
Copy Editors
Brandt D'mello
Insiya Morbiwala
Laxmi Subramanian
Project Coordinator
Vishal Bodwani
Proofreaders
Lesley Harrison
Lynda Sliwoski
Indexer
Rekha Nair
Graphics
Nilesh R. Mohite
Manu Joseph
Production Coordinator
Arvindkumar Gupta
Cover Work
Arvindkumar Gupta
About the Author
Qiang Ding (Melbourne, Australia) has been working within the Control-M space for more than a quarter of his life. During his early days at BMC Software, Qiang resolved countless number of critical technical issues for Control-M customers around the world from Fortune 500 companies to government organizations. In recent years, Qiang has travelled hundreds thousands of miles around Australia and the North AP area to help many organizations to design, manage, and optimize their batch workload automation environment and to extend his passion to others by delivering Control-M trainings to end users and BMC Partners.
Currently Qiang is temporary living in Sydney and working on a enterprise wide Control-M migration and Consolidation project for a major Australian bank. He enjoys working with other experts in the field and is constantly involved in finding ways for making improvements to the batch environment that he works on.
Acknowledgement
There are many people that I would like to thank for their contribution to the creation of this book and to those who have reviewed, proofread, commented, and provided quotes.
On a greater scale, I would like to thank Bentze Perlmutter and Graeme Byrnes, who originally taught me every technical detail of Control-M, followed by Bruce Roberts, who in the recent years embraced my Control-M knowledge from a pure technical level into business level. I also would like to thank people who I worked for and worked with in the past few years, those who had a faith in me and gave me the opportunity and trust, including Allen Lee, Amy You, Angel Wong, Bao Ling, Louis Cimiotti, Chris Cunningham, Craig Taprell, Curtis Eddington, David Timms, Digby Pritchard, Doug Vail, Ian Jones, Jason St. Clair, Jeffrey Merriel, Jim Darragh, Matthew Sun, Min Yuan, Moshe Miller, Rabin Sarkar, Rick Brown, Shaun Kimpton, Stephen Donnelly, Tom Geva, Tristan Gutsche, Xianhua Peng, Yuan Yuan, and Ze'ev Gross. Last but not least, I would like to thank my friend Mike Palmer who inspired me and guided me in every aspect of my life.
For all those who have provided support and guidance over the years and if I have yet to mention your name, I sincerely apologize.
About the Reviewers
Bentze Perlmutter has 15 years of IT experience, working in various companies and holding positions in operations, QA, technical support, engineering, systems management, and production control.
His main area of expertise is batch and workload automation using tools such as Control-M and AutoSys. He has worked on complex projects requiring evaluation of solutions, design and planning, implementation, administration, and on-going support within large organizations, mainly in the financial industry.
Robert Stinnett has worked with automation systems on various platforms from mainframe to distributed since 1992. He has been using Control-M since 2003 when he was the lead for bringing it to CARFAX, the leading provider and pioneer of vehicle history reports, where he has worked for the past 10 years.
Robert is active in many Control-M communities, has given presentations at various conferences on the capabilities and cost-benefits of using an automated workload management platform, and has written a number of open-source utilities to help take advantage of and extend Control-M's capabilities on the distributed side.
One of the next big things he sees for automation systems is their integration with Cloud. He is currently working on projects that explore how Cloud can be used for providing capacity on demand as well as providing redundancy and scalability to existing automation and scheduling implementations.
Robert is also an active member of the Computer Measurement Group where he sees the role of automation in IT as one of the major reasons to have a sound performance and capacity management program in place to help manage the continuing technological evolution taking place in businesses.
He can be reached at <[email protected]>.
www.PacktPub.com
Support files, eBooks, discount offers and more
You might want to visit www.PacktPub.com for support files and downloads related to your book.
Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.PacktPub.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at <[email protected]> for more details.
At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a range of free newsletters and receive exclusive discounts and offers on Packt books and eBooks.
http://PacktLib.PacktPub.com
Do you need instant solutions to your IT questions? PacktLib is Packt's online digital book library. Here, you can access, read and search across Packt's entire library of books.
Why Subscribe?
Free Access for Packt account holders
If you have an account with Packt at www.PacktPub.com, you can use this to access PacktLib today and view nine entirely free books. Simply use your login credentials for immediate access.
Instant Updates on New Packt Books
Get notified! Find out when new books are published by following @PacktEnterprise on Twitter, or the Packt Enterprise Facebook page.
Preface
Control-M is one of the world's most wildely used enterprise class batch workload automation product produced by BMC Software. With a strong knowledge of Control-M, you will be able to use the tool to meet ever growing batch processing needs. However, there is no book that can guide you to implement and manage this powerful tool successfully until now. With this book, you will quickly master Control-M, and be able to call yourself "a Control-M" specialist!
This book will lead you into the world of Control-M, and guide you to implement and maintain a Control-M environment successfully. By mastering this workload automation tool, you will see new opportunities opening up before you.
With this book, you will be able to take away and put into practice knowledge from every aspect of Control-M implementation, administration, design, and management of Control-M job flows, and more importantly how to move into workload automation, and let batch processing utilize the cloud.
You will start off with the history and concept of batch processing and how recenty it got evolved into workload automation, and then get an understanding of how Control-M fits into the big picture. Then we will look more in depth at the technical details of the tool - How to plan, install, use, as well as manage a Control-M environment, and finally look at how to leavage the tool to meet the already sophiscated and ever growing business demand. Throughout the book, you will learn important concepts and features of Control-M through detailed explainations and examples, as well as learn from the author's experience, accumulated over many years. By the end of the book, you will be set up to work efficiently with this tool, and also understand how to utilize the latest features of Control-M.
What this book covers
Chapter 1, Get To Know the Concept, gives a good understanding of the concept of batch processing and centralized enterprise scheduling what were the challenges and why they exist today. Besides that, it also provides an overall view of the latest concept workload automation.
Chapter 2, Exploring Control-M, gives an overview of the features of Control-M, important concepts, and reviews the architecture of Control-M.
Chapter 3, Building the Control-M Infrastructure, introduces the concept of the "Three Ages" to archive workload automation, and then looks at the different requirements and challenges at each stage. It also talks about the sizing and the technical considerations that are necessary for building a solid batch infrastructure. Finally, it shows how to get started into the technical details and prepare machines and the environment for the Control-M implementation.
Chapter 4, Creating and Managing Batch Flows with Control-M GUI, looks at the important job scheduling concepts of Control-M in depth and applies them by defning some simple jobs in Control-M Desktop, and manages them using the Control-M/EM GUI client. Then it goes one step further by defning a complete batch fow to meet a scheduling requirement.
Chapter 5, Administrating the Control-M Infrastructure, starts with installing the additional Control-M components BIM and Forecast, followed by discussing the tasks involved in expanding and updating the Control-M environment. It talks about the different methods of performing regular installation tasks, such as applying fix packs and installing Control-M/EM GUI clients, as well as demonstrates how to define Agentless remote hosts in both Linux and Windows environments. Towards the end, it explains how to perform some must-known administration tasks, including stop/start Control-M components, define Control-M/EM user authorizations, and customize the GUI.
Chapter 6, Advanced Batch Scheduling and Management, shows to add more jobs to Control-M by bulk load jobs from the crontab. It explains how to transform file processing job flow from time-based scheduling into event-driven scheduling, and improve some part of the job flow by using additional Control Modules. After defining the jobs, it revisits the Control-M/EM GUI client to discover more GUI features, such as advanced functionalities offered in ViewPoints and archived ViewPoints. It also gives an overview of how to use BIM and Forecast to proactively monitor jobs and estimate potential impacts, by creating What-if scenarios. Towards the end, it visits the Reporting Facility, takes a look at each available report type, and discusses how to automate reporting.
Chapter 7, Beyond Everyday Administration, focuses on the administration side of Control-M but in more depth. It starts with looking at the command-line utilities that can be used to affect the active environment. More importantly, it reviews the different security options provided by Control-M, as well as a demonstrates the Control-M mirroring and failover. After having secured the environment, it took us to the new level by perfecting the Control-M environment.
Chapter 8, Road to Workload Automation, makes a number of improvements to the file processing job flow from both the integration and performance aspects, by using the cutting-edge Control-M add-on features. These include enabling the job fow to have exposure to external applications, by using the BPI web service interface, integration with ESB with BPI web service and message queue jobs, rendering the processing truly parallel, and implementing load balancing. Towards the end, it turns our processing flow into workloads, and taps into the power of cloud computing to for limitless processing.
Who this book is for
This book is suitable for professionals who are beginning to use Control-M, but who also have some general IT experience, such as knowing the concepts of computer system architecture, operating systems, databases, basic computer networking. Some entry level skills in scripting languages will be of help along the way.
Also for those who are from the mainframe environment or moving from other schedulers to Control-M, you can use this book as a starting point.
Conventions
In this book, you will find a number of styles of text that distinguish between different kinds of information. Here are some examples of these styles, and an explanation of their meaning.
Code words in text are shown as follows: "By 12:00am, there are ten orders generated in total. Program PROCESS_ORDER is set to trigger at this time of the day to process them".
A block of code is set as follows:
Any command-line input or output is written as follows:
New terms and important words are shown in bold. Words that you see on the screen, in menus or dialog boxes for example, appear in the text like this: " One to one relationship simply means the jobs run one after another, for example, when Job A is completed, then Job B starts.".
Note
Warnings or important notes appear in a box like this.
Tip
Tips and tricks appear like this.
Reader feedback
Feedback from our readers is always welcome. Let us know what you think about this book — what you liked or may have disliked. Reader feedback is important for us to develop titles that you really get the most out of.
To send us general feedback, simply send an e-mail to <[email protected]>, and mention the book title through the subject of your message.
If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, see our author guide on www.packtpub.com/authors.
Customer support
Now that you are the proud owner of a Packt book, we have a number of things to help you to get the most from your purchase.
Errata
Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you find a mistake in one of our books — maybe a mistake in the text or the code — we would be grateful if you would report this to us. By doing so, you can save other readers from frustration and help us improve subsequent versions of this book. If you find any errata, please report them by visiting http://www.packtpub.com/support, selecting your book, clicking on the errata submission form link, and entering the details of your errata. Once your errata are verified, your submission will be accepted and the errata will be uploaded to our website, or added to any list of existing errata, under the Errata section of that title.
Piracy
Piracy of copyright material on the Internet is an ongoing problem across all media. At Packt, we take the protection of our copyright and licenses very seriously. If you come across any illegal copies of our works, in any form, on the Internet, please provide us with the location address or website name immediately so that we can pursue a remedy.
Please contact us at <[email protected]> with a link to the suspected pirated material.
We appreciate your help in protecting our authors, and our ability to bring you valuable content.
Questions
You can contact us at <[email protected]> if you are having a problem with any aspect of the book, and we will do our best to address it.
Chapter 1. Get to Know the Concept
Before we dive deep into the concept of Control-M, let's relax a little bit by beginning with a brief lesson on the history of batch processing. In the first chapter, we will be looking at the basic fundamentals of batch processing and the ever growing technical and business requirements, as well as related challenges people are facing in today's IT environment. Based on that, we will look at how can we overcome these difficulties by using centralized enterprise scheduling platforms, and discuss the features and benefits of those platforms. Finally, we will get into the most exciting part of the chapter, talking about a brand new concept, that is, workload automation.
By keeping these key knowledge points in mind, you will find it easy to understand the purpose of each Control-M feature later on in the book. More importantly, adopting the correct batch concepts will help you build an efficient centralized batch environment and be able to use Control-M in the most effective way in the future.
By the end of this chapter, you will be able to:
Introduce batch processing
We hear about hot IT topics everyday, everywhere. Pick up a tech magazine, visit an IT website, or subscribe to a weekly newsletter and you will see topics about cloud computing, SOA, BPM/BPEL, data warehouse, ERP — you name it! Even on TV and at the cinemas, you may see something such as an "Iron Man 2 in theatres soon + Sun/Oracle in data centre now" commercial. In the recent years, IT has become a fashion more than simply a technology, but how often do you hear the words "batch processing" mentioned in any of the articles or IT road shows?
The history of batch processing
Batch processing is not a new IT buzz word. In fact, it has been a major IT concept since the very early stage of electronic computing. Unlike today, where we can run programs on a personal computer whenever required and expect an instant result, the early mainframe computers could handle non-interactive processing only.
In the beginning, punched cards were used as the media for storing data (refer to the following). Mainframe system programmers were required to store their program and input data onto these cards by punching a series of dots and pass them to the system operators for processing. Each time the system operators had to stack up the program cards followed by the input data cards onto a special card reader so the mainframe computer could load the program into memory and process the data. The execution could run for hours or days and it would stop only when the entire process was complete or in case an error occurred. Computer processing power was expensive at that time. In order to minimize a computer's idle time and improve efficiency, the input data for each program was normally accumulated in large quantities and then queued up for processing. In this manner, lots data could be processed at once, rather than frequently re-stacking the program card multiple times for ea ch small amount of input data. Therefore, the process was called batch processing.
This is a file from the Wikimedia Commons. Commons is a freely licensed media file repository (http://en.wikipedia.org/wiki/File:Blue-punch-card-front-horiz.png).
With time, the punched card technology lost its glory and became obsolete and it was replaced by much more advanced storage technologies. However, the batch mode processing method amazingly survived and continued to play a major role in the computing world to handle critical business tasks. Although the surrounding technology has changed significantly, the underlying batch concept is still the same. In order to increase efficiency, programs written for batch-mode process (a.k.a batch jobs) are normally set to run when large amounts of input data are accumulated and ready for processing. Besides, a lot of routine procedure-type business processes are naturally required to be processed in batch mode. For example, monthly billing and fortnightly payrolls are typical batch-oriented business processes.
Note
The dictionary definition of batch processing: (Computer Science) A set of data to be processed in a single program run.
Traditionary Batch jobs are known to process large amounts of data at once. Therefore, it is not practical to expect the output to be given immediately. But there is still a predefined deadline for every batch processing, either it is set by the business requirement (also known as SLA — Service Level Agreement) or simply because it needs to finish so that the dependent batch processing tasks can start. For example, a group of batch jobs of an organization need to generate the payroll data and send it to the bank by Monday 5am, so the bank can have enough time to process it and ensure the funds get transferred to each of the organizaiton's employee's account by 8am Tuesday morning.
The rules and requirements of those business processes that require batch processing are becoming more and more sophisticated. This makes batch processing not only very time-consuming, but also task intensive, that is, the business process is required to be achieved in more than one step or even by many related jobs one after another (also known as job flow). In order for the computer system to execute the job or a job flow without human interaction, relevant steps within a job or jobs within a job flow need to be prearranged in the required logical order and the input data needs to be ready prior to the runtime of the job's step.
Batch processing versus interactive processing
As the time goes by, computer systems got another major improvement by having the ability to handle users' action-driven interactive processing (also called transaction processing or online processing). This was a milestone in computing history because it changed the way human minds work with computers forever, that is, for certain types of requests, users no longer need to wait for the processing to happen (only in batch mode during a certain period of time). Users can send a request to the computer for immediate processing. Such requests can be a data retrieving or modifying request. For example, someone checking his or her bank account balance on an ATM machine or someone placing a buy or sell order for a stock through an online broking website. In contrast to batch processing, computer systems handle each of the user requests individually at the time when it is submitted. CICS (Customer Information Control System) is a typical mainframe application designed for handling high-volume online transactions on the other hand, there is personal computer started to gain popularity which designed and optimised to work primarily in interactive mode.
In reality, we often see that batch processing and transaction processing share the same computing facility and data source in an enterprise class computing environmnet. As interactive processing aims at providing a fast response for user requests generated on a random basis, in order to ensure that there are sufficient resources available on the system for processing such requests, the resource intensive batch jobs that used to occupy the entire computing facility 24/7 had to be set to run only during a time frame when user activities are low, which back at the time is more likely to be during night, that is, as we often hear a more seasoned IT person with mainframe background call it nightly batch.
Here's an example of a typical scenario in a batch processing and transaction processing shared environment for an online shopping site:
In this example, programs for batch mode processing are set to run only when online transactions are low. This allows online processing to have the maximum system resources during its peak hours. During online processing's off peak hours, batch jobs can use up the entire system to perform resource-intensive processing such as sales conciliation or reporting.
In addition, because during the night time there are fewer changes to the data, batch jobs can have more freedom when manipulating the data and it allows the system to perform the backup tasks.
Time-based batch- and event-driven batch
What we have discussed so far Batch processing defined to run during a certain time is traditional time-based scheduling. Depending on the user's requirements, it could be a daily run, a monthly run, or a quarterly run, such as:
The timeframe allocated for batch processing is called a batch window. The concept sounds simple, but there are many factors that need to be taken into consideration before defining the batch window. Those factors include what time the resource and input data will be available for batch processing, how much data needs to be processed each time, and how long the batch jobs will take to process them. In case the batch processing fails to complete within the defined time window, not only does the expected batch output be delivered on time, but the next day's online processing may also get affected. Here are some of the scenarios:
A concept called event-triggered scheduling was introduced during the modern computing age to meet the growing business demand. An event can be as follows:
Rather than accumulating these events and processing them during the traditional nightly batch window, a mechanism has been designed within the batch processing space to detect such an event in real-time and process them immediately. By doing so, the event initiators are able to receive an immediate response, where in the past they have to wait until the end of the next batch window to get the response or output. Use the online shopping example again; during the day, orders get generated by online users. These orders are accumulated on the system and wait to be processed against the actual stock during the predefined batch windows. Customers have to wait till the next morning to receive the order committed e-mail and back order items report. With event-triggered batch processing, the online business is able to offer a customer an instant response on their order status, and therefore, provide a better shopping experience.
On the other hand, as a noticeable amount of batch processing work is spared during event time, the workload for batch processing during a batch window (for example, a nightly batch) is likely to be reduced.
Is this the end for batch processing?
There have been talks about totally replacing time-based batch processing with real-time event-driven processing to build a so called real-time enterprise. A group of people argue that batch processing causes latency to business processes and as event-driven solutions are becoming affordable, businesses should be looking at completely shifting to event-driven real-time processing. This approach has been discussed for years. However, its yet to completely replace batch processing.
Shifting the business process into real-time can allow businesses to have quicker reaction to changes and problems by making decisions based on live data feeds rather than historical data produced by batch processing. For example, an online computer store can use a real-time system to automatically adjust their retail price for exchange rate sensitive computer components according to live feed currency exchange rate.
The business may also become more competitive and gain extra profit by having each individual event handled in real time. For example, mobile phone companies would rather provide each SIM card as soon as it is purchased, than let the customers wait until the next day (that is when the over-night batch processing finish processing the data) and lose the potential calls that could be charged during the waiting time.
Note
Case study: Commonwealth Bank of Australia shifting into real-time banking
"In October 2010, Commonwealth Bank of Australia announced their future strategy in IT, that is, progressively upgrading their banking platforms to provide real-time banking for customers. Real-time banking is the ability to open and close an account, complete credit and debit transactions, and change features immediately. Real-time banking removes the delay we experience now from batch processing, where transactions are not completed until the next working day. Real-time banking will gradually replace batch processing, as our new banking platform replaces existing systems over the next few years."
Reference from Commonwealth Bank of Australia Website.
However, real-time processing is not a silver bullet. Although purchasing real-time solutions is becoming cheaper, moving the organization's entire batch processing into real-time processing can be an extremely costly and risky project. Also, we may find that current computing technology is still not powerful enough to handle everything in real time. Some IT professionals did a test during a project. They designed the system to process everything in real time and when the workload increased, the entire system just collapsed. We know hardware is becoming cheaper these days, but machine processing power is not the only limitation. When we talk about limited computing resources, it also can be the number of established sessions to a database or the availability of information for access. In case of a large request, such as generating a report, it may require exclusive access to the database. During the processing of data within the table, it is not available for other requests to modify. Therefore, such requests should be considered as batch processing and configured to run at an appropriate time.
The bottom line is not all business processes are required to be completed in real time. For example, a retail store only needs to generate purchase orders once a week. The store has no reason to transfer this business process into real-time processing, because they want to accumulate individual stock requests over a period of time and send it to the supplier in a single order. In this case, the shop can receive goods in bulk and save shipping cost, and at the same time there is a possibility to receive special offers that are given based on large order quantity.
Note
ZapThink's Ron Schmelzer wrote:
Batch processing, often thought of as an artifact left over from the legacy days, plays a vital role in systems that may have real-time processing upfront. As he observed, "behind the real-time systems that power the real-time enterprise are regularly-updated back office business systems. Batch processes remain essential for one key reason: it is simply not efficient to regenerate a complete forecast or business plan every time the business processes a single event such as an incoming customer order."
Reference from ZapThink:http://www.zdnet.com/blog/serviceoriented/soas-final-frontier-can-we-should-we-serviceorient-batch-processing/1439
Either running the business process in real time or batches, the system designer should take many factors into consideration, such as:
Moving IT into real time should be driven by actual business needs rather than the technology. The person who is going to design the system needs to carefully consider what needs to be processed in real time and what can be processed in batch to meet the business requirements and balance the system utilization. As a fact, large orgnisations today are still investing in batch processing and continuously trying to figure out how to make it run better.
Running batch processing tasks
To understand batch processing further we need to begin from it's father - the mainframe computers. Job Control Language (JCL) was introduced as a basic tool on the mainframe computers for defining how a batch program should be executed. A JCL is considered to be a job, also called a job card (inherited the name from punched cards). A job can have a single step or multiple steps (up to 255 steps in each JCL), and each step is an executable program or a JCL procedure (frequently used JCL statements are define into procedures for reusability). In JCL, a user needs to specify the name of the job, the program, or procedure to be executed during each step of the job, as well as the input and output of the step. Once the job is submitted for execution, the Job Entry Subsystem (JES) will interpret the JCL and send it to the mainframe operating system (MVS or Z/OS) for p rocessing (refer to the next diagram).
The system will read the submitted JCL to figure out what application to run, the location of the input, and where the output should go to.
Batch processing can be a standalone task, but in common cases business processes require many steps to deliver. Technically, all these steps can be combined within one processing unit (for example, steps with a JCL), but if a step fails during the processing, rerunning that specific step can be challenging. There are third-party products provided on the mainframe computers just for managing rerun JCL steps.
On the distributed systems, it is up to the program designer to design their own way in the program to detect the error and handle such rerun action. In most cases, we have to restart the entire process from the beginning. In order to have flexibility and ease of management in batch processing, these steps are normally broken down into individual processing units (also known as jobs) and are set to be executed in sequence. By doing so, when a step fails in the middle of the processing or towards the end of the entire processing, a rerun can be allowed from the point of failure easily by rerunning the problem job.
Here is a Hello World edition of batch processing. imagamingpc.com is an online computer shop specialized in build-to-order gaming computers. The owner implemented his own order processing application to handle online orders. The system works in the following way:
This is a very simple batch processing example. It's not hard to figure out that step 2 is an interaction processing and step 3 is a batch processing.
One of the very early challenges was that the order processing program often fails somewhere during processing. Sometimes it fails during generating the daily build list and sometimes it fails to send out e-mails to customers for pending build notification. No matter at which step the program fails, the program needs to be re-run right from the beginning, even if the problem was at the last stage only. As a result, every time when a re-run happens, the customer will likely be getting two confirmation e-mails. Also, when the number of orders is large on a given day, rerunning the entire thing can take a lot of time.
The owner of the business realized the problem and requested an IT person to come up with a better way to run this problem. Pretty quickly, the IT person came up with an idea, that is, break down each processing step into a separate program. In this case, if any stage failed, you would only need to re-run the failed task. As an outcome, the batch processing became the following:
By now, we have a series of programs linked together and executed in sequence to handle the business process, as opposed to having one program that does everything. In this case, if the processing stopped half way through, the IT person can easily re-run that particular step followed by the rest, rather than executing the entire process from the beginning. Therefore, duplicate e-mails are no longer sent and the time required for re-processing is significantly reduced.
Batch processing is about making the best use of computing resources to process data and deliver result on time. Schedule jobs so that they can run imminently one after another, and keep the computing resource occupied without idle time. But this may not be considered as highly efficient with today's high processing power machines that have the ability to handle multi-tasking. Dividing steps into jobs provides a possibility of parallel processing for job steps that are not inter-dependent, as a result, maximizing the system resource utilization as well as shortening the end-to-end processing time. Use our online computer shop example by breaking down processing into tasks and running tasks that are not interdependent in parallel, we can significantly shorten the processing time.
Another objective of batch processing is about delivering the processing result with minimal manual intervention. Technically, batch processing can be initiated by humans. In fact, back in the old days, companies employed operators on shifts to submit job streams at a set time of the day, check each job's execution result, and initiate restarts. However, running batch manually can be extremely challenging due to the nature of batch processing that it is complicated and detailed. The number of tasks for a business process can be large, and each task may require a complex input argument to run. Depending on the characteristics of the task an execution can take anything from minutes to hours.
In an environment that requires hundreds or thousands jobs to be scheduled each day, people who run the batch job not only need to work long hours to monitor each execution and ensure the right batch job gets triggered with correct input arguments at the correct time, but they also need to react to job failures, handle the complicated logic between jobs to decide which job to run next, and at the same time, keep in mind parallel processing, as well as ensure that each job is triggered as soon as its parent job(s) is/are completed to minimize machine idle time. Making mistakes is an unavoidable part of human nature, especially in a complex environment where everything is time-constrained. Businesses would rather invest in a batch automation tool than take the risk of having critical business problems due to batch processing delay and failures by human mistakes.
Automating batch processing
As the modern computing batch processing is far more complicated than just simply feeding punched cards in sequence into the mainframe as it was in old days, a lot more factors need to be taken into consideration when running batch processing due to its time consuming and task-intensive nature. Batch scheduling tools were born to automate such processing tasks, thus reducing the possibility of human mistake and security concerns.
There were home-grown toolsets developed on the mainframe computers for automating JCL scripts. Modern age distributed computer systems also came with some general ability to automate batch processing tasks. On a Unix or LINUX computer, CRON is a utility provided as part of the operating system for automating the triggering of executables. The equivalent tool on Windows is called task scheduler. With these tools, the user can define programs or scripts to run at a certain time or at various time intervals. These tools are mainly used for basic scheduling needs such as automating backups at a given time or system maintenance.
These tools do not have the ability to execute tasks according to pre-requisites other than time. Due to the limiting feature and unfriendly user interface, users normally find it challenging when trying to use these tools for complex scheduling scenarios, such as when there is a predefined execution sequence for a group of related program tasks.
Over the years, major software vendors developed dedicated commercial batch scheduling tools such as BMC Control-M to meet the growing needs in batch processing. These tools are designed to automate complicated batch processing requirements by offering the ability to trigger task executions according to the logical dependencies between them.
Basic elements of a job
Similar to CRON, users firstly are required to define each processing task in the batch-scheduling tool together with its triggering conditions. Such definitions are commonly known as "Job Definitions", which get stored and managed by the scheduling tool. The three essential elements within each job definition are:
What to trigger
From a batch scheduling tool point of view, it needs to know which object is to be triggered. It can be a JCL on the mainframe, a Unix shell script, a Perl program, or a Windows executable file. A job also can be a database query, a stored procedure that performs data lookup or update, or even a file transfer task. There are also application-specific execution objects, such as SAP or PeopleSoft tasks.
When to trigger (Job's scheduling criteria)
Each job has its own scheduling criteria, which tells the batch scheduling tool when the job should be submitted for execution. Job scheduling criteria contains the job's execution date and time. A job can be a daily, monthly, or quarterly job, or set to run on a particular date (for example, at the end of each month when it is not a weekend or public holiday). The job can also be set to run at set intervals (running cyclic). In such cases, the job definition needs to indicate how often the job should run and optionally the start time for its first occurrence and end time for its last occurrence (for example, between 3pm to 9pm, run the job every five minutes). Most of the job schedulers also allow users to specify the job's priority, how to handle the job's output, and what action to take if the job fails.
Dependencies (Job's predecessors and dependents)
Job dependency is the logic between jobs that tells which jobs are inter-related. According to the job dependency information, the batch scheduling tool groups the individual, but inter-related jobs together into a batch flow. Depending on the business and technical requirements, they can be defined to run one after another or run in parallel. The common inter-job relationships are:
A one to one relationship simply means the jobs run one after another, for example, when Job A is completed, then Job B starts.
A one to many relationship means many child jobs depend on the parent job, once the parent job is completed, the child jobs will execute. Sometimes there's a degree of decision making within it, such as if job A's return code is 1, then run Job B, or if the return code of Job A is greater than 1, then run Job C and Job D.
A many to one relationship refers to one child job that depends on many parent jobs. The dependency from parent jobs' completion can be an AND relationship, an OR relationship also can be AND and OR mixed, for example, in an AND scenario, Job D will run only if Job A, Job B, and Job C are all completed. In an OR scenario, Job D will run if any of Job A, Job B, or Job C are completed. In a mixed scenario, Job D will run if Job A or Job B and Job C is completed.
During normal running, the batch scheduling tool constantly looks at its record of jobs to find out which jobs are eligible to be triggered according to the job's scheduling criteria, and then it will automatically submit the job to the operating system for execution. In most cases, the batch scheduling tool can control the total number of parallel running jobs to keep the machine from being overloaded. The execution sequence among parallel jobs can be based on individual job's predefined priority, that is, the higher priority jobs can be triggered before the lower priority ones. After each execution, the job scheduling tool will get an immediate feedback (such as an operating system return code) from the job's execution. Based on the feedback, the job scheduling tool will decide the next action such as to run the next job according to the predefined inter-job dependency or rerun the current job if it is cyclic. Batch scheduling tools may also provide a user interface for the system operator to monitor and manage batch jobs, which gives the ability for the user to manually pause, rerun, or edit the batch job.
More advanced features of scheduling tools
Driven by business and user demand, more sophiscated features have been provided with modern scheduling tools apart from automating the execution of batch jobs, such as:
