40,81 €
Develop, deploy, and scale your applications with Google Cloud Platform
Key Features
Book Description
Google Cloud Platform (GCP) provides autoscaling compute power and distributed in-memory cache, task queues, and datastores to write, build, and deploy Cloud-hosted applications.
With Google Cloud Platform for Developers, you will be able to develop and deploy scalable applications from scratch and make them globally available in almost any language. This book will guide you in designing, deploying, and managing applications running on Google Cloud. You'll start with App Engine and move on to work with Container Engine, compute engine, and cloud functions. You'll learn how to integrate your new applications with the various data solutions on GCP, including Cloud SQL, Bigtable, and Cloud Storage. This book will teach you how to streamline your workflow with tools such as Source Repositories, Container Builder, and StackDriver. Along the way, you'll see how to deploy and debug services with IntelliJ, implement continuous delivery pipelines, and configure robust monitoring and alerting for your production systems.
By the end of this book, you'll be well-versed with all the development tools of Google Cloud Platform, and you'll develop, deploy, and manage highly scalable and reliable applications.
What you will learn
Who this book is for
Google Cloud Platform for Developers is for application developers. This book will enable you to fully leverage the power of Google Cloud Platform to build resilient and intelligent software solutions.
Das E-Book können Sie in Legimi-Apps oder einer beliebigen App lesen, die das folgende Format unterstützen:
Seitenzahl: 677
Veröffentlichungsjahr: 2018
Copyright © 2018 Packt Publishing
All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.
Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the authors, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.
Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.
Commissioning Editor: Vijin BorichaAcquisition Editor:Shrilekha InaniContent Development Editor:Abhishek JadhavTechnical Editor:Aditya KhadyeCopy Editor:Safis EditingProject Coordinator:Judie JoseProofreader: Safis EditingIndexer:Pratik ShirodkarGraphics:Tom ScariaProduction Coordinator: Aparna Bhagat
First published: July 2018
Production reference: 1280718
Published by Packt Publishing Ltd. Livery Place 35 Livery Street Birmingham B3 2PB, UK.
ISBN 978-1-78883-767-5
www.packtpub.com
Mapt is an online digital library that gives you full access to over 5,000 books and videos, as well as industry leading tools to help you plan your personal development and advance your career. For more information, please visit our website.
Spend less time learning and more time coding with practical eBooks and Videos from over 4,000 industry professionals
Improve your learning with Skill Plans built especially for you
Get a free eBook or video every month
Mapt is fully searchable
Copy and paste, print, and bookmark content
Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.PacktPub.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at [email protected] for more details.
At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a range of free newsletters, and receive exclusive discounts and offers on Packt books and eBooks.
Ted Hunter is a software engineering consultant working with fortune 500 companies to design cloud-native solutions and drive public cloud adoption, primarily within the Google ecosystem. He has a background in full stack development, DevOps transformation, and designing enterprise data solutions. He is currently a Solution Architect at Slalom Consulting, serving clients in the Southeastern United States.
Steven Porter is a consulting technology leader for Slalom Atlanta's Software Engineering practice, a Microsoft Regional Director, and a Google Certified Cloud Architect. His major focus for the past 5+ years has been IT modernization and cloud adoption with implementations across Microsoft Azure, Google Cloud Platform, AWS, and numerous hybrid/private cloud platforms.
Outside of work, Steve is an avid outdoorsman spending as much time as possible outside hiking, hunting, and fishing with his family of five.
Sanket Thodge is an entrepreneur by profession based out of Pune, India. He is an author of the bookCloud Analytics with Google Cloud Platform. He is a founder of Pi R Square Digital Solutions Pvt Ltd. With expertise as Hadoop Developer, Sanket explored Cloud, IoT, Machine Learning, and Blockchain. He has also applied for a patent in IoT and has worked with numerous startups and MNCs in providing consultancy, architecture building, development, and corporate training across globe.
If you're interested in becoming an author for Packt, please visit authors.packtpub.com and apply today. We have worked with thousands of developers and tech professionals, just like you, to help them share their insight with the global tech community. You can make a general application, apply for a specific hot topic that we are recruiting an author for, or submit your own idea.
Title Page
Copyright and Credits
Google Cloud Platform for Developers
Packt Upsell
Why subscribe?
PacktPub.com
Contributors
About the authors
About the reviewer
Packt is searching for authors like you
Preface
Who this book is for
What this book covers
To get the most out of this book
Download the example code files
Download the color images
Conventions used
Get in touch
Reviews
Why GCP?
The public cloud landscape
Amazon Web Services
Microsoft Azure
Google Cloud Platform
Standing on the shoulders of giants
A world-class global presence
Choosing your own adventure
Leading the way for big data
The Open Cloud and innovation
Dedication to customer success
Bottom-up security
In good company
Summary
The Google Cloud Console
Getting started – Google Cloud projects
Architectural role of Google Cloud projects
Creating a project
Free trials on GCP
The Google Cloud Console
Understanding the Cloud Console dashboard
The Google Cloud Shell
Launching the Cloud Shell
Supporting multiple sessions
Features and integrations
File management
Web Preview
The Cloud Shell Code Editor
Opening in Cloud Shell
Trying it out
Installing additional tools
Boost mode
Repairing the Cloud Shell
Other tools
Mobile apps
Developer tool integrations
Summary
APIs, CLIs, IAM, and Billing
Google Cloud APIs
Managing APIs
Google APIs Explorer
Trying out the APIs Explorer
The Google Cloud SDK
Installing the Google Cloud SDK
The gcloud command-line tool
The basics of gcloud
Command groups
Root commands
Global flags
Initializing the Google Cloud SDK
Authentication
Managing your Google Cloud SDK
Updating and rollbacks
Alpha and beta channels
Configurations in the Google Cloud SDK
Modifying configuration properties
Multiple configurations
Other command-line tools
bq
gsutil
kubectl
Automating tasks with gcloud
Modifying output and behavior
Formatting attributes
Formatting projections
Filtering
Google Cloud IAM
How IAM works
IAM roles
The structure of IAM policies
Organization-level policies
Project-level policies
Resource-level policies
Cross-project access
Managing IAM
Service accounts
Billing on Google Cloud
Billing accounts
Billing accounts and IAM
Budgets and billing alerts
Google Cloud Platform Pricing Calculator
Creating an estimate
Summary
Google App Engine
Compute services on the GCP
Google Compute Engine
Google Kubernetes Engine (GKE)
Google App Engine
Google Cloud Functions
General considerations
Google App Engine
Features and benefits
Developer velocity
Visibility
Scalability
Simple integrations
Structure of a Google App Engine application
Architecture of an App Engine solution
Microservices
Batch work and task queues
App Engine locations
IAM on the Google App Engine
App Engine service accounts
The standard and flexible environments
Standard environment
Flexible environment
Setting up the App Engine
The App Engine standard environment
Language support
Developing for the App Engine standard environment
The Python runtime
WSGI and CGI
Getting started
The App Engine development server
The Go runtime
Running multiple services locally
The Java 8 runtime
Deploying App Engine standard services
Deployment behavior
Splitting network traffic
Instance classes
Pricing in the standard environment
Spending limits
The App Engine flexible environment
Benefits of the flexible environment
More control over the infrastructure
Application portability
Language support
Developing for the flexible environment
Deploying App Engine flexible apps
Container technologies
Google Container Builder
Google Container Registry
Custom runtimes
Building custom runtime services
Deploying a service to the flexible environment
Pricing in the flexible environment
App Engine resources and integrations
Task queues
Push and pull queues
Push queues
Named queues
Pull queues
Creating tasks
Structuring tasks queues
Scheduled tasks
Deploying a cron definition
Trying the App Engine cron service
Scaling App Engine services
Autoscaling
Basic and manual scaling
Externalizing configuration and managing secrets
Application configuration files
Compute Engine metadata server
Runtime Configurator
Cloud Key Management Service (KMS)
General considerations
Networking and security
The App Engine firewall
Cloud Endpoints
Google Cloud IAP
Virtual private networks 
Summary
Google Kubernetes Engine
Google Kubernetes Engine
When to choose GKE
GKE or App Engine Flex
Creating and maintaining a GKE cluster
Node pools
Multi-zonal and regional clusters
Container Registry
Deploying workloads to GKE
Rolling updates
Rolling back updates
Scaling deployments
Manually scaling deployments
Automatically scaling deployments
Exposing GKE Services
Exposing services within a cluster
Exposing services to external traffic
Managing secrets with GKE
Creating/Storing secrets
Using secrets
Billing
Summary
Google Cloud Functions
Functions as a Service
Google Cloud Functions
Advantages of Cloud Functions
Price
Scalability
Developer velocity
Considerations when using Cloud Functions
Invoking Cloud Functions
HTTP functions
Processing HTTP requests
Background functions
Cloud Pub/Sub functions
Cloud Storage functions
Background function retries and termination
Developing Cloud Functions
Using the Cloud Console
Local development
Debugging functions
Deploying Cloud Functions
Deploying from a local machine
Deploying from a source repository
Integrating with other Google services
IAM and billing
Cloud Functions and IAM
Frameworks and tooling
Summary
Google Compute Engine
Understanding Compute Engine
IaaS
Infrastructure as Code (IaC)
More than virtual machines
When to use Compute Engine
A straightforward migration path
Host anything
Building a robust global presence
Long running and resource intensive processes
Security and compliance
Virtual machines on Google Compute Engine (GCE)
Machine types
Standard machine types
High-memory machine types
Mega-memory machine types
High-CPU machine types
Shared-core machine types
Custom machine types
Extended memory
Other resources
Disk storage
GPUs
Images
Public images
Premium images
Community images
Container images
Managing Compute Engine instances
Creating instances
Remote access
SSH access
SCP access
Remote Desktop Protocol (RDP) access
Metadata server
Default metadata
Project-wide metadata
Instance-specific metadata
Setting and removing metadata
Querying metadata from within instances
Trying it out
Modifying API responses
Startup and shutdown scripts
Startup scripts
Shutdown Scripts
Windows machines
Updates and patches
Availability policies
Maintenance behavior
Restart behavior
Relocating an instance
Storage solutions
Persistent disks
Standard and solid-state drive (SSD) persistent disks
Persistent disk performance
Boot disks
Managing persistent disks
Persistent disk snapshots
Local SSDs
Creating scalable solutions with GCE
Custom images
Creating images from a persistent disk
Copying an image
Creating images from snapshots
Golden images
Security concerns
Managed instance group (MIG)
Instance templates
Creating MIGs
Built for resilience
Autoscaling
Autohealing
Change management
Performing a rolling update
IAM and service accounts
Administrative operations
General roles
Compute resource roles
Network and security resource roles
Compute instance IAM
Pricing on GCE
Instance discounts
Preemptible instances
Committed use discounts
Sustained use discounts
Other resource costs
Always-free tier
Summary
NoSQL with Datastore and Bigtable
NoSQL solutions on GCP
NoSQL technologies
Google Cloud Datastore
When to use Datastore
Getting started
Datastore locations
Managing entities in the Cloud Console
Datastore core concepts
The structure of Datastore data
Entities, kinds, and properties
Data types
Entity identifiers
Namespaces
Ancestry paths and keys
Entity groups and consistency
Entity groups
Consistency and queries
Working with entities
Queries with GQL
Using GQL in the Cloud Console
Indexes
Single property indexes
Composite indexes
Datastore under the hood
The entities table
Key
Entity group
Kind
Properties
Custom indexes
Index tables
EntitiesByKind
EntitiesByProperty
EntitesByCompositeProperty and Custom Indexes
Datastore management and integrations
Administrative tasks
The Datastore Admin Console
gcloud operations
Integrations with other GCP services
App Engine standard environment
Other GCP services
Datastore pricing and IAM
Permissions in Datastore
Google Cloud Firestore
Comparison to Datastore
A promising future
Google Bigtable
Core concepts
Structure of Bigtable data
Columns and column families
Column families
Scalable and intelligent
Bigtable under the hood
Building on other Google technologies
Tablets and servers
Creating and managing clusters
Instances, clusters, and nodes
Development instances
Bigtable locations
Create a development cluster
Using gcloud
Scaling clusters
Promoting development clusters
Deleting a cluster
Interacting with data on Bigtable
The cbt command-line interface
The Bigtable HBase Client
Platform integrations
BigQuery external tables
Dataflow Bigtable IO
Bigtable pricing and IAM
Permissions in Bigtable
Summary
Relational Data with Cloud SQL and Cloud Spanner
Google Cloud SQL
Configuring Cloud SQL instances
Creating a Cloud SQL instance
Database engines
MySQL generations
Machine and storage types
Choosing a machine type
Configuring storage
Cloud SQL locations
When to use multiple instances
Connecting to Cloud SQL
Authorized networks
Connecting with gcloud
SSL support
Establishing an SSL Connection
The Cloud SQL Proxy
Setting up the Cloud SQL Proxy
Authenticating with the Cloud SQL Proxy
Trying it out
Managing Cloud SQL instances
Maintenance operations
Importing data to Cloud SQL
Exporting data to cloud storage
Backups and recovery
Trying it out
Point-in-time recovery
Updates
Database flags
Database flags and SLAs
Replicas and high availability
Read-only replicas
External replicas
High availability
Forcing a failover
Scaling Cloud SQL instances
Scaling Storage
Scaling compute
Alerting on resource pressure
Horizontal scaling
Migrating databases to Cloud SQL
Cloud SQL IAM and users
IAM policies
Database users
Default and system users
Additional users
Changing user passwords
Cloud SQL Proxy users
Cloud SQL pricing
Google Cloud Spanner
Instances and instance configurations
Regional configurations
Multi-region configurations
Nodes, databases, and tables
Creating a Cloud Spanner instance
Importing data into Cloud Spanner
Performing a simple query
Understanding Cloud Spanner
Cloud Spanner and CAP theorem
Maintaining consistency
TrueTime and linearization
Paxos groups
Read operations
Write operations
Transactions
Database design and optimizations
Query execution plans
Primary keys
Data collocation and interleaving
Secondary indexes and index directives
Cloud Spanner administration
Cloud Spanner IAM Roles
Cloud Spanner prices
Summary
Google Cloud Storage
GCS basics
Buckets
Bucket names
Domain-named buckets
The global bucket namespace
Objects
Object data
Object metadata
Virtual file structures
Using gsutil
Creating and using a bucket
Uploading files to GCS
Storage classes and locations
Regional and Multi-Regional Storage
Standard and durable reduced availability
Nearline and Coldline Storage
Cloud Storage locations
Nearline and Coldline Storage locations
Choosing the right storage class
Cloud Storage pricing
Bucket and object storage classes
Automating object management
Monitoring lifecycle events
Object versioning
Data governance in Cloud Storage
Cloud Storage IAM
ACLs
Limitations of concentric access control
Customer supplied encryption keys
Signed URLs
Capabilities and integrations
Integrating with Google Cloud Functions
Static web content and Backend Buckets
Summary
Stackdriver
Lessons from SRE
Monitoring and alerting
Preparation for this chapter
Stackdriver basics
Stackdriver and GCP projects
Creating and linking a Stackdriver account
Stackdriver Logging
Filtering and searching
Basic filtering
Advanced filtering
Exporting Stackdriver logs
Exporting to Cloud Storage
Exporting to BigQuery and Cloud Pub/Sub
Monitoring and alerting
The Stackdriver Monitoring console
Exploring Stackdriver metrics
Creating dashboards
Stackdriver alerting policies
Policy conditions
Creating an alerting policy
Notifications and documentation
Stackdriver incidents
Other types of metrics
Error reporting
Investigating errors
Stackdriver APM
Stackdriver Trace
Investigating application latency
Stackdriver Debugger
Debugging the todos services
Logpoints
Stackdriver Profiler
Summary
Change Management
Preparing for this chapter
Google Cloud Source Repositories
Google Cloud Deployment Manager
Declarative configuration management
Basic configurations
Resource types and properties
Deployments
Deploying a simple configuration
Deployment manifests
Updating deployments
Create and delete policies
Maintaining deployment state
Remediation
Templates
Creating a template
Other template features
Cloud Launcher and Deployment Manager
Runtime Configurator
Watchers
Waiters
Google Cloud Container services
Google Container Registry – GCR
Container Builder
Build triggers
Continuous deployment in Google Cloud
Summary
GCP Networking for Developers
Networking fundamentals
Virtual private networks
Subnetworks
Configuring VPC networks
Networks and compute resources
Firewall rules
Components of a firewall rule
Action
Direction
Target
Source or destination
Protocol and port
Priority
Securing networks with firewall rules
Routes
IP addresses
Internal and external IP addresses
Ephemeral and static IP addresses
Global IP addresses
Google load balancers
Network load balancers
Target pools
Forwarding rules
Health checks
Failover ratio and backup pools
Creating a TCP network load balancer
Internal load balancing
Global load balancers
Components of global load balancers
Backend services
Target proxies
Global forwarding rules
SSL and TCP proxies
HTTP(S) load balancers
Autoscaling load balanced resources
Google Cloud DNS
Access control and API management
Google Cloud Endpoints
Services
API providers
Access and discovery
Identity-Aware Proxy
Cloud Armor
Summary
Messaging with Pub/Sub and IoT Core
Google Cloud Pub/Sub
Topics and subscriptions
Push and pull message delivery
Pull subscriptions
Push subscriptions
Choosing a subscription model
Message acknowledgment
Nacking messages
Designing for resilience
Message loss
Processing failures
Duplicate messages
Out-of-order messages
Google Cloud IoT Core
Device management and registries
Device authentication and security
Consuming device data
Summary
Integrating with Big Data Solutions on GCP
Big data and Google Cloud Platform
Cloud Dataflow
Evolution of data processing at Google
Pipelines
Collections
Transformations
Element-wise transforms
Aggregate transforms
Composite transforms
Sources and sinks
Creating and executing pipelines
Executing pipelines locally
Executing pipelines on Cloud Dataflow
Executing streaming pipelines
Pipeline templates
Google provided pipeline templates
Managing Cloud Dataflow jobs
Google BigQuery
How BigQuery executes queries
Integrating with BigQuery
BigQuery as a Cloud Dataflow Sink
Batch loading files from Cloud Storage
Streaming inserts
Exploring BigQuery data
Summary
Other Books You May Enjoy
Leave a review - let other readers know what you think
Google Cloud Platform (GCP) provides autoscaling compute power and distributed in-memory cache, task queues, and datastores to write, build, and deploy Cloud-hosted applications.
With Google Cloud Platform for Developers, you will be able to develop and deploy scalable applications from scratch and make them globally available in almost any language. This book will guide you in designing, deploying, and managing applications running on Google Cloud. You'll start with App Engine and move on to work with Container Engine, compute engine, and cloud functions. You'll learn how to integrate your new applications with the various data solutions on GCP, including Cloud SQL, Bigtable, and Cloud Storage. This book will teach you how to streamline your workflow with tools such as Source Repositories, Container Builder, and StackDriver. Along the way, you'll see how to deploy and debug services with IntelliJ, implement continuous delivery pipelines, and configure robust monitoring and alerting for your production systems.
By the end of this book, you'll be well-versed with all the development tools of Google Cloud Platform, and you'll develop, deploy, and manage highly scalable and reliable applications.
Google Cloud Platform for Developers is for application developers and DevOps engineers that wish to become familiar with the various service offerings available on Google Cloud Platform. This book will enable you to fully leverage the power of Google Cloud Platform to build resilient and intelligent software solutions.
Chapter 1, Why GCP?, this chapter introduces readers to the Google Cloud Platform. It provides an overview of cloud computing, a brief history of GCP, as well as a comparison to other public cloud providers.
Chapter 2, The Google Cloud Console, this chapter serves to familiarize readers with the primary user interfaces they will use when interacting with Google Cloud.
Chapter 3, APIs, CLIs, IAM, and Billing, in this chapter, readers will learn about the various command line tools provided by Google for managing cloud resources. Readers will also learn about the other tools that will enable them to manage their Google Cloud projects throughout the book.
Chapter 4, Google App Engine, this chapter will explain what Google App Engine (GAE) is, the driving philosophies behind it, and how to use it to run highly-scalable services.
Chapter 5, Google Kubernetes Engine, this chapter is about the Google Container Engine (GKE) platform for running and managing services on Google Cloud.
Chapter 6, Google Cloud Functions, this chapter is about creating and executing Cloud Functions using Google's serverless platform.
Chapter 7, Google Compute Engine, this chapter is about Google's IaaS offering: Google Compute Engine (GCE). This chapter will introduce readers to on-demand VMs and how they can be managed, scaled, and customized to the user's needs.
Chapter 8, NoSQL with Datastore and Bigtable, this chapter will introduce readers to the document based storage solutions offered by Google, including Datastore (plus the new Firestore), and Bigtable.
Chapter 9, Relational Data with Cloud SQL and Cloud Spanner, this chapter will cover Google's relational data storage solutions, including managed MySQL and PostgreSQL via Cloud SQL, as well as globally consistent relational data via Cloud Spanner.
Chapter 10, Google Cloud Storage, this chapter is about Google's unified object storage platform: Google Cloud Storage (GCS).
Chapter 11, Stackdriver, this chapter will cover Google's Stackdriver monitoring, logging, and diagnostics suite to drive application insights, availability, and fast incident resolution.
Chapter 12, Change Management, this chapter will introduce readers to the various platform tools Google offers around the developer/operations experience, including source control, building and deploying services.
Chapter 13, GCP Networking for Developers, this chapter will introduce readers to networking on Google Cloud, covering the products available and how to use them to build custom networking and security solutions. These topics will be presented in a manner appropriate for developers rather than networking professionals.
Chapter 14, Messaging with Pub/Sub and IoT Core, this chapter will introduce readers to the distributed messaging offerings on Google Cloud. Readers will learn how to leverage Google Cloud Pub/Sub for high-throughput messaging used both in service to service communications and Big Data ingestion pipelines, as well as Cloud IoT Core for widely distributed event-driven application architectures.
Chapter 15, Integrating with Big Data Solutions on GCP, this chapter will provide a high level overview of big data solutions on Google Cloud Platform. Users will learn how to build highly scalable, fully managed big data solutions with the power of Cloud Dataflow and BigQuery.
This book is geared towards readers with a familiarity of basic application development and DevOps concepts. The exercises provided in this book include Java, Python, Node.js, Go, SQL, and shell scripting. These exercises are designed to be simple and easy to complete without prior knowledge of a specific language or framework.
All examples and exercises in this book can be completed directly within the Google Cloud Console and Google Cloud Shell, however many users will prefer to work within a local development environment. To do so, download and install the Google Cloud SDK available at https://cloud.google.com/sdk/.
You can download the example code files for this book from your account at www.packtpub.com. If you purchased this book elsewhere, you can visit www.packtpub.com/support and register to have the files emailed directly to you.
You can download the code files by following these steps:
Log in or register at
www.packtpub.com
.
Select the
SUPPORT
tab.
Click on
Code Downloads & Errata
.
Enter the name of the book in the
Search
box and follow the onscreen instructions.
Once the file is downloaded, please make sure that you unzip or extract the folder using the latest version of:
WinRAR/7-Zip for Windows
Zipeg/iZip/UnRarX for Mac
7-Zip/PeaZip for Linux
The code bundle for the book is also hosted on GitHub at https://github.com/PacktPublishing/Google-Cloud-Platform-for-Developers. In case there's an update to the code, it will be updated on the existing GitHub repository.
We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/. Check them out!
We also provide a PDF file that has color images of the screenshots/diagrams used in this book. You can download it here: https://www.packtpub.com/sites/default/files/downloads/GoogleCloudPlatformforDevelopers_ColorImages.pdf.
There are a number of text conventions used throughout this book.
CodeInText: Indicates code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles. Here is an example: "Web Preview runs on port 8080 by default, but can be mapped to any port from 8080-8084 via the Web Preview settings"
A block of code is set as follows:
{ "eventId": "27819225098479", "timestamp": "2018-01-27T18:11:24.836Z", "eventType": "providers/cloud.pubsub/eventTypes/topic.publish", "resource": "projects/<PROJECT_ID>/topics/<TOPIC_NAME>", "data": { "@type": "type.googleapis.com/google.pubsub.v1.PubsubMessage", "attributes": { "<KEY1>": "<VALUE1>", "<KEY2>": "<VALUE2>"},
Any command-line input or output is written as follows:
gcloud compute instances create my-custom-instance \
--region=us-central1 \
--memory=12GiB \
--cpu=2
Bold: Indicates a new term, an important word, or words that you see onscreen. For example, words in menus or dialog boxes appear in the text like this. Here is an example: "This can be found in the Cloud Shell menu under Send key combination | Install Chrome extension."
Feedback from our readers is always welcome.
General feedback: Email [email protected] and mention the book title in the subject of your message. If you have questions about any aspect of this book, please email us at [email protected].
Errata: Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you have found a mistake in this book, we would be grateful if you would report this to us. Please visit www.packtpub.com/submit-errata, selecting your book, clicking on the Errata Submission Form link, and entering the details.
Piracy: If you come across any illegal copies of our works in any form on the Internet, we would be grateful if you would provide us with the location address or website name. Please contact us at [email protected] with a link to the material.
If you are interested in becoming an author: If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, please visit authors.packtpub.com.
Please leave a review. Once you have read and used this book, why not leave a review on the site that you purchased it from? Potential readers can then see and use your unbiased opinion to make purchase decisions, we at Packt can understand what you think about our products, and our authors can see your feedback on their book. Thank you!
For more information about Packt, please visit packtpub.com.
Today's technology consumers demand always-on, real-time software solutions that are able to scale to rapidly changing loads. Companies demand deep insights into their customers to drive business decisions and predict market changes. Creative start-ups regularly disrupt long-standing industry leaders due to their ability to quickly innovate and bring new technology solutions to established problems.
The public cloud is a proven model for driving innovation. By lowering the turnaround for operations such as provisioning virtual machines or configuring networks, teams are able to spend less time waiting and more time solving business problems. By providing powerful and flexible permission systems, public clouds offer customers the ability to adopt self-service models for many operational tasks, further lowering the barrier for developers to get the ball rolling. By centralizing top talent across operations and security, public clouds are able to provide an extreme level of robustness and security in their products.
Perhaps the strongest motivator for many companies considering the public cloud is the rather large potential to reduce operational and infrastructure costs. By taking in the cost of building and managing data center, companies such as Amazon, Google, and Microsoft are able to achieve massive economies of scale. These economies of scale allow public clouds to outperform most private cloud solutions in terms of compute costs, storage, and infrastructure management costs—a benefit they then pass on to customers. Although many companies will fall somewhere between fully public cloud-based solutions and on-premise solutions (often called hybrid clouds), most businesses stand to gain significant savings by leveraging some level of public cloud services.
For a developer, public clouds offer many new and exciting ways to bring applications to your end users. In the simplest terms, this is achieved by abstracting away major components of application management that are not your direct concern. If your goal is to provide an API, your primary concern is likely building and running a collection of web services, not provisioning and maintaining the servers to host these services. If your application needs to persist user data, your primary concern is likely building out a well-tested data persistence layer backed by a reliable database, not managing the hard drives on which your data is persisted. Public clouds offer developers the ability to dedicate more time to solving the actual problems at hand by leveraging managed services.
In addition to these abstractions, public clouds offer a large number of solutions for supporting running applications. Common solutions include logging services, along with metric aggregation, tracing, and introspection services for application insights. Likewise, monitoring and alerting services are considered core functionality, and are usually deeply integrated with both logging services and the underlying application management platforms to provide a cohesive ecosystem for supporting robust cloud-native applications.
In this chapter, we will cover the following topics:
Understanding the big trends in the public cloud space
Identifying differences between the major cloud providers
Understanding why Google Cloud Platform may be a good choice for your company
The public cloud space is currently one of the most competitive and rapidly changing areas of technology. As more and more companies look to take the jump into public clouds, providers are fiercely competing to be the public cloud customers choose. Microsoft, Amazon, and Google stand at the top of the hill, with many others looking to secure a seat at the table. This fierce competition is great for customers, as it drives providers to constantly innovate and deliver more value.
Customer success has become the driving metric by which cloud providers are measured, which means providers are valued based on their ability to enable customers to achieve business goals. This creates a laser focus on delivering services and features that help you win. While Microsoft Azure, Amazon, and Google look to offer the core products and services customers need, they all have distinct strengths and advantages they bring to the problem space.
By capitalizing on these strengths, they are able to differentiate themselves and develop unique product-market fits. Understanding these market differentiators is critical when evaluating which public cloud provider is right for you.
Amazon released an early version of the AWS platform in the early 2000s and over the following decade it became the dominant public cloud. With constant innovation and a forward-thinking strategy, Amazon regularly won the first-to-market advantage. This helped secure the lion's share of the market, which Amazon has maintained over the past decade.
With an incredibly large user base and a legacy of excellent service, AWS has built a vast network of key customers and partners across business sectors. AWS currently offers the widest selection of products and services, and its platform is easily the most mature. Many companies choose AWS for its incredible customer support and track record of stability and security.
Microsoft Azure was originally released in 2010 and has become a go-to solution for many businesses that rely heavily on the Microsoft stack. With deep integrations into the larger Microsoft ecosystem, Azure is often a no-brainer for many Microsoft shops looking for a straightforward path to the cloud.
Aside from best-in-class support for many Microsoft services, Azure looks to provide the tools and resources that large enterprises and governments need. Azure offers incredibly easy-to-use tools, and their various products and services are incredibly easy to integrate with.
Over the past few years, Microsoft has looked to increase its market in areas outside of the Microsoft ecosystem. With big moves such as open sourcing .NET Core and bringing SQL Server to Linux, Microsoft is making waves in communities that had previously been outside of their focus. On the product side, Azure continues to expand their potential market with great support for platforms such as Kubernetes and Docker Swarm. These actions show that Microsoft is redefining themselves as a community-driven, cloud-first company.
In April 2008, the Google developer team announced a closed developer preview of their new Platform-as-a-Service offering: Google App Engine. Google invited 10,000 lucky (and brave) developers were to test and provide feedback on an early version of App Engine. By May, that number had increased to 75,000 active developers; Google announced fully open signups, making App Engine available to the masses.
In the years that followed, Google released a steady stream of products and features. With services such as Google Cloud Storage in 2010, Compute Engine in 2013, Cloud SQL in 2014, and Kubernetes Engine in 2015, Google has built out a diverse and comprehensive suite for developing cloud-native solutions. During this time, Google looked to expand their domain into varying areas such as infrastructure management, data analytics, Internet of Things, and machine learning. By 2017, Google had established data centers in 39 zones across 13 regions.
With fierce competition among the major public cloud providers, Google is looking to establish itself as a market leader. With services such as BigQuery, Bigtable, Cloud Pub/Sub, and Dataflow, Google has thrown down the gauntlet in the data analytics arena. With a robust global infrastructure and experience running applications at scale, Google is looking to win over developers wanting to build solutions that support small groups of early adopters and effortlessly scale to support floods of users as applications go viral. With decades of experience providing highly available web services such as Search and Gmail, Google is positioned to redefine reliability in the cloud.
Today, the Google Cloud Platform catalog includes several products and services that cover a large number of use cases and industries. Core services such as Compute Engine and Cloud Storage enable teams to build virtually any solution, while many specialized services such as the Cloud Vision API greatly lower the barrier of entry for teams to tackle more specific problem spaces. As Google moves full steam ahead into the public cloud space, the number of both core and specialized products and services continues to grow at breakneck speed, as shown in the following graphic:
Google Cloud Platform is the product of decades-long experience running some of the largest and most successful web services in history. The infrastructure Google offers in GCP is the same infrastructure Google uses internally, meaning customers directly benefit from the wealth of hard-won knowledge and ingenuity Google has amassed through running many of their well-known large-scale services. Extreme reliability and security are established norms at Google, and these qualities are deeply ingrained into GCP's underlying infrastructure.
Google also embeds and applies this knowledge and experience to their managed services. Google App Engine is the direct product of Google's expertise managing web-scale services and is designed to make scalability a non-issue. With easy-to-use service integrations and managed autoscaling, engineers can develop against simple interfaces to quickly create web services that scale to any load. Likewise, Kubernetes (and by extension Google Kubernetes Engine) is the result of Google's experience, successfully orchestrating massive numbers of web services via the internal data center scheduling and orchestration platform known as Borg. BigQuery is the result of externalizing Google's own analytics platform, called Dremel. Google Bigtable is built on top of Google's powerful internal lock system, Chubby. Cloud Datastore builds on Bigtable clusters to provide easy-to-use managed document stores. Cloud Storage, BigQuery, and Bigtable are all built on top of Google's large-scale clustered filesystem Colossus (originally Google File System (GFS)). The point is, when you use GCP, you are the direct beneficiary of Google's success.
Google's 13 regions are connected by the first multi-tier global fiber network from a major public cloud provider. With over 100 points of presence, Google Cloud offers your users low latency no matter where they are in the world. This private fiber optic network is the backbone of Google's own global presence, made available to GCP customers. On top of this, Google offers powerful networking tools for easily building out your own network architecture. These tools include fully software-defined networks, self-adjusting network routing between on-premises networks and the cloud via Cloud Routers and VPNs, and dedicated interconnection to bring Google's stellar network to your door.
Building globally available services comes with a distinct set of problems, which Google is committed to addressing. For instance, as a user base grows, geographical issues such as data consistency become more challenging. To solve this problem, Google created Cloud Spanner—a strongly consistent relation database that scales to thousands of nodes across the world. Content-heavy service providers look to provide a consistent experience across their user base. On a global scale, this can become challenging due to network limitations such as latency and congestion. To address issues like these, Google offers worldwide CDN services via multi-regional Cloud Storage buckets. To enhance your global reach further, Google offers a range of extremely powerful load balancing solutions. With features such as anycast IP for simplified DNS, health check integrations, and content-aware routing, Google's load balancers make it easy to reap the benefits of a global presence.
Google's service offerings give developers the freedom to choose how much control they want over the system. For example, a team looking to build a data analytics process can choose from solutions ranging from fully managed (Dataflow), partially managed (Dataproc), to fully self-managed (Hadoop on Compute Engine). On the application side, solutions can range from a fully serverless model with Cloud Functions, managed PaaS solutions leveraging App Engine, the partially-managed Google Kubernetes Engine, to the extreme of running applications on Compute Engine with load balancers, managed instance groups, and backend services.
This continuum of service offerings is common across many areas of Google Cloud and embodies the philosophy of developer and operations enablement. The decision on which solution best fits a specific need is, of course, not entirely so clear cut, but it is worth noting that the services offered on GCP are as diverse within specific problem spaces as they are across separate problem spaces. Google looks to provide specialized tools rather than adopt a one-size-fits-all approach.
Google is betting big on data. With so much business value being driven by data analytics, many modern technology companies are betting big on big data as well. Google offers a cohesive suite of tools to help you quickly and easily build out analytics solutions without getting bogged down in infrastructure management. From world-class data warehousing and analytics with BigQuery, to self-balancing data-processing pipelines on Dataflow, Google Cloud has tools to fit any need.
Teams can quickly start their data migration journey by moving existing Hadoop and Spark workloads to managed clusters on Dataproc. Rounding out these tools are services such as Pub/Sub messaging, Dataprep, and Google Data Studio for a fully managed, serverless, democratized analytics platform.
To further drive predictive analytics, Google is dedicated to bringing machine learning to the masses. With Cloud Machine Learning, users can easily get started with the powerful Google-born open source TensorFlow framework. This means developers can leverage the same tools Google uses internally to accomplish tasks such as speech and image recognition, all the while maintaining deep integrations with the rest of the big data offerings on GCP.
Google is making waves and building a reputation as the Open Cloud. Building on the core belief that developers should want to use GCP, Google consistently adopts and drives open standards and open source tools and frameworks. By open sourcing projects such as Kubernetes and TensorFlow, these projects are able to grow rapidly and organically. Instead of creating vendor lock-in, Google is then able to capitalize on these open source projects by providing the best developer experience on top of them, as seen in Kubernetes Engine and Cloud Machine Learning.
By adopting and adhering to open standards, Google further reduces the risk of vendor lock-in, and provides a lower barrier to entry for teams looking to move to managed services. This can be seen in a number of products, such as Cloud Bigtable, which adheres to the open-source Apache HBase interface, and Cloud Endpoints, which adheres to the OpenAPI specification. By working together with the wider community, Google creates a transparent, symbiotic relationship with developers that facilitates progress throughout the technology industry.
In addition to driving open sourcing and open standards, Google Cloud continuously innovates on ways to make more solutions feasible for organizations of all sizes. By providing per-second billing on compute resources, more teams can afford to build out massive-scale solutions such as spinning up hundreds of virtual machines for short-lived but intensive workloads. Innovating on the traditional approach of provisioning virtual machines, Google offers custom machine types that help developers optimize their use of cloud resources. With very competitive pricing, automatically applied sustained-usage discounts, proactive alerting on underutilized resources, and generous free tiers, Google helps teams minimize costs. Very often, Google Cloud is not just the best choice; it's the cheapest.
The folks at Google understand the perceived risks in adopting the public cloud. Giving up control over your infrastructure can be scary. Every business is unique in their technology needs, and there are many unknowns. Instead of a one-size-fits-all model of cold documentation and endless FAQs, Google is dedicated to providing a customer-centric experience to help you build the best possible solutions on GCP. The Google Cloud team has internalized this ideology and formalized it into the practice of Customer Reliability Engineering.
With Customer Reliability Engineering, or CRE, Google is taking a vested interest in the reliability of your applications. This goes beyond the reliability of the underlying cloud services your application is running on. CRE realizes that the primary concern of teams running applications on Google Cloud is not the reliability of GCP itself, but rather the reliability of the applications those teams are responsible for. The reliability of the Google Cloud infrastructure is, of course, a factor in the reliability of your applications (and those grounds are well covered—see https://landing.google.com/sre), but Google is determined to go beyond delivering a stable platform to ensuring that the applications running on the platform are built for reliability as well.
A major point of contention for some businesses considering migrating to a public cloud is security. Customers trust you with safeguarding their identity and privacy—a responsibility that should be held in the highest regard. Google understands the weight of this responsibility, and the engineers of Google Cloud are dedicated to extending the same level of security to your customers that they provide to their own. From purpose-built security chips on GCP servers to globally available private network solutions, Google is dedicated to providing security at all levels of the platform.
Google also understands the importance of making security easy. With design features such as encryption at rest and services such as the Data Loss Prevention API and Cloud Key Management, the Google Cloud team is driving customer security by making it accessible and approachable. In bringing security to the forefront of their offerings, Google is helping to make security one of the primary motivations for public cloud adoption.
Since you are reading this book, it is assumed that your team is either considering leveraging Google Cloud or is already doing so. In doing so, you will be joining a group of diverse and rapidly growing companies across business sectors and geographic locations. From large enterprise companies such as The Home Depot and Coca-Cola, to technology companies such as Evernote and Vimeo, many people are finding that Google Cloud Platform has the tools and services they need to succeed.
This growing traction also creates enormous opportunity for businesses to learn from each other in the wide range of problems being solved on GCP. For example, looking at how Spotify leverages Google Cloud to stream songs to their customers, we can learn about the viability of multi-regional Cloud Storage buckets as a global CDN. By studying Niantic, we can see the power of Google Kubernetes Engine to rapidly scale applications to thousands of nodes. Read about these companies and many more at https://cloud.google.com/customers.
In addition to the many amazing companies already leveraging Google Cloud, Google is strongly focused on developing a network of partners with major companies such as Cisco, Pivotal, and Salesforce. These partner companies are offering services and integrations that make it easier than ever to bring your business to the cloud.
The public cloud is an exciting and rapidly changing technology arena. With so many organizations either already leveraging public clouds or looking to make the transition, providers are in a constant arms race to win and retain customer business. This is great for customers, as it drives down costs and pushes cloud providers to deliver the products and services customers need to succeed in the cloud.
Amazon, Microsoft, and Google all offer excellent public cloud platforms, and choosing the right one for your needs is an important decision. Google looks to stand out as a leader in scalable managed services and big data. Google offers customers access to many of the same tools Google uses internally. By leveraging their wealth of knowledge and experience in running major services such as Search and Gmail, Google has built a platform that offers a high level of scalability and reliability. As we'll see in the following chapters, Google Cloud Platform makes it easy to build powerful cloud-based solutions.
The Google Cloud Platform (GCP) consists of a large and rapidly expanding catalog of products and services. Often, the solutions you build will touch many of these products and services, creating a need for a centralized and easy-to-use set of tools to interact with GCP. These tools must offer developers a way to quickly create and manage cloud resources across the Google Cloud Platform.
The quality of the developer environment is important in getting the most out of the cloud. Google looks to meet this need in a holistic manner with an easy-to-use web application offering core developer tools, including intuitive user interfaces and a readily available interactive shell with command-line tools. This chapter serves to get readers started using GCP and familiarize readers with the primary interfaces they will use when interacting with GCP.
In this chapter, we will cover the following topics:
Getting up and running with projects on Google Cloud
Familiarizing yourself with the Google Cloud Console
Interacting with the Google Cloud Shell
Developing and testing applications in-browser with the Cloud Shell Code Editor and Web Preview
Using other tools and plugins for interfacing with GCP
Before we begin diving into the various ways of interacting with GCP, you'll need to either create or join a Google Cloud project. But first, what is a project? A project is the fundamental organizational component of Google Cloud, containing cloud resources, collaborators, permissions, and billing information. For cloud resources, projects help associate things such as networks, virtual machines, and databases, and segregate them in a logical manner. For users and permissions, projects makes it easy to define team roles and their associated rights.
Services and accounts in one project will be walled off from resources in other projects unless explicitly enabled. This provides a fundamental layer of security that can be built upon further via Google's Identity & Access Management(IAM) system and the principle of least privilege. As we'll see, this has powerful implications on the overall operations and security of the systems you will build.
How your team chooses to organize services into projects plays a major determining factor in the overall architecture of the systems you will build. Since all resources in a project roll up to centralized permission management and billing, a common pattern in larger organizations is to create projects associated with business cost centers or departments. For example, in a given organization, you may have an IT cost center for an inventory management department and another for a customer service department, each with separate budgets. In this case, it may be ideal to create separate projects for each department to simplify team management and payments.
Projects are often separated further based on product teams within the same organization, often for reasons of budgeting or access control. The division of projects based on organizational structure makes it easier to secure data and APIs by limiting access to those with a valid need. This both minimizes opportunities for malicious users and lowers the risk of development teams accidentally stepping on each other's toes.
Another common pattern is to create separate projects based on the application development life cycle. By splitting projects into development life cycles, teams can create isolated environments for development, QA, and production services. As with projects based on organization structure, life cycle-specific projects offer an ideal primary layer of security by drawing clear boundaries on resource access.
For services, this means that non-production systems cannot (unintentionally or maliciously) affect production systems. For team members, this provides a clear way to implement a strategy of separation of concerns. In both cases, this can help prevent compromised credentials and resources from affecting production services or accessing sensitive data. Much more detail will be provided on this in Chapter 3, APIs, CLIs, IAM, and Billing.
Google Cloud projects are tied to user accounts and domains. Any Google account can create and be granted access to GCP projects. For individual use, this can be a standard Google account as used in Gmail. For organizations, this is often done through G Suite and Cloud Identity for domain-level user management backed by organizational identity services such as LDAP and SSO. This provides a rich control plane for administrators to manage user access across projects either individually or as groups. It also enables monitoring and alerting on policy changes, and allows integration with other systems such as third-party change management services.
Let's go ahead and create a Google Cloud project. To get started, visit https://console.cloud.google.com and log in to a valid Google account or any Cloud Identity enabled account. If this is your first time logging in, you'll need to accept the terms of service. After doing so, you will be presented with the Google Cloud Console (we'll cover this in detail in just a bit).
If you are not already a member of an existing project, you will need to create one now. This can be done by clicking the
Select a project
button on the top toolbar and clicking Create project, shown as follows:
You'll need to provide a project name, which will be the human readable identifier for your project. Google will automatically convert this name to a globally unique identifier, which you will often use when interacting with Google Cloud APIs. If you prefer, you can manually edit this globally unique project ID before creating the project, shown as follows:
After confirming, GCP will start the process of initializing your new project. This process can take a few minutes to complete, and can be monitored via the notifications icon in the top right of the
User Interface
(
UI
). Once finished, you'll need to enable billing to get started building solutions on GCP. Project billing is managed through billing accounts, which associates a user or organization with payment information. A billing account can be used for multiple projects and project billing can be managed by any project billing administrator. If you are the project creator, you will be able to manage project billing by default.
To enable billing, go to the
Project Billing
page by navigating in the Navigation menu, side menu to
Billing
, or visit
https://console.cloud.google.com/billing
. Click
Link a billing account
and
CREATE BILLING ACCOUNT
, which will guide you through the process of setting up a billing account to associate with this project. If the thought of doing this scares you, fear not! There is quite a bit you can do on GCP without spending any money, though project billing must be enabled to fully take advantage of this:
Google Cloud offers a generous free trial that, at the time of writing this book, includes credits worth $300 USD to be used within 12 months. This free trial credit can be used on all Google Cloud services, with some limitations such as Compute Engine instance hardware configurations. With moderation and diligence in releasing unused resources, the free trial credit can go a very long way. Once the free credit is expended or the allotted time runs out, the account will be paused and you'll have the option to enable billing by upgrading to a paid account.
In addition to the free trial, Google offers a free tier on many of their platform services that are available during and after the free trial window. These free tier quotas are more than enough to build a foundational understanding of many services on GCP. For example, users can run a simple web application via App Engine Standard, a persistence layer via Datastore, content hosting in Cloud Storage, and incorporate event-driven functionality via Cloud Pub/Sub and Cloud Functions—all within the free tier. Along with providing an excellent learning opportunity for new users, this free tier makes it possible to build out a small proof of concept that is ready to scale seamlessly with user adoption.
The Google Cloud Platform offers a range of channels for managing and interacting with project resources and services, including the Google Cloud Console, command-line tools, mobile applications, IDE plugins, web APIs, and libraries for many popular programming languages. The Google Cloud Console is the primary interface for GCP, offering a one-stop shop for all of your cloud operations. Combined with the power of the Google Cloud Shell, there is very little you cannot do from the Cloud Console.
With a simple and straightforward UI, the Cloud Console also offers an approachable way to acquaint oneself with the various features and tools of Google Cloud as shown in the following screenshot:
The Google Cloud Console dashboard offers a high-level view of the state of your project, as well as important updates about GCP.
Upon logging in, you'll be presented with the DASHBOARD view of your GCP project, as shown in the preceding screenshot. This provides a centralized, high-level view of the state of your project, as well as many convenient links to help get you started using Google Cloud, such as documentation, active issues and platform status, and general announcements and news. Sections are broken down into customizable widgets. By clicking the Customize button, you can tailor your experience by showing or hiding widgets, reordering widgets, and configuring graphs for certain resources such as Compute Engine VMs.
Along with the DASHBOARD view, a real-time stream of activity within your project is available in the ACTIVITY view. This offers users a quick way to see what high-level operations are taking place such as administrative tasks, error alerts, information about quotas, and changes to project settings. The ACTIVITY view includes powerful search and filtering functionality to siphon out the information you care about. This offers a good way to get a quick idea of what's going on across your project and should be one of the first places you look when diagnosing problems.
On the left, you'll see the collapsible Navigation menu, which is immediately available throughout the Google Cloud Console. All of the major Google Cloud offerings have associated console views, and this menu allows quick navigation between them. Products are organized into high-level groups such as compute, storage, and networking, each containing the related service offering for that group. Many of the menu items expand to list more product-specific views for immediate access.
