Learn Ethereum. E-Book

Xun (Brian) Wu

• Herausgeber: Packt Publishing
• Kategorie: Wissenschaft und neue Technologien
• Sprache: Englisch

Seitenzahl: 992

Veröffentlichungsjahr: 2023

Beschreibung

Ethereum is a blockchain-based, decentralized computing platform that allows you to run smart contracts. With this book, you’ll discover the latest Ethereum tools, frameworks, wallets, and layer 2, along with setting up and running decentralized applications for the complete, end-to-end development experience.
Learn Ethereum, 2nd Edition is a comprehensive overview of the Ethereum ecosystem, exploring its concepts, mechanisms, and decentralized application development process. You’ll delve into Ethereum's internals, technologies, and tools, including Ethereum 2.0 and the Ethereum Virtual Machine (EVM), gas, and its account systems. You’ll also explore Ethereum's transition to proof of stake, L1/L2 scaling solutions, DeFi protocols, and the current marketplace. Additionally, you’ll learn about EVM-compatible blockchains, connectivity techniques, and advanced topics such as sharding, off-chain scaling, DAOs, Metaverse, and NFTs.
By the end of this book, you’ll be well-equipped to write smart contracts and develop, test, and deploy DApps using various tools, wallets, and frameworks.

Leseprobe

Learn Ethereum

A practical guide to help developers set up and run decentralized applications with Ethereum 2.0

Xun (Brian) Wu

Zhihong Zou

Dongying Song

BIRMINGHAM—MUMBAI

Learn Ethereum

Copyright © 2023 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the authors, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

Group Product Manager: Reshma Raman

Publishing Product Manager: Arindam Majumder

Book Project Manager: Kirti Pisat

Senior Editor: Nathanya Dias

Technical Editor: Devanshi Ayare

Copy Editor: Safis Editing

Project Coordinator: Farheen Fathima

Proofreader: Safis Editing

Indexer: Rekha Nair

Production Designer: Alishon Mendonca

DevRel Marketing Coordinator: Nivedita Singh

First published: September 2019

Second edition: September 2023

Production reference: 2230823

Published by Packt Publishing Ltd.

Grosvenor House

11 St Paul’s Square

Birmingham

B3 1RB, UK.

ISBN 978-1-80461-651-2

www.packtpub.com

Contributors

About the authors

Xun (Brian) Wu is a senior blockchain architect and consultant. With over 20 years of hands-on experience across various technologies, including Blockchain, big data, AI, systems, and infrastructure, Brian has worked on more than 50 projects in his career.

He has authored nine books, which have been published by O’Reilly, Packt, and Apress, focusing on popular fields within the Blockchain industry. The titles of his books include: Learn Ethereum (First Edition), Learn Ethereum (Second Edition), Blockchain for Teens, Hands-On Smart Contract Development with Hyperledger Fabric V2, Hyperledger Cookbook, Blockchain Quick Start Guide, Security Tokens and Stablecoins Quick Start Guide, Blockchain by Example, and Seven NoSQL Databases in a Week.

I would like to thank my family and friends for their patience and support throughout this endeavor.

Zhihong Zou is an innovative technology leader with over 20 years of experience delivering cutting-edge enterprise solutions. He has led transformative solutions in telecommunications, healthcare, and government that leverage blockchain, AI/ML, and modern cloud architectures to solve complex business challenges. As an insightful visionary and trusted advisor, Zhihong is passionate about driving digital transformation and future-proofing organizations. He holds an M.Sc degree in computational mathematics and an M.Sc in computer science.

Zhihong has authored multiple papers on numerical computing and two popular editions of the book Learn Ethereum. In his free time, Zhihong enjoys sharing his blockchain expertise as an advocate and writer.

I would like to thank my family for their understanding, encouragement, support and patience throughout this journey.

Dongying Song has been a noteworthy leader in Blockchain, Ethereum, big data, ML, data science, and the cloud application development space for over 8 years. As an experienced software engineer and data scientist, she has worked for top-tier banks and pharmaceutical and technology companies. Dongying holds a master's degree in statistics from Columbia University and majored in mathematics during her undergraduate years. Her recent activities focus on Blockchain, Ethereum, and smart contract developments.

About the reviewer

Mayukh Mukhopadhyay is a technology consultant with more than a decade of experience in designing business continuity solutions amidst of complex digital transformation programs. He did his M.Eng from Jadavpur University and MBA from IIT Kharagpur. He is currently pursuing Ph.D. (industry track) from IIM Indore. He has authored the book Ethereum Smart Contract Development and also served as the technical reviewer of the Ethereum Cookbook. In his leisure time, he peer-reviews various blockchain and business research papers for Scopus-indexed journals and IEEE conferences.

I am perennially indebted to my daughter Abriti and spouse Mrittika for making sacrifices in every walk of life to support my academic endeavors.

Table of Contents

Preface

Part 1: Blockchain and Ethereum Basics

1

Blockchain and Cryptocurrency

Technical requirements

Introducing blockchain technology

Decentralized P2P networks

How does blockchain work?

Rehashing cryptography

Public-key cryptography

Cryptographic hash function

Digital signatures

Anatomizing a blockchain consensus mechanism

What is consensus?

Proof-of-work

Proof-of-stake

Forking

Understanding Bitcoin and cryptocurrency

Bitcoin basics

What is a wallet?

Transactions, UTXO, and account balances

Genesis block and coin supply

How does Bitcoin payment work?

Limitations in Bitcoin

Altcoins

Overview of blockchain use cases in the industry and government

Financial services

Healthcare

Blockchain use cases led by government

Ushering in the world of Ethereum

Smart contract

EVM

Account

Summary

2

Ethereum Architecture and Ecosystem

Technical requirements

Introducing the Eth1 Architecture

Ethereum – the world computer

Web 2.0, Web3, to the Metaverse

DApps

Diving deep into Eth1

Accounts

Two types of accounts

EOAs

CAs

Transactions and messages

Smart contracts

Ether and gas

The EVM

Addresses and wallets

Examining mining in Eth 1.0

Mining and the consensus protocol

Ethereum transactions and block structure

Transaction validation and block verification

Understanding scaling challenges in Ethereum

The blockchain scalability trilemma

Modular blockchain architecture

Ethereum scaling solutions

Introducing Beacon Chains and Eth2

PoS in Eth2

How the beacon chain works

Benefits from the transition to PoS

Merging Eth1 and Eth2

Merging Eth1 data into Eth2

Scaling Ethereum with rollups

Optimistic rollups

ZK rollups

Summary

3

Decentralized Finance

Technical requirements

Introducing decentralized finance

Mastering Ethereum token standards

Definition of cryptocurrency

Crypto funding mechanism

Ethereum token standards

Analyzing stablecoins and MakerDAO Dai

Understanding DeFi protocols

Basic DeFi constructs

Lending and borrowing

Decentralized exchanges

Decentralized derivatives and insurance

Yield farming and DeFi continuum

Making sense of cryptoeconomics

DeFi after the collapse of FTX

Summary

4

EVM-Compatible Blockchain Networks

Technical requirements

Understanding EVM blockchain ecosystems

Introducing BSC

Consensus mechanism in BSC

Block creation and finality

BNB governance

BNB ecosystem and roadmap

Scaling Ethereum with Polygon chains

How Polygon PoS and Plasma work

Block production and checkpoints

Consensus mechanism in Polygon PoS

Multi-chain solutions with Polygon Edge

Polygon ecosystem and roadmap

Diving deep into high-performance Avalanche chains

Avalanche blockchain architecture overview

Avalanche consensus mechanism

Subnets and enterprise blockchains

Governance in Avalanche chains

Avalanche ecosystem

Bridging interoperability gaps between blockchains

Cross-chain integration challenges

Common cross-chain bridge designs

Trustless bridge solutions using Polkadot

Generalized bridge solutions with Cosmos

Decentralized oracle as a bridge

Glancing over non-EVM blockchain networks

TRON blockchain overview

Introduction to Solana

Summary

5

Deep Research and the Latest Developments in Ethereum

Technical requirements

Understanding the challenges in distributed systems

The CAP theorem

Horizontal scaling versus vertical scaling

Scaling Ethereum

Making sense of the Ethereum roadmap

Pivotal to rollup-centric

Overview of the post-merge Ethereum roadmap

Sharding and data availability sampling

Data availability sampling

Discovering MEV and PBS

Overview of MEV

MEV implementation in Ethereum

Proposer builder separation

zkEVM and EVM improvements

Overview of zk-SNARK

Types of zkEVM implementations

Workings of zkEVM

Smart contract wallets and account abstraction

Account abstraction and EIP 4337

How a smart contract wallet works

DAOs

Introduction to DAOs

DAO governance case study using Uniswap

NFTs, Web3, and Metaverse

Introduction to a world of NFTs, Web3, and Metaverse

The current state of NFTs

Web3 and the future of the internet

The Metaverse, virtual reality, and future interactions

Summary

Part 2: Ethereum Development Fundamentals

6

Fundamentals of Solidity

Technical requirements

Introducing Solidity

Tools for the Solidity development environment

Learning about the fundamental programming structure in Solidity

The layout of a Solidity source file

State variables

Built-in data types

User-defined data types

Functions

The pure, view, and payable functions

Constructor and destructor functions

Fallback function

Function overloading

Function modifiers

Events

Global variables, contextual variables, and functions

Enabling the contracts library

Understanding inheritance, abstract contracts, and interfaces

Interface

Abstract contract

Inheritance

Multiple inheritance and the diamond problem

Function overriding

Examining smart contract execution under the hood

Mastering advanced programming concepts in Solidity

Smart contract security

Keeping contracts simple and modular

Use the checks-effects-interactions pattern

DoS with block gas limit

Handling errors in external calls

Best practices in smart contracts

Writing upgradable smart contracts

Economic consideration of smart contract execution

Types of smart contract

Putting it all together – rental property leasing

Summary

7

Web3 API Fundamentals

Technical requirements

Introducing Web 3.0

Setting up an Ethereum environment

Installing Geth

Setting up a private Ethereum network

Generating and configuring the genesis file

Initializing the chain instance

Starting a Geth node

Connecting to Ethereum networks

Learning the fundamentals of web3.js – the Ethereum JavaScript API

web3.js project setup

web3.js Account

web3.js ABI

Web3 providers

Setting up the Ethereum testnet environment

The web3.js Transaction API

Getting started with web3.py

Prerequisites

Introducing Web3j – Web3 Java Ethereum DApp API

Setting up a simple Spring Boot application

Checking the balances of Ethereum accounts

solc and the Web3j CLI

Deploying smart contracts using Web3j

Transacting with a smart contract

Querying the state of a smart contract

Summary

8

Developing Your Own Cryptocurrency

Technical requirements

Understanding token standards

ERC-1155

Batch transfers

Batch balance

Batch approval

ERC-1155’s TokenReceiver

Setting up an Ethereum development environment

Working with Truffle

Creating an ERC-20 token – MyERC20Token

Creating basic token information

Defining and implementing the ERC-20 interface

Assigning an admin role

Working with the whitelist function

Locking and unlocking an account

The mint and burn tokens

Creating an ERC-721 token – DigitalArtERC721Token

Designing the decentralized digital art marketplace

Setting up the DigitalArtERC721Token project

Creating the basic token information

Defining and implementing the ERC-721 standard

Defining the art and art transaction struct

Creating a non-fungible digital art token

Implementing the buyArt() function

Implementing the resellArt() function

Implementing the findArt() function

Implementing the findMyArts() function

Implementing the findAllPendingArt() function

Getting all art transactions through getArtAllTxn()

Creating an ERC-1155 token – ERC1155NFTToken

Uploading NFT images

Writing the ERC-1155 NFT token contract

ERC-1155

Summary

Part 3: Ethereum Development Fundamentals

9

Smart Contract Development and Test Fundamentals

Technical requirements

Understanding Remix development fundamentals

Working with the Solidity Compiler

Testing

Deploying and running transactions

Debugging

Understanding development using Truffle and unit testing

The Truffle console and development features

Running a Truffle migration

Truffle unit testing

Truffle Box

Understanding Hardhat development and unit testing

Hardhat Runner

Hardhat Network

Project structure

Installation and environment setup

Hardhat project setup

Creating and compiling the ERC1155 contract

Testing the contract

Deploying the contract

Security testing

Static and dynamic analysis

Working with a linter

Summary

10

Writing a Frontend to Build the NFT Marketplace DApp

Technical requirements

Understanding DApps

Creating your first NFT

Set up a wallet

Obtain Ether (ETH)

Visit OpenSea

Create NFT item

Working with the Web3 JavaScript API

Setting up a DApp development environment

Installing Ganache desktop

Creating a development workspace

Setting up the project for our DApp

Deploying a smart contract

Installing MetaMask and connecting to Ganache

Setting MetaMask Ganache accounts

Building frontend UI components

Setting up project dependencies

Getting the instance of a deployed contract

Building a navigation bar

Running the digital art market DApp

Publishing your art (the first account)

Publishing your art (the second account)

Summary

11

Ethereum Tools and Frameworks

Technical requirements

Understanding the Ethereum development tools and framework

Layer 1 node connections

Working with Infura

Working with the Infura Ethereum API

Using Remix with Infura

Introducing Alchemy

Testing tools

The Ethereum client API

Working with Ethereum storage

Working with Swarm

Getting to know the popular smart contract libraries

Working with OpenZeppelin

Setting up a dev environment

Access control

Programming languages and integrated development environments

Vyper

Summary

Part 4: Production and Deployment

12

Setting Up Ethereum Private Chain

Technical requirements

Understanding private Ethereum networks

Setting up a local private network

Prerequisites

Installation on MacOS with Homebrew

Creating a private Ethash network using Besu

Private enterprise Ethereum in production use

Hyperledger

Hyperledger Fabric

Hyperledger Sawtooth

Corda

Quorum

Resources for private enterprise Ethereum development

Ethereum node services

Ethereum clients

Summary

13

Deployment of Your DApps

A brief introduction to test networks

Deploying a smart contract to a test network using Truffle

Installing Truffle

Deploying a smart contract to the Goerli testnet using Truffle

Deploying a smart contract to the Sepolia testnet

Deploying a smart contract to a test network using Hardhat

Installing Hardhat

Deploying a smart contract to the Goerli testnet using Hardhat

Summary

14

Building Ethereum Wallets

Technical requirements

Understanding the wallet technology

Understanding non-deterministic and deterministic wallets

Mnemonic code words (BIP-39)

Generating a private key in Ethereum

Multiparty signature wallets

Stealth addresses

Basic stealth address protocol

ISAP

Dual-Key stealth address protocol

Confidential transactions

Creating an Ethereum wallet

Creating a non-deterministic wallet

Viewing the Keystore file generated by the geth command

Creating a non-deterministic wallet with MyCrypto

Creating an HD wallet

Working with third-party Ethereum wallets

Wallets on different platforms

Smart contract wallets

Hot and cold wallets

Transferring funds between wallets

Summary

15

Oracles, Technology, and Layer 2 in Practice

Technical requirements

Understanding oracles

Examples of blockchain oracles

Event data for prediction markets

Blockchain network

Types of oracles

Working with oracles

Introduction to Chainlink

Chainlink VRF DiceRollGame

Diving deep into cross-chain technology

How does a cross-chain bridge work?

Vulnerabilities in a cross-chain bridge

Popular cross-chain bridges

Cross-chain in practice

Bridge tokens from Ethereum to Avalanche

Bridging ETH from Ethereum to Polygon (MATIC)

Working with Layer 2

Optimistic rollups

ZK rollups

Writing a ZKP program using ZoKrates

Summary

Part 5: Conclusion

16

Conclusion

Technical requirements

Facing the challenges of Ethereum and blockchain

Scalability and economic security

User experience, privacy, and mainstream adoption

Ethereum governance

CBDC and government regulations

Emerging trends in Ethereum and the blockchain ecosystem

Application blockchain

Enterprise blockchain and blockchain as a service

The multi-chain world

AML/CFT, KYC, and decentralized identity

Blockchain-powered Metaverse

Meeting the future of Ethereum

Summary

Index

Other Books You May Enjoy

Part 1: Blockchain and Ethereum Basics

In this Part, we will start with the concept of Blockchain and dive deep into Ethereum fundamentals, its architecture and its ecosystem. We will then introduce Decentralized Finance (DeFi) and analyze various DeFi protocols. We will discuss various Ethereum Virtual Machine (EVM) compatible Blockchains and cross-chain integration mechanisms. To keep you at the forefront of this evolving field, we will also bring you up to speed with the latest developments and most advanced research topics within the Ethereum community.

This part comprises the following chapters:

Chapter 1, Blockchain and CryptocurrencyChapter 2, Ethereum Architecture and EcosystemChapter 3, Decentralized FinanceChapter 4, EVM-Compatible Blockchain NetworksChapter 5, Deep Research and the Latest Developments in Ethereum

1

Blockchain and Cryptocurrency

It is a common belief that the bankruptcy filed by Lehman Brothers, a Wall Street banking giant, on September 15, 2008, triggered the global financial crisis in 2008-2009. Excessive risk exposure to the subprime mortgage and financial derivative markets by large banks almost brought down the global financial system. The crisis was the ultimate consequence of a fundamental breakdown of trust in the relationship between customers and the financial institutions that should have been serving them.

Shortly after that, Satoshi Nakamoto, a mysterious and anonymous entity, published a whitepaper on October 31, 2008, called Bitcoin: A Peer-to-Peer Electronic Cash System, which is considered the origin of Bitcoin and all cryptocurrencies. Satoshi proposed a completely decentralized approach for Peer-to-Peer (P2P) payment without central banks or intermediaries. He outlined the principles and functions of what would be developed and introduced as Bitcoin in the following year.

The central technology behind his invention is referred to as blockchain and has since evolved well beyond Bitcoin and digital payment. It is now a suite of technologies, forming the foundation of distributed ledgers and cryptocurrency. No one knows who or where Satoshi is, whether it is one individual or a group, but the whitepaper has profoundly changed money, digital and cryptocurrencies, business, and the world. You can learn more about the Bitcoin whitepaper authorship debate at https://www.judiciary.uk/wp-content/uploads/2022/08/Wright-v-McCormack-Judgment.pdf.

The purpose of this book is to help you to understand blockchain technologies, introduce you to the tools and technologies of the Ethereum ecosystem, and get you started with developing smart contracts and end-to-end decentralized applications.

In this chapter, we will start with basic concepts in Bitcoin, Ethereum, cryptocurrency, and blockchain.

In this chapter, we will cover the following topics:

Introducing blockchain technologyRehashing cryptographyAnatomizing the blockchain consensus mechanismUnderstanding Bitcoin and cryptocurrencyOverview blockchain use cases in the industry and governmentUshering in the world of Ethereum

Technical requirements

All of the source code in this book can be found at the following GitHub link: https://github.com/PacktPublishing/Learn-Ethereum-Second-Edition/.

Introducing blockchain technology

You might have heard the parable of the blind men and the elephant. It is a folktale about six blind men’s individual descriptions of the same elephant based on their own perceptions from touching particular parts of the animal, each of them giving very different descriptions of what they think the creature looks like. It highlights the fact that different perspectives can lead to distinct viewpoints, emphasizing the limits of perception and the importance of a complete context.

When Satoshi invented Bitcoin, the fundamental concept of its vision was to build a blockchain, a shared public ledger (longest Proof-of-Work (PoW) chain), that verifies and immutably records all transactions through a decentralized computer network (P2P network) and a consensus mechanism with computational proof. Satoshi thus came up with an elegant solution to the double-spend problem in digital money. A double-spend is an attack where someone tries to spend money in a transaction that isn’t actually available anymore as the money has already been spent.

Blockchain is a new elephant in the digital world. To most of the public, blockchain is nothing but an obscure pseudonym for all cryptocurrencies, including Bitcoin, Ethereum, and more. So, what is blockchain? What does a blockchain look like? How does it work? Where can we use blockchain? Do you need a blockchain? There are many ways to describe a blockchain from different perspectives, but there is no universal definition of a blockchain.

On the contrary, there are prevalent debates over the essential attributes or qualities of a blockchain. It is perceived as a new architecture using existing technologies, the next generation of the internet and web, a future database and distributed shared ledger, the new Napster (a P2P file-sharing system used in the 90s) with a pure decentralized P2P network, a cryptocurrency, a trustless secure transaction system, and so on. In reality, it is all of these. Only by combining all of these perspectives can we understand the whole picture of blockchain technologies and get a sense of the true potential of blockchain.

The following picture illustrates different viewpoints of blockchain technology:

Figure 1.1 – Different viewpoints on blockchain technologies

So, what is a blockchain anyway? Think of blockchain as a new architecture paradigm and a new trust protocol. It is a computer science primitive forming the foundation of most cryptocurrencies and decentralized applications. It is a P2P transaction model that can enable two parties to transact in a way that is tamper-resistant and cryptographically proven. As the technology behind Bitcoin and other cryptocurrencies, blockchain is an open, distributed ledger that can be simultaneously used and shared within a large, decentralized, publicly accessible network.

In essence, blockchain is a distributed shared ledger technology supported by three pillars, as shown in the following figure; these are P2P networks, cryptography, and a consensus mechanism:

Figure 1.2 – Key components of blockchain

To understand how blockchain works, let’s start with the fundamental concepts and key building blocks of blockchain technologies. Then, we’ll discuss the key differences between centralized, distributed, and decentralized systems. We will then dive into the blockchain data structure and discuss how transactions, blocks, and chains are maintained and how the network reaches a consensus on the state of the chain, as well as how to secure the blockchain with cryptographic technologies.

The following lists the key building blocks of blockchain technologies:

Transactions: A transaction is a value transfer between two parties. It could be a transfer of money, tangible assets, or cryptocurrency. Transactions are broadcasted to the blockchain network. They are validated and verified by all nodes and collected into blocks. Once the block reaches a certain depth — in Bitcoin, this is six blocks — those transactions in the block can be considered irreversible.Block: All verified transaction records are collected into a data structure called a block. It has a header and body part, where the header contains a cryptographic hash of the previous block, a timestamp, and a Merkle tree root hash of all transactions in the block. The body is the container of transaction data. A Merkle tree is like the digital fingerprint of transactions in the block, which we will discuss extensively later in this section.The chain of block (blockchain): A blockchain is a linked list of a chain of blocks. Blocks are linked together using a cryptographic hash as the pointer to the previous block.Decentralized P2P network: It is a P2P network in which interconnected nodes share resources between themselves without the use of a central authority or some sort of intermediary.Consensus protocol: The consensus protocol in blockchain is a set of rules that all network nodes will enforce when considering the validity of a block and its transactions. The consensus mechanism is the process used by the network nodes to achieve agreement on the network state. It is a fault-tolerant mechanism to ensure the reliability and integrity of the network.Mining: Mining is the process by which network nodes in blockchain systems add new blocks to the blockchain and get rewarded with crypto-incentives.

In the next section, we will discuss how P2P networks work.

Decentralized P2P networks

To explain how blockchain works, let’s look at the steps involved with the existing business model for completing a simple payment transaction.

A customer, Alice, needs to pay $10 to Bob, who is in a geographically distant region from Alice and happens to have an account in the same bank as Alice. She can make the payment either by visiting a bank branch or using the web. Let’s say she tries to do it online through the bank’s web portal. She will need to authenticate herself using her username and password and then put the transfer order in and wait for the bank system to confirm whether the transaction is completed.

As shown in the following diagram, in order to support such online banking activities in the traditional world, the bank has to establish an identity, access management system, and authenticate Alice’s login credentials. Behind the scenes, the bank needs to develop a bank web portal and a backend system to verify whether Alice has the right account with the bank and has enough money to pay Bob, upon which the bank can transfer $10 out of Alice’s account and put $10 in Bob’s account. The bank has to maintain a ledger to record the details of the transaction in a database and show the balance each person has.

The following diagram shows a centralized bank system model:

Figure 1.3 – Centralized bank system model

As the business grows, customers needs change with it. The traditional brick-and-mortar business model is being replaced by the digital banking and commerce model. This requires technological changes in the bank system too. Banks nowadays deploy a distributed system model to serve the ever-growing needs of their customers.

The following diagram shows the distributed bank system model:

Figure 1.4 – Distributed bank system model

The fundamental issue with the preceding centralized or distributed system model is the risk from single points of failure. Failure could come from malicious network attacks, system failures, or security and privacy breaches; it could come from business failures in the bank itself, which can cause millions of people to lose their homes due to the bankruptcy of big banks during a global financial crisis. It could happen due currency failure, such as the currency collapse in Venezuela, where the lifetime savings of average citizens suddenly became worthless overnight. Also, payments can be blocked due to government censorship.

Satoshi Nakamoto believed that the root problem with the traditional fiat system is all the trust required to make it work. Citizens have to trust the central bank not to devalue the currency. Consumers have to trust the bank to manage their money. But history has shown again and again that this trust is often breached.

Satoshi designed an elegant decentralized P2P electronic cash system, and the technology behind that, blockchain, is the solution, where transactions are maintained in a distributed shared ledger and replicated across a global P2P network. Security and privacy are ensured with cryptographic technologies, and transaction integrity is achieved through a consensus mechanism.

The following diagram shows a decentralized bank system model:

Figure 1.5 – Decentralized bank system model

As new transactions are made, they are broadcasted to all network nodes, and over time all transactions that have occurred are sequenced together in the public ledger and made available on all replicated network nodes, as shown in the following diagram:

Figure 1.6 – Decentralized public ledger

Now that we understand the difference between centralized and decentralized models, let’s see how blockchain works.

How does blockchain work?

Using the previous example, as shown in the following diagram, let’s assume Alice wants to buy something from Bob and she agrees to pay Bob 10 bitcoins (BTC):

Figure 1.7 – Money transfer between two parties

Let’s walk through the high-level processes step by step to demonstrate how blockchain works:

Create blockchain transactions: A transaction is a value transfer between two parties. When Alice sends 10 BTC to Bob, it will create a transaction with one or more inputs and two or more outputs, where the inputs reflect Alice’s account, and the outputs reflect which account(s) Alice intends to transfer to. The transaction is then digitally signed with Alice’s private key and broadcasted to the P2P network. The receiver will use the digital signature to verify the ownership of Alice’s funds. We will discuss digital signatures and cryptographic hash functions in detail in later sections.Validate the transactions and add to the transaction pool: Once the transaction is submitted to the blockchain network, the bookkeeper node (usually a full node in a P2P network that receives the transactions) will validate it according to protocol rules defined by the blockchain network. If the transaction is valid, the bookkeeper will add it to the transaction pool and relay the transaction to the peers in the network.Create the candidate blocks: Transactions in the transaction pool are collected into the block periodically. In a Bitcoin network, every 10 minutes, a subset of network nodes, called mining nodes or miners, will collect all valid transactions from the transaction pool and create the candidate blocks. The following diagram shows the structure of a candidate block:

Figure 1.8 – Creation of candidate blocks

As illustrated in the preceding diagram, the high-level processes are as follows:

The candidate block packages the recent valid transactions into the block structure based on block specifications.For each transaction in the package, it creates a cryptographic hash of the transaction data, recursively calculates the hash out of existing hashes, and creates a Merkle root of all transactions, as depicted in the following diagram:

Figure 1.9 – Merkle tree

The miner node looks for the latest block on the blockchain and adds its hash to the block header of the candidate block as the reference from the candidate block it intends to link to.

Mine the new block: Once the candidate block is created, the race starts for the chance to add new blocks and win the rewards. The process for such a race is called mining. The winning of the race is determined by the consensus mechanism. We will discuss different consensus mechanisms in later sections. In blockchain systems such as Bitcoin or Ethereum 1.0, the PoW consensus mechanism is applied to mining. Miners will keep trying to find a random number, the nonce in the block header structure, until the hash meets certain challenging conditions. For example, one such challenging condition is, the resulting block hash is smaller than a target number, or in some cases, the hash has a few leading zeros. In practice, every random number has the same chance to win the race, so practically, you can just start a loop through from 1 to 232 until it finds such a nonce, the unique hash meeting the condition. It requires huge CPU hashing power to find such a nonce. The challenging condition, called difficulty, can be adjusted based on the target number or bits in the block header structure. The difficulty in winning the race grows exponentially the smaller the target number is or the fewer bits are in the block header structure.Add a new block to the blockchain: The first winning node will announce the new block to the rest of the network for verification. Once the block is verified and approved by the majority of the network miners, it will be accepted and becomes the new head of the chain. Since all blocks are chained together by linking the hash to the previous block, any tampering with the ledger becomes impossible since it will require PoW on all previous transitions.

All miners have the chance to solve the puzzle, but only the winning miner has the authority to add the block to the chain and claim the bounty. Once the new block is added to the blockchain, all in-progress miners will stop their mining efforts on the newly added block and start the race again on a new block.

The following diagram summarizes the step-by-step process when new transactions are submitted to the blockchain network:

Figure 1.10 – How blockchain works in one picture

Cryptography plays a critical role in maintaining the transaction state in the blockchain and ensuring immutability. Cryptography is not new. In the next section, we will go over some key concepts in cryptography.

Rehashing cryptography

Cryptography is the study of secure communication techniques that prevent third parties or the public from reading private messages and allow only the intended recipient of a message to view its contents. It is the cornerstone of information security, which serves as the basis for delivering secure business applications and services. Modern cryptography concerns itself with the following five objectives of information security:

Confidentiality: This is the concept of preventing sensitive data from being accessible by any unauthorized entitiesIntegrity: This means protecting sensitive data from unauthorized changes during transit from one party to another partyAuthentication: This is the process of ensuring that user identity is truly what the user claims it to be, whether the user is human or a systemAuthorization: This is the concept of determining what actions an authenticated user is allowed to performNon-repudiation: When a user performs an action on data, the action must be bound with the user so that it can’t deny performing such actions

Cryptography deals with the design of algorithms for encryption and decryption, which are intended to ensure the secrecy and authenticity of the messages or transactions in question. Let’s start with some key elements in modern cryptography:

Encryption: This is the process of converting plain text or data into an unintelligible form, typically using a mathematical algorithm.Decryption: This is the process of reversing encryption, converting an encrypted message back into its original text and data.Hash: This is the process of converting any data block (arbitrary size or message) into a fixed-length hash code. A cryptographic hash function is a deterministic mathematical function performing such a conversion using cryptography, and it always maps to the same result for a given data block.

Cryptography is the linchpin and one of the three pillars of blockchain technology, along with the consensus mechanism and P2P network. It is used in many different forms, including, for example, wallets (for proof of cryptocurrency ownership), transactions (for PoW consensus), and P2P communication. In the following subsections, we will go over key blockchain-related cryptography topics, including public-key cryptography, digital signatures, cryptographic hashing, and Merkle trees.

Public-key cryptography

Public-key cryptography is a form of cryptographic function in which encryption and decryption are performed using two different keys — one public and one private key. They are generated in pairs. It is also called asymmetric cryptography. The public key can be shared with the public, but the private key is meant to be a secret code only known by its owner.

The keys are used in tandem too. Either of the two keys can be used in encryption, with the other one used for decryption. It is computationally improbable to determine the private key given only knowledge of the cryptographic algorithm and the public key.

Public-key cryptography is mostly used to do the following three things:

Secure the message transmission between two parties and ensure the confidentiality of messages or dataAuthenticate the sender and ensure the message is indeed sent from the senderCombine it with the cryptographic hashing function and provide a digital signature on a document before sending it to the receiver

We will go over the first two here and discuss digital signatures in the following section:

Public-key cryptography for confidentiality: In this case, as depicted in the following diagram, the receiver’s keys are used to encrypt messages between two parties during transmission. The sender (Alice) uses the receiver’s public key to encrypt a message, and the receiver (Bob), who holds their own private key in secrecy, can decrypt the messages using their private key:

Figure 1.11 – Confidentiality with public key

Public-key cryptography for authentication: In this case, as shown in the following diagram, the sender’s keys are used to authenticate the sender’s message. The sender uses its own private key to encrypt a message before sending it to the intended parties. The receiver can use the sender’s public key to confirm the message’s authenticity and decrypt it. The combination of this approach with the message’s cryptographic hashing function provides a digital signature, which we will discuss in the next section:

Figure 1.12 – Authentication with public key

Public-key cryptography is an essential technology underpinning wallets and transactions in the blockchain. We will discuss the Bitcoin wallet in the Understanding Bitcoin and cryptocurrency section.

Cryptographic hash function

A cryptographic hash function is an algorithm used to randomly convert a string of binary data into a condensed representation of a message — a message digest. Its output is called a hash value, digital fingerprint, digest, or checksum. It is deterministic and always results in the same hash value for a given message. It is capable of taking any size of data block and producing a fixed-size hash value that uniquely identifies the original data block. It is a one-way, irreversible function; the only way to recreate the input data is to try a brute-force approach with all possible values to see whether there is a match, which is almost computationally infeasible.

Notable hash functions include MD5, SHA-1, SHA-2 and SHA-3. Although they are still widely in use, MD5 and SHA-1 are cryptographically broken due to collision attacks found in the algorithm, and are thus no longer recommended.

Cryptographic functions have been widely used in blockchain technology, including the following:

Merkle trees: As we showed earlier, when a miner node pulls transactions from the transaction pool, it packages them in a block, where the block header has a field referencing the Merkle root of all transactions.Block chaining: Blocks in the blockchain are chained together with a reference to the previous block using a cryptographic hash.PoW: The PoW consensus algorithm itself is a game in solving a cryptographic hash function. We will discuss it in more detail in the Understanding the blockchain consensus mechanism section.

In addition to cryptographic hash functions, digital signatures have been broadly leveraged in blockchain networks too. We will discuss their usage in the next subsection.

Digital signatures

A digital signature is a set of algorithms for determining the authenticity and integrity of digital messages or documents. It assures the recipient that the message was indeed created by the expected sender and that the message was not altered during transmission. The sender cannot deny having sent the message.

When Alice sends a document to Bob, she will follow certain steps to digitally sign the document, as shown in the following diagram:

Figure 1.13 – Digital signature

The steps to digitally sign the document are as follows:

Calculate the message digest of the document Alice wants to send to Bob with a cryptographic hash function, usually any SHA-2 or SHA-3 algorithm.Encrypt the message digest with Alice’s private key, append the encrypted message digest to the original document, and send the combined message out.Once Bob receives the combined message from Alice, he will separate the encrypted message digest from the document itself. Bob will use Alice’s public key to decrypt the encrypted message digest.At the same time, Bob will calculate the message digest of the received document and compare the resulting message digest with the decrypted message digest to see whether there is a match. If yes, Bob is assured that the document originated from Alice without any tampering.

In blockchain, a digital signature is a way to prove ownership of the underlying cryptocurrency or electronic coin. When Alice needs to pay Bob 10 BTC, she will digitally sign a hash of the previous transaction, which can prove that Alice has ownership of the 10 BTC.

In summary, cryptography is one of three foundational pillars in blockchain technology. Public-key cryptography is the basis for blockchain wallets and transactions, and the cryptographic hash function is a key element underpinning the PoW consensus mechanism. A digital signature is used as proof of ownership of electronic coins or cryptocurrency.

In the next section, we will introduce and look at a blockchain consensus mechanism in detail and discuss how cryptography technologies are leveraged to reach consensus among decentralized parties.

Anatomizing a blockchain consensus mechanism

A fundamental problem in large-scale distributed systems is how to achieve overall system reliability in the presence of failures. Systems need to be fault-tolerant. This requires a process for distributed, often heterogeneous systems to reach a consensus and agree on the network state, whether it is a database commit or an action to take. In this section, we will discuss two types of consensus algorithms, PoW and PoS.

What is consensus?

Consensus in a blockchain is the process by which a network of mutually distrusted nodes reaches an agreement on the global state of the chain of blocks. In blockchain, transactions or data are shared and distributed across the network. Every node has the same copy of the blockchain data. Consensus allows all of the network nodes to follow the same rules to validate transactions and add new blocks to the chain, and therefore allows it to maintain uniformity in all copies of a blockchain.

Sometimes, it is also called a consensus mechanism or consensus algorithm. A consensus mechanism focuses on the