Operations Guide

Purpose and Scope

This guide provides operational guidelines for deploying, operating, and maintaining StableNet nodes and validator infrastructure.
It covers network deployment procedures, validator operations, and overall node maintenance.

For system architecture and component design, refer to the Core Architecture documentation.
For detailed consensus behavior, see the Anzeon WBFT Consensus Protocol.
For governance contract usage, refer to the Governance System documentation.

Network Deployment

Genesis File Creation

The StableNet network is initialized using a genesis file generated by the genesis_generator tool.
This tool creates the genesis block including Anzeon consensus parameters and system contracts.

Genesis Configuration Parameters

StableNet genesis configuration defines consensus- and governance-specific parameters under the anzeon section.

Parameter	Location	Description
wbft.epochLength	anzeon.wbft.epochLength	Number of blocks per epoch (default: 10)
wbft.blockPeriodSeconds	anzeon.wbft.blockPeriodSeconds	Target block time in seconds (default: 1)
wbft.requestTimeoutSeconds	anzeon.wbft.requestTimeoutSeconds	Consensus timeout in seconds (default: 2)
wbft.proposerPolicy	anzeon.wbft.proposerPolicy	Proposer selection policy (0 = round-robin)
init.validators	anzeon.init.validators	Initial validator addresses for the first epoch
init.blsPublicKeys	anzeon.init.blsPublicKeys	BLS public keys for each validator
systemContracts	anzeon.systemContracts	System contract configuration

System Contract Initialization

Each system contract is configured via initialization parameters at genesis.
Below is a list of key system contracts and their initialization parameters.

GovValidator (Address: 0x0000000000000000000000000000000000001001)

• members: Comma-separated list of operator addresses
• validators: Comma-separated list of validator addresses
• blsPublicKeys: List of validator BLS public keys (same order as validators)
• quorum: Minimum number of votes required to approve governance actions
• expiry: Governance proposal expiration time (seconds, default: 604800 = 7 days)
• gasTip: Gas tip applied to transactions from non-authorized accounts (wei)

NativeCoinAdapter (Address: 0x0000000000000000000000000000000000001000)

• name: Native token name (e.g. "WKRC")
• symbol: Native token symbol (e.g. "WKRC")
• decimals: Token decimals (typically 18)
• currency: Pegged fiat currency code (e.g. "KRW")
• masterMinter: GovMasterMinter contract address
• minters: Comma-separated list of contracts with minting authority

Bootnode Configuration

StableNet networks use bootnodes for peer discovery.
Bootnodes can be specified via the --bootnodes flag at runtime or via configuration files.

Bootnode example (--bootnodes flag)

--bootnodes enode://722f1f35c6fe2ac829aa9e0d7ef55bbe782ef3cea1e2354eefbe1584057a01b7@127.0.0.1:8589,enode://aa46e4a40781022e480bb05d1fd578fff164dc2ad379dc00370456c5ce9c2479@127.0.0.1:8589

Bootnodes are selected in the following priority order:

1. Explicitly specified via the --bootnodes flag
2. BootstrapNodes defined in the configuration file
3. Network preset flags (--mainnet, --testnet)
4. Default mainnet bootnode configuration

Network Flags and Configuration

The following command-line flags are commonly used to deploy and run StableNet networks.

Flag	Purpose	Default
--datadir	Data directory for databases and keystore	~/.stable
--mainnet	StableNet mainnet	-
--testnet	StableNet test network	-
--networkid	Network ID (auto-detected from chain ID if 0)	0
--port	P2P listening port	30303
--bootnodes	Comma-separated enode URLs for bootstrap	Network-specific
--maxpeers	Maximum number of peers	50
--nodiscover	Disable peer discovery	false
--nat	NAT port mapping mechanism	”any”

Validator Operations

Validator Key Structure

StableNet validators use three distinct cryptographic keys, each serving a different role in the consensus and governance systems.

Key Generation and Management

ECDSA Keys

ECDSA keys are generated and managed using the standard Ethereum keystore format.

# Generate operator key (for governance)
gstable account new --datadir /data/validator1

# Generate validator key (for block signing)
gstable account new --datadir /data/validator1 --keystore /data/validator1/validator-keystore

# Import an existing private key
gstable account import private.key --datadir /data/validator1

The derived public address is used as the node’s etherbase (coinbase) and must exactly match the address registered in GovValidator.validators.

BLS Key Derivation

BLS keys are deterministically derived from the validator’s ECDSA private key.
This guarantees that the same ECDSA key always produces the same BLS key pair.

// Derive a BLS key from an ECDSA private key
blsKey, err := bls.DeriveFromECDSA(ecdsaPrivateKey)
if err != nil {
    log.Fatal("Failed to derive BLS key", "err", err)
}
blsPublicKey := blsKey.PublicKey()

The derived BLS public key must be registered in GovValidator.blsPublicKeys at the same index as the corresponding validator address.

Validator Registration

Validators are registered in the GovValidator contract through governance proposals.
Only registered validators can participate in consensus and block proposal.

Starting a Validator Node

The following example starts a validator node with mining enabled.

gstable \
    --datadir /data/validator1 \
    --networkid 8282 \
    --mine \
    --miner.etherbase 0xYOUR_VALIDATOR_ADDRESS

Key validator flags:

Flag	Purpose
--mine	Enable block production
--miner.etherbase	Validator address (coinbase)

Validator Mining Process

When a validator node starts mining, the following initialization sequence occurs:

1. Load validator state

   • Load the registered validator set from genesis and the latest chain state via GovValidator.
   • Verify that the node’s etherbase is part of the active validator set.

2. Validate validator keys

   • Ensure the address derived from the signing key matches the etherbase address.
   • Verify that the corresponding BLS public key is correctly registered in GovValidator.blsPublicKeys.

3. Initialize the WBFT consensus engine

   • Load Anzeon WBFT parameters (epoch length, block period, timeout, etc.) from genesis.
   • Apply the WBFT configuration for the current block height.

4. Determine validator role

   • Decide whether the node is the proposer for the current round based on the proposer policy.
   • If selected as proposer, prepare a block proposal.

5. Start consensus message handling

   • Activate network handlers for consensus messages.
   • Initialize signing and verification logic for each consensus phase, including BLS signature aggregation.

6. Block production and consensus participation

   • If proposer, propose a block including WBFTExtra.
   • If not proposer, validate the proposed block and broadcast signed messages according to the consensus phase.

Gas Tip Policy

In StableNet, the gas tip (priority fee) for non-authorized accounts is governed via on-chain governance and stored in the GovValidator contract.
Gas tip application differs by account type.

Authorized Accounts

Authorized accounts follow the standard Ethereum model and directly specify maxPriorityFeePerGas per transaction.

• Per-transaction gas tip configuration
• The specified maxPriorityFeePerGas is applied as-is
• Competitive prioritization during congestion
• Not subject to the fixed gas tip policy defined by governance

Non-Authorized Accounts

Non-authorized accounts may also specify maxPriorityFeePerGas, but the value is not used to determine the actual gas tip.

• Regardless of the transaction’s maxPriorityFeePerGas, the fixed gas tip defined by governance is applied
• The transaction-specified gas tip is used only for transaction replacement logic
• Transactions below the fixed gas tip are filtered out by the transaction pool

Gas Tip Updates

• Validators may propose updates to the fixed gas tip via governance proposals
• Proposals require quorum approval and are applied immediately upon execution

Epoch Transitions

StableNet operates on epoch-based cycles, and validator set changes are applied only at epoch blocks.

Key points:

• Epoch length is configured via anzeon.wbft.epochLength (default: 10 blocks)
• Validator set changes approved during epoch N become active at the start of epoch N+1
• The first epoch uses the validator set defined in anzeon.init.validators
• Subsequent epochs load validator sets from GovValidator.validators

Governance Participation

Validators participate in governance using their operator keys.
Governance manages network configuration and validator sets through proposals and voting.

Proposal Types

• ADD_VALIDATOR: Add a new validator to the set
• REMOVE_VALIDATOR: Remove an existing validator
• SET_GAS_TIP: Update the mandatory gas tip value
• CONFIGURE_SYSTEM: Update system parameters and configuration

Voting Process

• Any governance member can submit a proposal via a contract call
• Other members vote using vote(proposalId, approve)
• Proposals are executed automatically upon reaching quorum
• Proposals expire after the configured expiry period (default: 7 days)

Node Monitoring and Maintenance

Logging Configuration

StableNet nodes use a structured logging system with configurable verbosity levels.
Operators can configure global log levels, module-specific levels, and output formats.

# Set global log level
gstable --verbosity 3  # 0=silent, 1=error, 2=warn, 3=info, 4=debug, 5=trace

# Module-specific log levels
gstable --vmodule "consensus/*=5,miner=4,eth/*=3"

# File logging and rotation
gstable --log.file /var/log/gstable/node.log --log.rotate

# JSON-formatted logs for log aggregation
gstable --log.json

Key log modules to monitor:

Module	Purpose	Important Events
consensus/wbft	WBFT consensus	Round changes, validator proposals, commit messages
miner	Block production	Block sealing, transaction inclusion, gas tip updates
eth/handler	Chain sync	Peer connections, block propagation, sync progress
core/blockchain	Chain processing	Block insertion, reorgs, state root mismatches
eth/downloader	Sync progress	Sync mode, peer selection, download rates

Metrics and Monitoring

StableNet supports metrics collection and integration with external monitoring systems to ensure operational visibility.

# Enable metrics server
gstable --metrics --metrics.addr 0.0.0.0 --metrics.port 6060

# Enable InfluxDB metrics export
gstable \
    --metrics.influxdb \
    --metrics.influxdb.endpoint http://influxdb:8086 \
    --metrics.influxdb.database gstable \
    --metrics.influxdb.username monitor \
    --metrics.influxdb.password secret

Disk Space Management

StableNet nodes consume significant disk space to store blockchain data.
Continuous monitoring and protective shutdown mechanisms are recommended for stable operation.

Minimum Free Disk Space

To prevent data corruption due to low disk space, a minimum free disk threshold can be configured.
If available disk space falls below this threshold, the node shuts down automatically.

# Set minimum free disk space (in MB)
gstable --datadir.minfreedisk 50000  # Shutdown if free space drops below 50 GB

The node periodically checks available disk space and performs a graceful shutdown when the threshold is reached.

Database Size Monitoring

Database usage varies by sync mode and pruning configuration.
The table below shows typical database sizes and expected monthly growth.

Configuration	Approximate Size	Growth Rate
Full node (archive)	500+ GB	~30 GB/month
Full node (pruned)	150–200 GB	~10 GB/month
Snap sync	100–150 GB	~8 GB/month

The following directories should be continuously monitored:

• chaindata/: Main blockchain state and block database
• ancient/: Freezer directory for old blocks (if configured separately)
• blobpool/: Blob transaction pool data
• ethash/: DAG storage (if Ethash consensus is used)

Pruning and Garbage Collection

When using the hash-based state scheme, pruning and garbage collection can be configured.

# Archive mode (no pruning)
gstable --gcmode archive --cache.gc 0

# Full mode (pruning enabled)
gstable --gcmode full --cache.gc 25  # Use 25% of cache for GC

Note: Path-based state databases use automatic pruning and do not support manual GC configuration.

Database Maintenance

State Scheme Selection

StableNet supports two state storage schemes. Operators should select the appropriate scheme based on operational requirements.

# Hash-based scheme (traditional)
gstable --state.scheme hash

# Path-based scheme (recommended for new nodes)
gstable --state.scheme path

Advantages of the path-based scheme:

• Automatic state pruning
• Lower disk usage
• Improved synchronization performance

Advantages of the hash-based scheme:

• Mature and well-tested
• Manual control over pruning and garbage collection
• Suitable for environments requiring offline pruning tools

Offline Pruning Recovery

Nodes using the hash-based scheme automatically attempt recovery at startup if an interrupted pruning operation is detected.

// Automatic recovery at startup
if scheme == rawdb.HashScheme {
    if err := pruner.RecoverPruning(stack.ResolvePath(""), chainDb); err != nil {
        log.Error("Failed to recover state", "error", err)
    }
}

This recovery process runs automatically during startup and restores database consistency after abnormal pruning interruptions.

Chain Recovery and SetHead

If blockchain state is corrupted or needs to be rolled back to a specific point, the debug_setHead RPC method can be used.

# Roll back to a specific block number
gstable attach /data/gstable.ipc --exec 'debug.setHead("0x10000")'  # Move to block 65536

# Roll back by number of blocks
gstable attach /data/gstable.ipc --exec 'debug.setHead("0x-100")'  # Move back 256 blocks

Use cases:

• Recovery from corrupted block data
• Rolling back after detecting chain anomalies
• Rebuilding the chain on the correct fork after a network split

Warning: setHead deletes all blocks and state data after the specified point.
Always ensure a backup exists and fully understand the impact before use.

Node Upgrade Procedure

Node upgrades should be performed sequentially using the following steps:

1. Stop the node

   • Gracefully shut down the running gstable process.

2. Replace binaries

   • Back up the existing binary and replace it with the new version.
   • Verify execution permissions and binary paths.

3. Review configuration files

   • Check for new or modified configuration options in the new version.
   • Ensure compatibility with existing configuration files.

4. Restart the node

   • Restart the node with the same options.
   • For validator nodes, ensure the --mine flag is included.

5. Inspect initial logs

   • Review startup logs for database migrations, consensus initialization, and network connectivity issues.

Pre-upgrade checklist:

• Review release notes for breaking changes
• Back up the database (chaindata/ directory)
• Test the upgrade on non-critical nodes first
• For validator nodes, announce planned downtime to peers in advance

Post-upgrade verification:

# Check node version
gstable version

# Check database version
gstable --datadir /data version-check

# Check synchronization status
gstable attach --exec 'eth.syncing'

# Check peer connectivity
gstable attach --exec 'admin.peers.length'

# For validator nodes, check mining status
gstable attach --exec 'eth.mining'

Abnormal Shutdown Detection

StableNet nodes detect abnormal shutdowns indirectly via database state and automatically run recovery mechanisms on startup.

Abnormal Shutdown Handling

• On startup, LevelDB WAL (Write-Ahead Log) is used to recover incomplete writes
• Consistency between state database, snapshots, and ancient data is verified
• If consistency cannot be restored automatically, resynchronization or state reconstruction may be triggered

Manual recovery after abnormal shutdowns:

# Verify database integrity
gstable --datadir /data verify-db

# Rebuild indexes if necessary
gstable --datadir /data rebuild-indexes

Bootnode Operations

Dedicated bootnodes can be deployed to support peer discovery in StableNet networks.

# Generate bootnode key
bootnode --genkey=boot.key

# Run the bootnode
bootnode \
    --nodekey=boot.key \
    --addr=:30303 \
    --netrestrict=0.0.0.0/0 \
    --verbosity=3

# Print enode URL
bootnode --nodekey=boot.key --writeaddress

Bootnodes provide lightweight peer discovery services without maintaining full blockchain state.
Clients can reference the bootnode as follows:

gstable --bootnodes enode://[nodekey-pubkey]@[ip]:30303

Network Monitoring Dashboard

For stable validator operations, it is recommended to configure dashboards with the following key metrics.

Validator Key Performance Indicators (KPIs)

Metric	Target	Alert Threshold
Block production rate	~1 block/second	< 0.9 blocks/sec
Missed blocks	0%	> 1%
Peer count	10–50	< 5 peers
Sync delay	< 2 blocks	> 10 blocks behind
Disk free space	> 100 GB	< 50 GB
Memory usage	< 80%	> 90%
CPU usage	< 50%	> 80% sustained