> ## Documentation Index
> Fetch the complete documentation index at: https://docs.stablenet.network/llms.txt
> Use this file to discover all available pages before exploring further.

# Validator Operations

## Purpose and Scope

This document provides a **guide for operating validator nodes** in the StableNet network.\
It covers validator key management, node configuration, block production (mining), governance participation, monitoring practices, and common failure-handling scenarios.

Validators are the core operators of the StableNet blockchain, responsible for block production, consensus participation, and network governance.

Related documents:

* [Network Deployment](/en/operations-guide/11.1-network-deployment)
* [Node Monitoring and Maintenance](/en/operations-guide/11.3-node-monitoring-and-maintenance)
* [Anzeon WBFT Consensus Protocol](/en/consensus-and-block-production/4.1-anzeon-wbft-consensus-protocol)
* [Validator Governance (GovValidator)](/en/governance-system/5.3-validator-governance-\(govvalidator\))

## Validator Architecture Overview

### Validator Key Structure

StableNet validators use **three types of keys**, each serving a distinct purpose.

| Key Type          | Purpose                                           | Storage                     | Usage Frequency               |
| ----------------- | ------------------------------------------------- | --------------------------- | ----------------------------- |
| **Operator Key**  | Governance voting, validator add/remove           | External wallet or multisig | Low (governance actions only) |
| **Validator Key** | Consensus message signing, coinbase address       | Node `nodekey` file         | High (every block)            |
| **BLS Key**       | Derived from Validator Key, aggregated signatures | Derived internally          | High (every block)            |

* The **Operator Key** can be a standard EOA (Externally Owned Account) or a multisig contract address for enhanced security.
* The **Validator Key** and **BLS Key** must always be accessible to the node process for consensus participation.
* StableNet nodes automatically derive the validator address from the `nodekey` and use it consistently across the Anzeon chain.

### Genesis Configuration for Validators

Validators are defined in **two locations** within the genesis file.

1. **`anzeon.init` section**
   * Used to construct `EpochInfo` in the genesis block (block 0)
   * Defines the active validator set for the **first epoch** (from block 0 up to the first epoch block)

2. **`anzeon.systemContracts.govValidator` section**
   * Defines the **initial state** of the GovValidator contract (storage initialized at genesis)
   * From the **second epoch onward**, validator BLS keys are read from this contract when creating new `EpochInfo` at epoch blocks
   * After genesis, validator additions and removals are only possible via governance (GovValidator proposals, voting, and execution)

The `govValidator.params` object contains three parallel lists:

* `members`
* `validators`
* `blsPublicKeys`

These lists must have **identical lengths and ordering**.\
Entries at the same index across all lists together represent a single validator identity.

* **Operator address** (`members`): Used for governance voting and validator management
* **Validator address** (`validators`): Used for consensus message signing and as the coinbase address; derived from the node’s `nodekey`
* **BLS public key** (`blsPublicKeys`): Derived from the validator key; used for WBFT PREPARE/COMMIT aggregated signatures

## Node Configuration

### Initial Setup

#### 1. Validator Key Generation

The validator address and BLS key are derived from the node’s **nodekey**.\
The nodekey is generated using `bootnode -genkey` as described in [Network Deployment](/en/operations-guide/11.1-network-deployment).

The BLS public key can be prepared in one of the following ways:

* Automatically derived from the nodekey by `genesis_generator`
* Generated via bootnode tooling and registered in genesis and GovValidator

#### 2. Configuration File Setup

Each validator node uses a `config.toml` file to configure data directories, RPC, P2P, static nodes, and bootnode settings.\
For multi-node deployments, you may directly use the `config.toml` generated by `genesis_generator` or customize it for the operational environment.

#### 3. Network Configuration

Mainnet and testnet validators can use network presets (`-mainnet`, `-testnet`) or specify bootnodes using `-bootnodes`.\
For private networks, follow the static node and bootnode configuration described in\
[11.1 Network Deployment](/en/operations-guide/11.1-network-deployment).

### Important Configuration Parameters

#### Mining Configuration

In Anzeon/WBFT chains, the etherbase is **automatically** set to the address derived from the node’s nodekey, regardless of `-miner.etherbase`.\
This guarantees that the validator key and block coinbase always match.\
The code (e.g., `eth/backend.go`) consistently derives the validator address from the P2P node key.

## Starting Validator Operations

### Node Startup

Run the node using a data directory initialized with the genesis file.\
To enable mining (block production), specify the `-mine` flag.

Example:

`gstable -datadir <path> -networkid <chainId> -mine -config config.toml`

If required, add the `-bootnodes` option to specify bootnodes.

### Mining Lifecycle

Mining begins once the node is synchronized with the network.\
Under WBFT consensus, **only the proposer selected for the current round** proposes a block, while other validators participate via prepare and commit messages.

### WBFT-Specific Mining Loop

In WBFT, mining is **event-driven rather than timer-based**.\
When the consensus engine starts a new round, it signals the worker to generate and submit a block.

**Execution flow summary:**

* **Trigger**\
  When WBFT core starts a new round (`startNewRound`), `NotifyNewRound` is invoked in the backend and forwarded to the worker’s `readyToCommit` channel.\
  An optional delay until the next block time may also be included.

* **Worker**\
  Upon receiving a signal on `readyToCommitCh`, the worker creates new sealing work.\
  Any in-progress block build is aborted and retried using the new round context.\
  When WBFT is enabled, block creation is triggered *only* by this channel signal; simple chain head updates do not trigger new work.

* **Proposer**\
  A proposer is selected for each round.\
  Although all validators may build blocks, only the proposer’s PRE-PREPARE block is used for consensus.

This integration is achieved by registering a `readyToCommit` callback with the WBFT engine during backend startup, linking the consensus engine and execution backend (worker).

## Governance Participation

### Proposal Voting

Governance members participate via the GovValidator contract.\
Using the **Operator Key**, they can create proposals (`propose`), vote (`vote`), and execute (`execute`).\
Refer to [Validator Governance (GovValidator)](/en/governance-system/5.3-validator-governance-\(govvalidator\)) for details.

### Gas Tip Governance

Governance members determine the gas tip (priority fee) for non-authorized accounts via GovValidator governance.\
The selected gas tip is queried from GovValidator on every block and automatically applied to both the worker and transaction pool.

## Key Management Best Practices

### Security Considerations

| Key Type          | Security Level | Backup Strategy                      | Access Pattern |
| ----------------- | -------------- | ------------------------------------ | -------------- |
| **Operator Key**  | Highest        | Offline backup, multisig recommended | Rare           |
| **Validator Key** | High           | Encrypted backup                     | Continuous     |
| **BLS Key**       | High           | Encrypted backup                     | Continuous     |

### Protecting the Operator Key

Because the operator key controls governance actions, it carries the highest privilege.

Recommended practices:

1. **Use multisig contracts**\
   Use a multisig contract address instead of a plain EOA
2. **Cold storage**\
   Keep the operator key offline except during voting
3. **Hardware wallets**\
   Use hardware wallets for operator key signing
4. **Isolated signing services**\
   Operate a dedicated signing service for operator actions

### BLS Key Management

BLS keys are mandatory for WBFT participation. The following must be strictly observed.

1. **Registration consistency**\
   The BLS public key registered in GovValidator (or genesis `init`) must match the BLS public key derived from the node’s nodekey.\
   The consensus engine verifies signatures using the registered key; mismatches prevent participation.

2. **Derivation during initialization**\
   During initialization, the WBFT backend derives the BLS secret key by calling `bls.DeriveFromECDSA` on the node’s ECDSA private key (nodekey).\
   There is no separate BLS key file loading path.

3. **Secure backups**\
   Losing the nodekey makes it impossible to recover the same BLS key pair.\
   In such cases, governance must be used to register a new validator with a new nodekey and BLS key.

To obtain the BLS public key for genesis or GovValidator registration:

* Derivation logic: `genesis_generator.deriveAccount()`
* Reference: *Data derived from a Nodekey* in\
  [Network Deployment](/en/operations-guide/11.1-network-deployment)

In production, either generate validator and BLS keys via `genesis_generator` or use a BLS12-381–capable utility, ensuring the same key pair is used by the node.

### Validator Rotation

Active validators and BLS keys can be changed **only via governance**, allowing zero-downtime rotation.

1. **Preparation**\
   Generate a new nodekey and BLS key pair and obtain the validator address and BLS public key.

2. **Propose addition**\
   Submit a proposal to add the new validator using the operator key.

3. **Propose removal**\
   After approval, submit a proposal to remove the old validator.

4. **Zero-downtime transition**\
   Run both validator nodes concurrently during the transition period.

Governance process:

* Call `GovValidator.propose()` with the operator key
* Members vote via `GovValidator.vote()`
* Execute via `GovValidator.execute()` once quorum is reached
* Changes take effect at the next epoch block

### Key Storage

Restrict file permissions so sensitive key files (e.g., `nodekey`) are readable **only by the validator node**.\
Example: `chmod 600 nodekey`\
Ensure the directory containing key files also has restricted permissions.

## Validator Status Monitoring

### Key Metrics to Monitor

Regularly monitor block production rate, peer count, sync lag, disk and memory usage, and BLS/consensus-related errors.\
See [Node Monitoring and Maintenance](/en/operations-guide/11.3-node-monitoring-and-maintenance) for details.

### Monitoring Commands

After attaching to the node console using `gstable attach`, use:

* `eth.mining` : Mining (block production) status
* `eth.blockNumber` : Current block height
* `admin.peers.length` : Number of connected peers
* `admin.peers` : Detailed peer information

Validator lists can be queried via GovValidator contract calls.

### Log Monitoring

Inspect logs for:

* Block proposal and commit flow
* BLS signature failures and consensus errors
* Peer connections and disconnections
* Gas tip updates
* Governance-related events

Investigate and address warnings or errors promptly.

### Log Levels

The default `-verbosity` is often sufficient.\
For deeper inspection of consensus, P2P, or worker behavior, increase `-verbosity` or use `-vmodule` to enable detailed logs for specific packages.

### Metrics Export

Enable Prometheus-compatible metrics using the `-metrics` flag.\
Default endpoint:

* `http://localhost:6060/debug/metrics`

The metrics port and path can be customized via flags.

## Common Troubleshooting

### Issue: Mining Does Not Start

**Symptoms**: `eth.mining` returns `false` and no blocks are produced.

**Diagnostics**:

1. Attach to the node and check `eth.mining` and `eth.coinbase`
2. Verify the node was started with the `-mine` flag
3. Check `eth.syncing` (mining may be delayed while syncing)

**Resolution**:

* Restart the node with `-mine`
* Ensure the nodekey file exists in `-datadir` (or default data directory)
* Recheck mining status after synchronization completes

### Issue: Block Production Stops

**Symptoms**\
The validator no longer produces blocks.

**Diagnostics**:

1. Query GovValidator to confirm the node’s address is still in the validator set
2. Inspect logs for WBFT proposal/commit messages or errors
3. Check peer connectivity via `admin.peers.length` and `admin.peers`

**Resolution**:

* Verify validator address and BLS public key registration
* Ensure the node’s BLS key matches the registered public key
* Increase peer connectivity (20–30 peers recommended for stability)
* Check firewall and network access for the P2P port (default 30303)

### Issue: Gas Tip Mismatch

**Symptoms**\
Transactions fail with `"gas tip too low"` errors.

**Diagnostics**:

1. Check the current required gas tip via `GovValidator.gasTip()` or `eth.maxPriorityFeePerGas`
2. Verify the transaction’s `maxPriorityFeePerGas` meets or exceeds the required value
3. Confirm the node is fully synchronized and recent governance changes are applied

**Resolution**:

* Ensure the node is synced (gas tip is fetched automatically by the worker)
* Allow time for block propagation after governance changes
* Submit transactions with `maxPriorityFeePerGas` ≥ required network gas tip

### Issue: Node Crash or Restart

**Symptoms**\
The node process exits unexpectedly.

**Diagnostics**:

1. Check available disk space on `-datadir`
2. Inspect logs for database errors after restart
3. Review system and kernel logs for OOM kills or I/O errors

**Resolution**:

* Free disk space (≥500GB recommended for production)\
  Optionally configure `-datadir.minfreedisk`
* Increase RAM or adjust `-cache` if memory pressure occurs
* If DB corruption is suspected, back up data, clear `-datadir`, and resync
* Use SIGINT or SIGTERM for graceful shutdown and verify shutdown logs

### Issue: Validator Removed from Set

**Symptoms**\
The node stops producing blocks and is no longer in the validator set.

**Diagnostics**:

1. Query `GovValidator.getValidators()` to confirm removal
2. Check recent governance proposals, votes, and executions
3. Investigate potential operator key compromise or misuse

**Resolution**:

* If key compromise is suspected, immediately prepare key rotation
* Coordinate with governance members to propose re-adding the validator
* Review proposal history and logs to determine the cause
* Protect operator keys using multisig and cold storage

## Performance Optimization

### Resource Requirements

| Resource | Minimum    | Recommended   | Notes                                     |
| -------- | ---------- | ------------- | ----------------------------------------- |
| CPU      | 4 cores    | 8+ cores      | More cores improve parallel tx processing |
| RAM      | 8 GB       | 16+ GB        | More RAM reduces disk I/O                 |
| Disk     | 500 GB SSD | 1 TB NVMe SSD | Disk speed affects state access           |
| Network  | 10 Mbps    | 100+ Mbps     | Low-latency, stable network recommended   |

### Cache Configuration

Use the `-cache` option to allocate memory (MB) to the node.\
Validators are recommended to use **at least 4096MB**, considering full sync and block production load.\
Increasing the value reduces disk I/O if memory is available.

### Network Optimization

Set an appropriate `-maxpeers` value and maintain connections to trusted peers using `-bootnodes` and static nodes.\
Ensure the P2P port (default 30303) is open and the network environment has low latency and packet loss.

## Epoch Transitions and Validator Set Updates

### Epoch Structure

WBFT manages blocks in epochs, and validator set changes apply **only at epoch blocks**.

The epoch length is defined by `config.anzeon.epochLength` in genesis\
(e.g., `epochLength = 10`).

At the last block of each epoch (`blockNumber % epochLength == 0`), the WBFT engine:

1. Reads the current validator list from GovValidator
2. Reads the BLS public key list from GovValidator
3. Updates `WBFTExtra.EpochInfo` in the next block header
4. Activates the updated validator set starting from the next block

### Validator Set Synchronization

New or restarting nodes read the validator set from the latest block’s `WBFTExtra.EpochInfo` during chain synchronization.

Because GovValidator updates are applied at epoch blocks, once synchronization completes, the node’s active validator set automatically matches the network’s current state.

## Validator Rewards and Economics

StableNet validators are rewarded through the following structure.

### Validator Reward Model

**Transaction Fees**

* Both Base Fee and Gas Tip are paid to validators
* **Base Fee is not burned**\
  (Unlike Ethereum, no burn mechanism is used to preserve 1:1 fiat backing)
* **Gas Tip** is paid directly to the block producer\
  (credited via `Header.Coinbase`)
* **No block rewards**\
  There is no inflationary block subsidy by design

### Gas Fee Policy

StableNet uses a gas fee policy optimized for a stablecoin environment.

#### Gas Tip Policy

Gas Tip application depends on account type.

1. Authorized Accounts

* Same as Ethereum: **user-specified Gas Tip per transaction**
* `maxPriorityFeePerGas` is applied as provided

2. Non-Authorized Accounts

* **Fixed Gas Tip value set by governance**
* Gas Tip is adjustable via GovValidator governance
* Even if a higher `maxPriorityFeePerGas` is specified, the **governance-defined fixed Gas Tip is enforced**

#### Base Fee Policy

Unlike Ethereum EIP-1559, StableNet uses a **dual-threshold fixed-rate adjustment model**.

**Base Fee Parameters**

| Parameter           | Value           | Description                 |
| ------------------- | --------------- | --------------------------- |
| IncreasingThreshold | 20              | Increase if gas usage > 20% |
| DecreasingThreshold | 6               | Decrease if gas usage \< 6% |
| BaseFeeChangeRate   | 2               | Adjustment rate (2%)        |
| MinBaseFee          | 20,000 gwei     | Minimum Base Fee            |
| MaxBaseFee          | 20,000,000 gwei | Maximum Base Fee            |

**Base Fee Adjustment Rules**

Base Fee is adjusted based on the **parent block’s gas usage**.

1. **Gas usage > 20%**
   * Increase Base Fee by **2%**
   * Roughly corresponds to >1,000 simple transfers per block

2. **6% ≤ Gas usage ≤ 20%**
   * **No change** to Base Fee

3. **Gas usage \< 6%**
   * Decrease Base Fee by **2%**
   * Roughly corresponds to \< 300 simple transfers per block

Base Fee is always clamped within `MinBaseFee` and `MaxBaseFee`.

## Validator Summary Checklist

**Setup**

* Generate and securely store operator, validator, and BLS keys
* Configure `config.toml` for the operational environment
* Register the validator via governance
* Set up monitoring and alerting

**Operations**

* Run validator nodes with `-mine`
* Verify mining and block production
* Monitor peer count and sync status
* Track gas tip governance changes
* Participate in governance proposals and voting

**Maintenance**

* Regularly back up configuration files and keys
* Monitor disk space and system resources
* Review logs for errors and warnings
* Test upgrades on testnet before production rollout
* Coordinate changes with other validators and governance members

**Security**

* Protect operator keys with multisig or cold storage
* Encrypt validator and BLS keys
* Enforce correct file permissions (`chmod 600`)
* Restrict unnecessary network access via firewall rules
* Conduct regular security reviews of access control and key management