# XDPoS V2 Compatibility & Checkpoint Sync - Complete PR

## Overview

This PR implements comprehensive XDPoS V2 compatibility fixes and checkpoint sync infrastructure for the GP5 (Geth 1.17 fork) to enable syncing from block 56M on Apothem testnet without hitting "validators not legit" errors at the V2 switch block.

**Branch**: `feat/trusted-checkpoint-sync`  
**Base**: `xdc-network`  
**Commits**: 56 commits  
**Docker Images**: v130 → v133

---

## 🎯 Goals Achieved

1. ✅ **Checkpoint Sync Infrastructure**: Nodes can sync from trusted checkpoints (50M, 55M, 56M, 56.7M, 56.8287M)
2. ✅ **V2 Genesis Config Detection**: Fixed `hasV2=false` issue by patching missing V2 config from genesis
3. ✅ **V2 Engine Wiring**: Confirmed `SetEngineV2()` properly initializes V2 consensus engine
4. ✅ **V2 Snapshot Deferred Init**: Allows checkpoint sync to proceed when gap header is missing
5. 🟡 **V2 Snapshot Without State**: Partial fix - identified validators are stored in `Extra` field, not `Validators` array
6. 🟡 **Stats Page Visibility**: `--ethstats` configured but nodes not yet visible

---

## 🔧 Major Fixes Implemented

### 1. TrustedSyncCheckpoints Infrastructure (P1-P7)

**Files**: `params/config.go`, `core/blockchain.go`, `core/headerchain.go`, `eth/backend.go`

Added trusted checkpoint system allowing nodes to sync from known-good block hashes without downloading from genesis:

```go
// TrustedSyncCheckpoints in params/config.go
{Number: 50000000, Hash: ..., Root: ...},
{Number: 55000000, Hash: ..., Root: ...},
{Number: 56000000, Hash: ..., Root: ...},
{Number: 56700000, Hash: 0xb2f51bf5..., Root: 0x702f5e59...},  // NEW
{Number: 56828700, Hash: 0x5ac967a2..., Root: 0x425aa614...},  // V2 switch
```

### 2. V2 Genesis Config Patch (Issue #379)

**File**: `core/genesis.go`

Fixed critical bug where Apothem genesis JSON loaded without V2 configuration:

```go
// In chainConfigOrDefault() and SetupGenesisBlockWithOverride()
if genesis.Config.XDPoS != nil && genesis.Config.XDPoS.V2 == nil && 
   genesis.Config.ChainID != nil && genesis.Config.ChainID.Uint64() == 51 {
    genesis.Config.XDPoS.V2 = params.XDCApothemChainConfig.XDPoS.V2
    log.Warn("XDC: patched missing V2 config in genesis for apothem", ...)
}
```

**Impact**: Without this, `hasV2=false` and V2 blocks fall back to V1 validation, causing "validators not legit".

### 3. Checkpoint Sync Without State (Issues #467, #468, #471)

**Files**: `core/blockchain.go`, `core/state_processor.go`, `eth/downloader/downloader.go`

- P2: Skip checkpoint block in body download to fix "unknown ancestor"
- P3: Align resultCache offset with first scheduled body
- P4: Patch TrustedSyncCheckpoints in `SetupGenesisBlockWithOverride`
- P5: Use empty state when parent trie missing during checkpoint sync
- P5b: Auto-detect checkpoint sync without state when parent state missing
- P5c: Fix checkpoint sync for checkpoint blocks without state

### 4. V2 Deferred Snapshot Initialization

**File**: `consensus/XDPoS/engines/engine_v2/engine.go`

When syncing from checkpoint after V2 switch, the gap header (SwitchBlock - Gap) may not exist in DB:

```go
// In initial() function
gapHeader := chain.GetHeaderByNumber(lastGapNum)
if gapHeader == nil {
    log.Warn("[initial] V2 gap header not found, deferring snapshot init", ...)
    x.v2StartBlockWithoutSnapshot = true
    return nil, nil
}
```

### 5. repairSnapshot with Checkpoint Header Fallback (Issue #459)

**File**: `consensus/XDPoS/engines/engine_v2/engine.go`

When state is unavailable during checkpoint sync, attempt to read validators from checkpoint header:

```go
// CHECKPOINT SYNC FIX: State not available, read from header
checkpointHeader := chain.GetHeaderByNumber(checkpointNumber)
if checkpointHeader != nil && len(checkpointHeader.Validators) > 0 {
    // Extract masternodes from header.Validators
    // Create and store snapshot
}
```

**Status**: Partially working - discovered V2 stores validators in `Extra` field, not `Validators` array.

---

## 🔬 Critical Discovery from Debug Logs (v133)

Both nodes produced this output at V2 checkpoint 56,829,600:

```
INFO [repairSnapshot] V2 checkpoint: checking checkpoint header
    checkpoint=56,829,600
    gap=56,829,150
    validatorsLen=0      ← EMPTY! V1 field not used in V2
    validatorLen=65      ← Proposer signature (65 bytes)
    penaltiesLen=0
    extraLen=529         ← V2 encoded data (validators + QC + round)
```

**Conclusion**: V2 checkpoint headers store the validator list in the `Extra` field (529 bytes), NOT in the `Validators` array (0 bytes). The next developer must parse the `Extra` field to extract masternodes.

---

## 📊 Node Status

### Node 168 (95.217.56.168)
- **Container**: `xdc-gp5-v133-cp56700-apothem-168`
- **Status**: 🔴 Stuck at V2 checkpoint 56,830,293
- **Error**: "validators not legit" - V2 snapshot missing
- **Bootnode**: xdc03-v268 (167.235.13.113:30320)

### Node 113 (167.235.13.113)
- **Container**: `xdc03-gp5-v133-cp56700-apothem-113`
- **Status**: 🔴 Stuck at V2 checkpoint 56,830,293
- **Error**: Same as node 168
- **Bootnode**: xdc03-v268 (self)

### Both Nodes
- Sync from checkpoint 56M: ✅ Working
- V2 config detection: ✅ Working (`hasV2=true`)
- V2 engine wiring: ✅ Working
- V2 snapshot creation: 🔴 Blocked (validators in Extra field)

---

## 🐛 Known Issues / Remaining Work

### P0: V2 Snapshot Without State
- **Problem**: `repairSnapshot` cannot create V2 snapshot during checkpoint sync
- **Root Cause**: V2 validators stored in `Extra` field, not `Validators` array
- **Next Step**: Parse `Extra` field (529 bytes) to extract validator addresses
- **File**: `consensus/XDPoS/engines/engine_v2/engine.go:repairSnapshot()`

### P1: Ethstats Visibility
- **Problem**: Nodes not appearing on https://stats.xdcindia.com/
- **Config**: `--ethstats gp5-168-cp56700@xdc_openscan_stats_2026@stats.xdcindia.com:443`
- **Note**: May need sync to complete first or check stats server requirements

### P2: Fleet Deployment
- **Status**: Only nodes 168 and 113 deployed
- **Action**: Deploy working image to all fleet nodes once V2 fix is complete

---

## 🏗️ Build Instructions

```bash
# On 168 server (Ubuntu 24.04)
cd /root/go-ethereum-v111
git fetch origin feat/trusted-checkpoint-sync --force
git reset --hard origin/feat/trusted-checkpoint-sync

# Build binary
make geth

# Build Docker image (Ubuntu 24.04 base for glibc compatibility)
docker build --no-cache -t anilchinchawale/gp5-xdc:vXXX -f Dockerfile.gp5-ubuntu24 .

# Push to Docker Hub
docker push anilchinchawale/gp5-xdc:vXXX

# Deploy (example for node 168)
docker stop xdc-gp5-v133-cp56700-apothem-168
docker rm xdc-gp5-v133-cp56700-apothem-168
rm -rf /mnt/data/xdc-nodes/gp5-cp56700/geth/chaindata/*
docker run -d \
  --name xdc-gp5-vXXX-cp56700-apothem-168 \
  -v /mnt/data/xdc-nodes/gp5-cp56700:/root/.XDC \
  -p 30303:30303 -p 8555:8555 -p 8546:8546 -p 8547:8547 \
  anilchinchawale/gp5-xdc:vXXX \
  --syncfromblock 56700000 \
  --datadir /root/.XDC \
  --networkid 51 \
  --apothem \
  --rpcvhosts '*' \
  --rpcaddr 0.0.0.0 \
  --http \
  --http.addr 0.0.0.0 \
  --http.api db,eth,debug,net,shh,txpool,admin,XDPoS \
  --http.corsdomain '*' \
  --http.vhosts '*' \
  --ws \
  --ws.addr 0.0.0.0 \
  --ws.api db,eth,debug,net,shh,txpool,admin,XDPoS \
  --ws.origins '*' \
  --gcmode archive \
  --bootnodes enode://1d8b21ac... @167.235.13.113:30320 \
  --ethstats 'gp5-168-cp56700:xdc_openscan_stats_2026@stats.xdcindia.com:443'
```

---

## 📁 Files Changed (56 commits)

### Core Changes
- `params/config.go` - TrustedSyncCheckpoints + 56.7M checkpoint
- `core/genesis.go` - V2 config patching for Apothem
- `core/blockchain.go` - Checkpoint sync integration
- `core/headerchain.go` - Checkpoint header validation
- `core/state_processor.go` - Checkpoint sync without state

### Consensus Engine
- `consensus/XDPoS/engines/engine_v2/engine.go` - repairSnapshot + initial() fixes + debug logging
- `consensus/XDPoS/engines/engine_v2/utils.go` - Debug logging for VerifyHeaders

### Downloader/Sync
- `eth/downloader/downloader.go` - Checkpoint body download fixes
- `eth/downloader/queue.go` - Result cache alignment

### Docker/Build
- `Dockerfile.gp5-ubuntu24` - Ubuntu 24.04 base + correct ENTRYPOINT
- `Dockerfile` - Fixed ENTRYPOINT JSON syntax

### Documentation
- `V2_COMPATIBILITY_DEVELOPMENT_LOG.md` - Complete development history
- `V2_COMPATIBILITY_HANDOFF_ISSUE.md` - Handoff for next developer

---

## 🔗 Related Issues

- #459 - "validators not legit" at V2 checkpoint
- #379 - V2 genesis config missing
- #467 - Checkpoint sync restart failure
- #468 - Foundation wallet issue
- #471 - Checkpoint sync requirements documentation
- #293 - V2 switch block issues
- #500 - Handoff issue (this work)

---

## 👥 Credits

- **Development**: Hermes Agent (Kimi k2.6) with Opus 4.7 validation
- **Validation**: Claude Opus 4.7 for deep code analysis
- **Testing**: Nodes 168 (95.217.56.168) and 113 (167.235.13.113)
- **Docker Hub**: anilchinchawale/gp5-xdc

---

*This PR represents ~50+ iterations of build-deploy-test-fix cycles across multiple days. The checkpoint sync infrastructure is solid; the remaining work is parsing V2's Extra field for validator extraction during stateless sync.*