# XDPoS V2 Compatibility - Complete Development Log & Handoff

## Executive Summary

This issue documents the complete development effort to make GP5 (Geth 1.17 fork) XDPoS V2 compatible for checkpoint sync from block 56M on Apothem testnet. The work is captured in branch `feat/trusted-checkpoint-sync` (PR #461).

**Status**: Nodes successfully sync from checkpoint 56M but get stuck at V2 checkpoint 56,829,600 with "validators not legit" error.

**Critical Finding**: V2 checkpoint headers store validator list in the `Extra` field (529 bytes), NOT in the `Validators` array (0 bytes).

---

## Issues Fixed (Chronological)

### 1. ✅ Checkpoint Hash Mismatch (v130 → v131)
- **Problem**: Checkpoint at 56,700,000 had wrong hash
- **Fix**: Updated `params/config.go` with correct hash from public RPC
- **Commit**: `e457df542`

### 2. ✅ V2 Genesis Config Missing (v129 → v130)
- **Problem**: `hasV2=false` because genesis JSON had no V2 config
- **Fix**: Patched `core/genesis.go` to inject V2 config for Apothem (chain ID 51)
- **Commit**: `1025f7e73`

### 3. ✅ V2 Engine Not Wired
- **Problem**: `EngineV2` was nil
- **Fix**: `SetEngineV2()` in `eth/backend.go:329`
- **Status**: Working

### 4. ✅ V2 Snapshot Init Failure (v131 → v132)
- **Problem**: `initial()` failed with "V2 gap header not in chain DB"
- **Fix**: Deferred snapshot initialization when gap header missing
- **Commit**: `f39f7d380`

### 5. 🔴 repairSnapshot Fails Without State (v132 → v133)
- **Problem**: Cannot read validator contract state during checkpoint sync
- **Fix Attempt**: Read validators from checkpoint header - but header has 0 validators!
- **Debug Finding**: `validatorsLen=0 validatorLen=65 penaltiesLen=0 extraLen=529`
- **Conclusion**: V2 stores validators in `Extra` field, not `Validators` array

---

## Critical Debug Output (v133)

```
INFO [repairSnapshot] V2 checkpoint: checking checkpoint header
    checkpoint=56,829,600
    gap=56,829,150
    validatorsLen=0      ← EMPTY!
    validatorLen=65      ← Proposer signature
    penaltiesLen=0
    extraLen=529         ← Contains V2 encoded data
```

**Key Insight**: The `Extra` field (529 bytes) likely contains the validator list in V2 format. Need to parse this field to extract masternodes for snapshot creation.

---

## Root Cause Analysis

### Why "validators not legit" happens:
1. Node syncs from checkpoint 56M without state (fast sync)
2. Reaches V2 checkpoint 56,829,600
3. `calcMasternodes` needs snapshot at gap block 56,829,150
4. Snapshot doesn't exist in DB
5. `repairSnapshot` tries to read contract state → fails (no state)
6. Fallback to checkpoint header → `Validators` array is empty
7. Returns error → "validators not legit"

### Why Validators array is empty:
- V2 blocks use `ExtraFields_v2` structure which only has `Round` and `QuorumCert`
- Validator list is encoded differently in V2 vs V1
- Likely stored in RLP-encoded `Extra` field

---

## Next Steps for Next Developer

1. **Parse V2 Extra field**: The `Extra` field (529 bytes) at checkpoint 56,829,600 contains the validator list. Need to understand V2 encoding and extract addresses.

2. **Look at V2 header encoding**: Check `core/types/consensus_v2.go` and `core/types/block.go` for how V2 stores validators in `Extra`.

3. **Implement extraction**: Modify `repairSnapshot` to:
   - Try reading `Validators` array first (for V1 compatibility)
   - If empty, parse `Extra` field for V2 encoded validator list
   - Create snapshot with extracted masternodes

4. **Build v134** with the fix and deploy to both nodes.

5. **Verify sync passes** 56,830,293.

---

## Technical Details

### V2 Configuration
```
SwitchBlock: 56,828,700 (V1)
First V2 Block: 56,828,701
Gap: 450
Epoch: 900
First V2 Checkpoint: 56,829,600
Stuck Block: 56,830,293
```

### Docker Image Build
```bash
# On 168 server
make geth
docker build --no-cache -t anilchinchawale/gp5-xdc:vXXX -f Dockerfile.gp5-ubuntu24 .
docker push anilchinchawale/gp5-xdc:vXXX
```

### Current Containers
- **Node 168**: `xdc-gp5-v133-cp56700-apothem-168` (stuck at 56,830,293)
- **Node 113**: `xdc03-gp5-v133-cp56700-apothem-113` (stuck at 56,830,293)

---

## Files Modified in Branch

- `params/config.go` - TrustedSyncCheckpoints + 56,700,000 checkpoint
- `core/genesis.go` - V2 config patching for Apothem
- `consensus/XDPoS/engines/engine_v2/engine.go` - repairSnapshot + initial() fixes + debug logging
- `Dockerfile.gp5-ubuntu24` - Ubuntu 24.04 base + correct ENTRYPOINT

---

## Resources

- **Branch**: `feat/trusted-checkpoint-sync`
- **Latest Commit**: `8150c7f61` (with development log)
- **Docker Hub**: `anilchinchawale/gp5-xdc:v133`
- **Stats Page**: https://stats.xdcindia.com/
- **Apothem RPC**: https://rpc.apothem.network/

---

*This issue serves as a complete handoff document for the next developer to continue V2 compatibility work.*