Skip to main content

Upgrade Your Node

Version & Network

Mainnet: crossfi-mainnet-1 • Client: crossfi-node v0.4.0

Cosmovisor​

The Cosmos SDK provides a convenient process manager that wraps around the crossfid binary and can automatically swap in new binaries upon a successful governance upgrade proposal. Cosmovisor is highly recommended for validator operations as it enables:

  • Automatic upgrades: Seamlessly switches to new binaries during governance upgrades
  • Zero-downtime upgrades: Minimizes validator downtime during network upgrades
  • Backup management: Automatically backs up old binaries (configurable)
  • Rollback capability: Can revert to previous versions if needed
  • Integration with systemd: Works perfectly with system service managers for automatic restarts
Why Use Cosmovisor?

For validator operations, Cosmovisor combined with systemd ensures MTTR (Mean Time To Recovery) < 2 minutes, which is critical for maintaining high validator uptime and avoiding slashing penalties.

More information can be found in cosmos.network docs and cosmos-sdk/cosmovisor/readme.

Setup​

Install Cosmovisor

  • Install the latest version of Cosmovisor: 
    # Install from source
    go install cosmossdk.io/tools/cosmovisor/cmd/cosmovisor@latest

    # Verify installation
    cosmovisor version
  • Note: Ensure you have Go 1.19+ installed for the latest version.

Set Environment Variables

  • Configure Cosmovisor environment variables: 
    # Add to your shell profile (~/.bashrc, ~/.zshrc, or ~/.profile)
    echo "# Setup Cosmovisor" >> ~/.profile
    echo "export DAEMON_NAME=crossfid" >> ~/.profile  
    echo "export DAEMON_HOME=$HOME/.crossfi" >> ~/.profile
    echo "export DAEMON_RESTART_AFTER_UPGRADE=true" >> ~/.profile
    echo "export DAEMON_ALLOW_DOWNLOAD_BINARIES=false" >> ~/.profile
    echo "export UNSAFE_SKIP_BACKUP=false" >> ~/.profile

    # Apply changes
    source ~/.profile
  • Environment Variables Explained:
    • DAEMON_RESTART_AFTER_UPGRADE=true: Automatically restart after upgrades
    • DAEMON_ALLOW_DOWNLOAD_BINARIES=false: Disable automatic binary downloads (recommended for security)
    • UNSAFE_SKIP_BACKUP=false: Keep backups of old binaries

Create Directory Structure

  • Set up the required Cosmovisor directory structure: 
    # Create directories
    mkdir -p $DAEMON_HOME/cosmovisor/genesis/bin
    mkdir -p $DAEMON_HOME/cosmovisor/upgrades

    # Copy current binary
    cp $(which crossfid) $DAEMON_HOME/cosmovisor/genesis/bin/

    # Verify setup
    cosmovisor version
    ls -la $DAEMON_HOME/cosmovisor/
  • The directory structure should look like: 
    ~/.crossfi/cosmovisor/
    ├── current -> genesis or upgrades/<name>
    ├── genesis
    │   └── bin
    │       └── crossfid
    └── upgrades
        └── <upgrade_name>
            └── bin
                └── crossfid

Test Cosmovisor

  • Test that Cosmovisor can run your node: 
    # Test cosmovisor (should show same version as crossfid)
    cosmovisor version

    # Start node with cosmovisor (for testing)
    cosmovisor run start --help
  • ⚠️ Important: Don't run cosmovisor run start yet if you plan to use systemd - we'll configure that next.

Preparing an Upgrade​

Cosmovisor will continually poll the $DAEMON_HOME/data/upgrade-info.json for new upgrade instructions. When an upgrade is ready, node operators can download the new binary and place it under $DAEMON_HOME/cosmovisor/upgrades/<name>/bin where <name> is the URI-encoded name of the upgrade as specified in the upgrade module plan.

Version & Network

Mainnet: crossfi-mainnet-1 • Client: crossfi-node v0.4.0

It is possible to have Cosmovisor automatically download the new binary. To do this set the following environment variable.

export DAEMON_ALLOW_DOWNLOAD_BINARIES=true

Cosmovisor with systemd​

For production validator operations, it's essential to run Cosmovisor as a system service to ensure automatic restarts and maintain high availability.

Why systemd + Cosmovisor?​

🔄 Automatic Recovery

  • Restarts node immediately after crashes
  • Maintains MTTR < 2 minutes
  • Reduces slashing risk from downtime
  • Works during system reboots

⚙️ Production Benefits

  • Professional service management
  • Integrated logging with journald
  • Resource limit enforcement
  • Dependency management

systemd Service Setup​

Create Service File

  • Create a systemd service file for your CrossFi validator: 
    sudo nano /etc/systemd/system/crossfid.service
  • Add the following configuration (replace <username> with your actual username): 
    [Unit]
    Description=CrossFi Daemon (Cosmovisor)
    After=network-online.target
    Wants=network-online.target

    [Service]
    Type=exec
    User=<username>
    Group=<username>
    ExecStart=/home/<username>/go/bin/cosmovisor run start
    Restart=always
    RestartSec=3
    LimitNOFILE=4096
    Environment="DAEMON_NAME=crossfid"
    Environment="DAEMON_HOME=/home/<username>/.crossfi"
    Environment="DAEMON_ALLOW_DOWNLOAD_BINARIES=false"
    Environment="DAEMON_RESTART_AFTER_UPGRADE=true"
    Environment="DAEMON_LOG_BUFFER_SIZE=512"
    Environment="UNSAFE_SKIP_BACKUP=false"

    [Install]
    WantedBy=multi-user.target

Configure Service

  • Key Configuration Options:
    • Restart=always: Always restart the service if it stops
    • RestartSec=3: Wait 3 seconds before restarting
    • LimitNOFILE=4096: Set file descriptor limit
    • After=network-online.target: Wait for network connectivity
  • Environment Variables:
    • DAEMON_ALLOW_DOWNLOAD_BINARIES=false: Security best practice
    • DAEMON_RESTART_AFTER_UPGRADE=true: Auto-restart after upgrades
    • DAEMON_LOG_BUFFER_SIZE=512: Optimize logging performance

Enable and Start Service

  • Enable and start the CrossFi service: 
    # Reload systemd configuration
    sudo systemctl daemon-reload

    # Enable service to start on boot
    sudo systemctl enable crossfid.service

    # Start the service
    sudo systemctl start crossfid.service

    # Check service status
    sudo systemctl status crossfid.service

Monitor and Manage

  • Essential monitoring commands: 
    # View live logs
    sudo journalctl -u crossfid.service -f

    # View recent logs
    sudo journalctl -u crossfid.service --since "1 hour ago"

    # Check service status
    sudo systemctl status crossfid.service

    # Restart service (if needed)
    sudo systemctl restart crossfid.service

    # Stop service
    sudo systemctl stop crossfid.service

Testing Automatic Restart​

Test Process Recovery

  • Kill the crossfid process to test automatic restart: 
    # Find the crossfid process ID
    ps aux | grep crossfid

    # Kill the process (replace PID with actual process ID)
    sudo kill -9 <PID>

    # Check that systemd restarts it within seconds
    sudo systemctl status crossfid.service
  • Expected behavior: The service should restart within 3 seconds.

Monitor Recovery Time

  • Monitor logs to confirm quick recovery: 
    # Watch logs in real-time during the test
    sudo journalctl -u crossfid.service -f
  • You should see:
    • Process termination logged
    • Automatic restart initiated
    • Node resuming operations
    • Total downtime < 10 seconds

Test Reboot Persistence

  • Verify the service starts automatically after system reboot: 
    # Check if service is enabled
    sudo systemctl is-enabled crossfid.service

    # Should return: enabled
  • After a system reboot, verify the service starts automatically: 
    sudo systemctl status crossfid.service

Advanced Configuration​

Resource Limits​

For high-performance validators, consider additional resource limits:

[Service]
# Memory limits (adjust based on your system)
MemoryMax=8G
MemoryHigh=6G

# CPU limits (optional)
CPUQuota=200%

# I/O limits (optional)
IOWeight=500

# Process limits
LimitNPROC=32768
LimitNOFILE=65536

Logging Configuration​

Configure advanced logging options:

[Service]
# Log to specific file (optional)
StandardOutput=journal+console
StandardError=journal+console

# Additional environment variables
Environment="DAEMON_LOG_BUFFER_SIZE=1024"
Environment="DAEMON_LOG_LEVEL=info"

Security Hardening​

For enhanced security, add these options:

[Service]
# Run with restricted privileges
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/home/&lt;username&gt;/.crossfi

# Network restrictions
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6

Troubleshooting​

Common issues and solutions:

  • Permission issues: Ensure the user has access to all paths
  • Binary not found: Verify cosmovisor and crossfid are in PATH
  • Environment variables: Check all required variables are set
# Debug service startup sudo systemctl status crossfid.service -l sudo journalctl -u crossfid.service --since "10 minutes ago"
Production Recommendation

For mission-critical validators, consider setting up monitoring alerts that notify you if the service restarts frequently, indicating potential underlying issues.

Manual Software Upgrade​

First, stop your instance of `