Service Management¶

Once your Verdikta Arbiter Node is installed, you'll need to know how to manage and monitor your services effectively. This section covers all aspects of day-to-day node operation.

Management Overview¶

Your Verdikta Arbiter Node consists of multiple services that work together:

graph TB
    A[Management Scripts] --> B[AI Node]
    A --> C[External Adapter]
    A --> D[Chainlink Node]
    A --> E[PostgreSQL Database]

    F[Status Monitoring] --> B
    F --> C
    F --> D
    F --> E

    G[Log Analysis] --> B
    G --> C
    G --> D
    G --> E

    subgraph "Core Services"
        B
        C
        D
        E
    end

Quick Management Commands¶

Located in your installation directory (default: ~/verdikta-arbiter-node/):

Start All Services¶

cd ~/verdikta-arbiter-node
./start-arbiter.sh

Stop All Services¶

cd ~/verdikta-arbiter-node
./stop-arbiter.sh

Check Service Status¶

cd ~/verdikta-arbiter-node
./arbiter-status.sh

Management Topics¶

Service Architecture¶

Service Dependencies¶

Understanding service dependencies helps with troubleshooting:

graph TD
    A[PostgreSQL] --> B[Chainlink Node]
    B --> C[External Adapter]
    C --> D[AI Node]

    E[Docker] --> A
    F[Node.js] --> C
    F --> D

    subgraph "External Dependencies"
        G[OpenAI API]
        H[Anthropic API]
        I[IPFS Service]
        J[Base Sepolia RPC]
    end

    D --> G
    D --> H
    D --> I
    B --> J

Service Ports¶

Default port assignments:

Common Management Tasks¶

Daily Operations¶

• Health Checks: Verify all services are running
• Log Review: Check for errors or warnings
• Performance Monitoring: Track response times and resource usage

Weekly Maintenance¶

• Backup Creation: Backup configuration and database
• Log Rotation: Archive old logs to save space
• Update Checks: Review for available updates

Monthly Tasks¶

• Performance Analysis: Review monthly performance metrics
• Security Updates: Apply security patches
• Configuration Review: Verify settings are optimal

Service Health Indicators¶

Healthy Status¶

=== Verdikta Arbiter Node Status ===

[AI Node] Running on port 3000 ✓
[External Adapter] Running on port 8080 ✓  
[Chainlink Node] Running on port 6688 ✓
[PostgreSQL] Running on port 5432 ✓

All services are running correctly!

Warning Signs¶

• Services showing as "Stopped" or "Error"
• High memory or CPU usage
• Frequent restart cycles
• Network connectivity issues
• API authentication failures

Critical Issues¶

• Multiple services down
• Database corruption
• Out of disk space
• Network isolation
• Contract deployment failures

Resource Monitoring¶

System Resources¶

Monitor these key metrics:

• CPU Usage: Should stay below 80% under normal load
• Memory Usage: Watch for memory leaks in Node.js services
• Disk Space: Ensure sufficient space for logs and database
• Network: Monitor bandwidth for IPFS and API calls

Service-Specific Metrics¶

AI Node¶

• Request processing time
• AI model response rates
• IPFS retrieval performance
• Error rates by request type

Chainlink Node¶

• Job execution success rate
• Oracle response times
• Gas usage patterns
• LINK token balance

External Adapter¶

• Bridge request/response volume
• Data validation errors
• Timeout incidents
• Connection pool status

Automated Monitoring¶

Chainlink Health Watchdog (recommended)¶

The install ships chainlink-health-watchdog.sh in the install root. It is purpose-built for the failure mode behind the July 2026 commit-stage outage: the Chainlink node's RPC pool drops to 0 live nodes ("No live RPC nodes available"), transactions cannot be broadcast, and every service still looks "up" from the outside. Each pass it verifies the chainlink container is running, that every check on http://localhost:6688/health is passing, and that no recent "No live RPC nodes available" lines appear in the container log.

# Install a cron entry that runs every 2 minutes
~/verdikta-arbiter-node/chainlink-health-watchdog.sh --install-cron 2

# Optional stopgap: also restart the chainlink container when the 0-live
# condition is detected (always alerts first; max one restart per 30 min)
~/verdikta-arbiter-node/chainlink-health-watchdog.sh --install-cron 2 --self-heal

Alerts always go to syslog (logger -t verdikta-watchdog). To be paged, configure one or both of the following in ~/verdikta-arbiter-node/installer/.env:

# URL that receives one JSON event per watchdog run (OK heartbeat / ALERT /
# RECOVERED, tagged with your operator address + network). Point this at the
# Verdikta arbiter status page to appear on its "Arbiter Alerts" card:
WATCHDOG_ALERT_WEBHOOK="https://arbiters.verdikta.org/api/alerts"
# Shell command the human-readable alert text is piped into (paging channel)
WATCHDOG_ALERT_COMMAND="mail -s 'arbiter alert' ops@example.com"

No shared secret is needed for the status page: each event is signed with your operator owner key (the PRIVATE_KEY already in installer/.env) as an EIP-191 personal message — a purely local computation, no transaction and no gas — and the status page verifies the signer against your operator's on-chain owner(). Unsigned events are rejected, so the key, Node.js, and the ethers package (all present on a standard install) are required for reporting.

When the webhook points at the status page, healthy runs send heartbeats too — so the page can flag your arbiter as "not reporting" if the whole machine goes dark (a failure mode no self-hosted alert can catch).

Repeated alerts for the same unresolved condition are rate-limited (default 30 minutes), and a one-time "RECOVERED" notice is sent when the node returns to healthy.

Service Health Checks¶

You can additionally automate the broader diagnostics with cron jobs:

# Add to crontab (crontab -e)
# Check status every 5 minutes
*/5 * * * * cd ~/verdikta-arbiter-node && ./arbiter-status.sh >> ~/arbiter-health.log

Log Rotation¶

Container logs (chainlink, cl-postgres) are created with bounded Docker logging (json-file, 100 MB × 5 files). Installs created before this change can apply it in place — the chainlink container is recreated with the same image, volume, and network (the operator upgrade flow also offers this):

~/verdikta-arbiter-node/installer/util/apply-docker-log-rotation.sh          # check + prompt
~/verdikta-arbiter-node/installer/util/apply-docker-log-rotation.sh --check  # report only

Application logs (ai-node/logs, external-adapter/logs) are rotated by rotate-logs.sh in the install root: finished logs are compressed after a day, archives are pruned after 14 days, and any live log exceeding 200 MB is copy-truncated (last 5,000 lines preserved).

# Install a daily cron entry (03:17)
~/verdikta-arbiter-node/rotate-logs.sh --install-cron

# Or run once / preview
~/verdikta-arbiter-node/rotate-logs.sh --dry-run

Restart on Failure¶

Implement automatic restart for critical services:

# systemd service file example
[Unit]
Description=Verdikta Arbiter Node
After=docker.service

[Service]
Type=oneshot
ExecStart=/home/user/verdikta-arbiter-node/start-arbiter.sh
ExecStop=/home/user/verdikta-arbiter-node/stop-arbiter.sh
RemainAfterExit=yes
Restart=on-failure

[Install]
WantedBy=multi-user.target

Emergency Procedures¶

Service Recovery¶

If services become unresponsive:

1. Stop All Services

   ./stop-arbiter.sh
   

2. Check System Resources

   df -h  # Disk space
   free -m  # Memory usage
   top     # CPU usage
   

3. Review Logs

   # Check for recent errors
   tail -n 100 ~/verdikta-arbiter-node/*/logs/*.log
   

4. Restart Services

   ./start-arbiter.sh
   

Database Recovery¶

If PostgreSQL issues occur:

# Check database status
docker ps | grep postgres

# View database logs
docker logs postgres-db

# Restart database only
docker restart postgres-db

Network Issues¶

For connectivity problems:

# Test external API connectivity
curl -s https://api.openai.com/v1/models
curl -s https://api.anthropic.com/

# Test blockchain connectivity
curl -s -X POST -H "Content-Type: application/json" \
  --data '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}' \
  YOUR_RPC_ENDPOINT

Performance Optimization¶

Resource Tuning¶

• AI Node: Adjust worker processes based on CPU cores
• Chainlink Node: Optimize database connection pools
• PostgreSQL: Tune memory settings for workload
• Docker: Set appropriate resource limits

Scaling Considerations¶

• Horizontal Scaling: Multiple arbiter nodes for load distribution
• Vertical Scaling: Increase resources for single node performance
• Load Balancing: Distribute requests across multiple nodes

Best Practices¶

Security¶

• Regular Updates: Keep all components updated
• Access Control: Limit network access to necessary ports
• Key Management: Rotate API keys and credentials regularly
• Monitoring: Set up alerts for security events

Reliability¶

• Backup Strategy: Regular backups of configuration and data
• Redundancy: Consider multiple nodes for critical deployments
• Monitoring: Comprehensive health and performance monitoring
• Documentation: Keep operational procedures documented

Performance¶

• Resource Monitoring: Track usage patterns and optimize
• Log Management: Implement proper log rotation and archival
• Network Optimization: Use efficient API calls and caching
• Database Maintenance: Regular PostgreSQL maintenance

Getting Started¶

Ready to learn service management? Start with:

1. Starting Services - How to properly start your node
2. Status Monitoring - Monitor your node's health
3. Log Analysis - Understanding and analyzing logs