Service Management¶

Once your Verdikta Arbiter Node is installed, you'll need to know how to manage and monitor your services effectively. This section covers all aspects of day-to-day node operation.

Management Overview¶

Your Verdikta Arbiter Node consists of multiple services that work together:

graph TB
    A[Management Scripts] --> B[AI Node]
    A --> C[External Adapter]
    A --> D[Chainlink Node]
    A --> E[PostgreSQL Database]

    F[Status Monitoring] --> B
    F --> C
    F --> D
    F --> E

    G[Log Analysis] --> B
    G --> C
    G --> D
    G --> E

    subgraph "Core Services"
        B
        C
        D
        E
    end

Quick Management Commands¶

Located in your installation directory (default: ~/verdikta-arbiter-node/):

Start All Services¶

cd ~/verdikta-arbiter-node
./start-arbiter.sh

Stop All Services¶

cd ~/verdikta-arbiter-node
./stop-arbiter.sh

Check Service Status¶

cd ~/verdikta-arbiter-node
./arbiter-status.sh

Management Topics¶

Service Architecture¶

Service Dependencies¶

Understanding service dependencies helps with troubleshooting:

graph TD
    A[PostgreSQL] --> B[Chainlink Node]
    B --> C[External Adapter]
    C --> D[AI Node]

    E[Docker] --> A
    F[Node.js] --> C
    F --> D

    subgraph "External Dependencies"
        G[OpenAI API]
        H[Anthropic API]
        I[IPFS Service]
        J[Base Sepolia RPC]
    end

    D --> G
    D --> H
    D --> I
    B --> J

Service Ports¶

Default port assignments:

Installs made before 2026-10-21 published all four on every interface; see Closing the exposed ports.

Common Management Tasks¶

Daily Operations¶

• Health Checks: Verify all services are running
• Log Review: Check for errors or warnings
• Performance Monitoring: Track response times and resource usage

Weekly Maintenance¶

• Backup Creation: Backup configuration and database
• Log Rotation: Archive old logs to save space
• Update Checks: Review for available updates

Monthly Tasks¶

• Performance Analysis: Review monthly performance metrics
• Security Updates: Apply security patches
• Configuration Review: Verify settings are optimal

Service Health Indicators¶

Healthy Status¶

=== Verdikta Arbiter Node Status ===

[AI Node] Running on port 3000 ✓
[External Adapter] Running on port 8080 ✓  
[Chainlink Node] Running on port 6688 ✓
[PostgreSQL] Running on port 5432 ✓

All services are running correctly!

Warning Signs¶

• Services showing as "Stopped" or "Error"
• High memory or CPU usage
• Frequent restart cycles
• Network connectivity issues
• API authentication failures

Critical Issues¶

• Multiple services down
• Database corruption
• Out of disk space
• Network isolation
• Contract deployment failures

Resource Monitoring¶

System Resources¶

Monitor these key metrics:

• CPU Usage: Should stay below 80% under normal load
• Memory Usage: Watch for memory leaks in Node.js services
• Disk Space: Ensure sufficient space for logs and database
• Network: Monitor bandwidth for IPFS and API calls

Service-Specific Metrics¶

AI Node¶

• Request processing time
• AI model response rates
• IPFS retrieval performance
• Error rates by request type

Chainlink Node¶

• Job execution success rate
• Oracle response times
• Gas usage patterns
• LINK token balance

External Adapter¶

• Bridge request/response volume
• Data validation errors
• Timeout incidents
• Connection pool status

Automated Monitoring¶

Chainlink Health Watchdog (recommended)¶

The install ships chainlink-health-watchdog.sh in the install root. It was built for the failure mode behind the July 2026 commit-stage outage: the Chainlink node's RPC pool drops to 0 live nodes ("No live RPC nodes available"), transactions cannot be broadcast, and every service still looks "up" from the outside. Each pass it verifies that all four components are running (the cl-postgres and chainlink containers, the AI Node on :3000, the External Adapter on :8080), that every check on http://localhost:6688/health is passing, and that no recent "No live RPC nodes available" lines appear in the container log.

The alert subject says which it is: chainlink container NOT RUNNING / arbiter components NOT RUNNING: ... when something is down (after a host reboot the body adds how long ago the host booted and whether verdikta-arbiter.service is enabled), chainlink node CANNOT BROADCAST (0 live RPC / broadcaster down) when the container runs but its RPC pool is dead, and arbiter STOPPED by operator (stop-arbiter.sh) while the .arbiter-stopped marker that stop-arbiter.sh leaves exists (an upgrade or maintenance window).

# Install a cron entry that runs every 2 minutes
~/verdikta-arbiter-node/chainlink-health-watchdog.sh --install-cron 2

# Optional: also bring the node back. Components down -> start-arbiter.sh
# (starts whatever is missing, containers included); 0-live RPC -> restart
# the chainlink container. Always alerts first; one action per 30 min; never
# while stop-arbiter.sh's marker exists or in the first 5 min after a boot.
~/verdikta-arbiter-node/chainlink-health-watchdog.sh --install-cron 2 --self-heal

Alerts always go to syslog (logger -t verdikta-watchdog). To be paged, configure one or both of the following in ~/verdikta-arbiter-node/installer/.env:

# URL that receives one JSON event per watchdog run (OK heartbeat / ALERT /
# RECOVERED, tagged with your operator address + network). Point this at the
# Verdikta arbiter status page to appear on its "Arbiter Alerts" card:
WATCHDOG_ALERT_WEBHOOK="https://arbiters.verdikta.org/api/alerts"
# Shell command the human-readable alert text is piped into (paging channel)
WATCHDOG_ALERT_COMMAND="mail -s 'arbiter alert' ops@example.com"

No shared secret is needed for the status page: each event is signed with your operator owner key (the PRIVATE_KEY already in installer/.env) as an EIP-191 personal message — a purely local computation, no transaction and no gas — and the status page verifies the signer against your operator's on-chain owner(). Unsigned events are rejected, so the key, Node.js, and the ethers package (all present on a standard install) are required for reporting.

When the webhook points at the status page, healthy runs send heartbeats too — so the page can flag your arbiter as "not reporting" if the whole machine goes dark (a failure mode no self-hosted alert can catch).

Repeated alerts for the same unresolved condition are rate-limited (default 30 minutes), and a one-time "RECOVERED" notice is sent when the node returns to healthy.

Service Health Checks¶

You can additionally automate the broader diagnostics with cron jobs:

# Add to crontab (crontab -e)
# Check status every 5 minutes
*/5 * * * * cd ~/verdikta-arbiter-node && ./arbiter-status.sh >> ~/arbiter-health.log

Log Rotation¶

Container logs (chainlink, cl-postgres) are created with bounded Docker logging (json-file, 100 MB × 5 files). Installs created before this change can apply it in place — the chainlink container is recreated with the same image, volume, and network (the operator upgrade flow also offers this):

~/verdikta-arbiter-node/installer/util/apply-docker-log-rotation.sh          # check + prompt
~/verdikta-arbiter-node/installer/util/apply-docker-log-rotation.sh --check  # report only

Application logs (ai-node/logs, external-adapter/logs) are rotated by rotate-logs.sh in the install root: finished logs are compressed after a day, archives are pruned after 14 days, and any live log exceeding 200 MB is copy-truncated (last 5,000 lines preserved).

# Install a daily cron entry (03:17)
~/verdikta-arbiter-node/rotate-logs.sh --install-cron

# Or run once / preview
~/verdikta-arbiter-node/rotate-logs.sh --dry-run

Start at boot¶

A hosting-provider reboot used to leave every component down until an operator ran start-arbiter.sh by hand (2026-10-18: six VMs rebooted, the main mainnet node was down for 15 hours and missed three requests). Since then the installer and the upgrade make a node come back on its own:

• the chainlink and cl-postgres containers are created with --restart unless-stopped, so Docker restarts them after a reboot or daemon restart unless an operator stopped them;
• a systemd unit, verdikta-arbiter.service, runs the install's start-arbiter.sh after Docker and the network are up, as the install's user. It is a oneshot unit with no Restart=: nothing supervises or restarts the components behind your back, stop-arbiter.sh works as before, and the node stays down until start-arbiter.sh runs again (by hand or at the next boot).

arbiter-boot-service.sh in the install root owns both pieces:

~/verdikta-arbiter-node/arbiter-boot-service.sh status    # exit 0 = will come back after a reboot
sudo ~/verdikta-arbiter-node/arbiter-boot-service.sh enable   # unit + restart policies (install/upgrade offer this)
sudo ~/verdikta-arbiter-node/arbiter-boot-service.sh disable  # remove the unit, containers back to --restart no

disable also turns the container policies off on purpose: with the unit gone, containers that restart on their own would leave a half-up node after a reboot (the Chainlink node taking requests, no adapter to answer them). arbiter-status.sh and arbiter-doctor.sh (svc.boot_start) report the state; systemctl status verdikta-arbiter shows the last boot's start log.

Port exposure¶

arbiter-firewall.sh in the install root keeps ports 5432, 6688, 3000 and 8080 off the internet (iptables drop rules on the public interface only, kept across reboots by verdikta-arbiter-firewall.service; the arbiter's own traffic is not affected):

~/verdikta-arbiter-node/arbiter-firewall.sh status        # exit 0 = closed
sudo ~/verdikta-arbiter-node/arbiter-firewall.sh enable   # install/upgrade offer this
sudo ~/verdikta-arbiter-node/arbiter-firewall.sh disable  # reopen: rules, unit and AI Node marker removed

arbiter-status.sh ("Public ports") and arbiter-doctor.sh (svc.public_ports) report the state. Because a host cannot test its own external reachability, they check bind addresses and rules; confirm from another machine with nc -z -w5 <public-ip> 5432.

Emergency Procedures¶

Service Recovery¶

If services become unresponsive:

1. Stop All Services

   ./stop-arbiter.sh
   

2. Check System Resources

   df -h  # Disk space
   free -m  # Memory usage
   top     # CPU usage
   

3. Review Logs

   # Check for recent errors
   tail -n 100 ~/verdikta-arbiter-node/*/logs/*.log
   

4. Restart Services

   ./start-arbiter.sh
   

Database Recovery¶

If PostgreSQL issues occur:

# Check database status
docker ps | grep postgres

# View database logs
docker logs postgres-db

# Restart database only
docker restart postgres-db

Network Issues¶

For connectivity problems:

# Test external API connectivity
curl -s https://api.openai.com/v1/models
curl -s https://api.anthropic.com/

# Test blockchain connectivity
curl -s -X POST -H "Content-Type: application/json" \
  --data '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}' \
  YOUR_RPC_ENDPOINT

Performance Optimization¶

Resource Tuning¶

• AI Node: Adjust worker processes based on CPU cores
• Chainlink Node: Optimize database connection pools
• PostgreSQL: Tune memory settings for workload
• Docker: Set appropriate resource limits

Scaling Considerations¶

• Horizontal Scaling: Multiple arbiter nodes for load distribution
• Vertical Scaling: Increase resources for single node performance
• Load Balancing: Distribute requests across multiple nodes

Best Practices¶

Security¶

• Regular Updates: Keep all components updated
• Access Control: Limit network access to necessary ports
• Key Management: Rotate API keys and credentials regularly
• Monitoring: Set up alerts for security events

Reliability¶

• Backup Strategy: Regular backups of configuration and data
• Redundancy: Consider multiple nodes for critical deployments
• Monitoring: Comprehensive health and performance monitoring
• Documentation: Keep operational procedures documented

Performance¶

• Resource Monitoring: Track usage patterns and optimize
• Log Management: Implement proper log rotation and archival
• Network Optimization: Use efficient API calls and caching
• Database Maintenance: Regular PostgreSQL maintenance

Getting Started¶

Ready to learn service management? Start with:

1. Starting Services - How to properly start your node
2. Status Monitoring - Monitor your node's health
3. Log Analysis - Understanding and analyzing logs