Building an NVIDIA NIM API Watchdog

NVIDIA's free NIM tier gives you access to some powerful models: Llama, DeepSeek, GLM, with just an API key. The catch? The stated rate limit is 40 requests per minute, but in practice the limits feel more aggressive. Timeouts and 429s show up without warning.

I got tired of guessing when the API would cooperate, so I built a watchdog script that pings the endpoint on a schedule, logs the response status, and raises an alert when things go sideways.

The problem

Rate limiting on free API tiers isn't new, but NVIDIA's implementation has some quirks. The docs say 40 RPM. That number feels aspirational. During testing, I hit 429 errors at far lower request rates, and the retry-after headers (when they exist) don't always match what actually happens.

For an AI agent that runs autonomously, this is a real problem. If the API drops mid-task, the agent stalls. If it hits a 429 during a cron job, the job fails silently or burns through retries without making progress.

What the watchdog does

The script is straightforward:

Sends a lightweight completion request to the NIM endpoint
Logs the HTTP status code and round-trip time
Tracks consecutive failures and recovery events
Alerts when the API has been down for more than a configurable threshold

import requests
import time
from datetime import datetime

NIM_URL = "https://integrate.api.nvidia.com/v1"
NIM_KEY = os.environ["NVIDIA_API_KEY"]

def check_nim():
    headers = {
        "Authorization": f"Bearer {NIM_KEY}",
        "Content-Type": "application/json"
    }
    payload = {
        "model": "nvidia/llama-3.1-8b-instruct",
        "messages": [{"role": "user", "content": "ping"}],
        "max_tokens": 5,
    }
    start = time.time()
    r = requests.post(
        f"{NIM_URL}/chat/completions",
        headers=headers, json=payload, timeout=30
    )
    elapsed = time.time() - start
    return r.status_code, elapsed

Nothing fancy. The point is steady observation, not clever engineering.

Running it as a cron job

The watchdog runs hourly through Hermes Agent's built-in cron system. Each run logs a single line: timestamp, status code, and response time. If three consecutive checks fail, it sends an alert. When the API recovers, it logs that too.

One thing I learned the hard way: don't try to benchmark the rate limit by sending bursts. That just gets you throttled harder. A single request per check gives you a cleaner signal.

Results so far

After running this for a couple of weeks, the pattern is clear. The API works most of the time, but there are windows, sometimes hours long, where responses slow to a crawl or flat-out fail. The 40 RPM claim is for burst capacity. Sustained use seems to hit a lower ceiling that NVIDIA doesn't document.

The watchdog catches these dips. When a scheduled job fails, I can check the logs and see whether it was the API or something else. That alone saves a lot of guessing.

What's next

The obvious improvement is fallback providers. When NIM is down, the agent could route to OpenRouter, Groq, or Gemini instead. Hermes Agent supports multiple providers in its config. The hard part is simply getting API keys set up for each one.

I'd also like to aggregate the watchdog data over time and see if there are predictable patterns. Do outages cluster at certain hours? Is there a weekly cycle? Right now I have hunches but not evidence.

The script is on GitHub if you want to adapt it for your own setup.