Documentation Missing: Exit Codes Prevent Cache Generation in Local Checks

[TobenderZephyr]

---

Issue Type

Documentation Gap / Unclear Behavior

---

Summary

The Checkmk documentation for local checks does not clearly explain that script exit codes affect cache generation. Scripts that exit with non-zero status codes (1, 2, 3) will not have their output cached, even when placed in cache-interval subdirectories like /usr/lib/check_mk_agent/local/900/.

This causes cached local checks to fail silently, with no output appearing in the agent's <<<local:sep(0)>>> section.

---

Affected Documentation

• Official Local Checks Documentation
• Section: 3.7. Executing asynchronously and caching output

---

The Problem

Expected Behavior (Based on Documentation)

When a local check script is placed in a cache interval subdirectory:

/usr/lib/check_mk_agent/local/900/my_check

Users expect:

1. Script runs every 900 seconds
2. Output is cached
3. Cached output is returned on intermediate agent queries

Actual Behavior

If the script exits with a non-zero code (even when outputting valid Checkmk local check format), the agent:

1. Executes the script
2. Discards the output
3. Does not create/update cache file
4. No output appears in agent dump

Root Cause

Checkmk agent interprets non-zero exit codes as "script failure" and refuses to cache "failed" executions, regardless of valid output format.

---

Reproduction Steps

1. Create a local check that exits with status 1 (WARNING)

cat > /usr/lib/check_mk_agent/local/900/test_warning << 'EOF'
#!/bin/bash
echo '1 "Test Service" count=5 WARNING - This is a warning'
exit 1  # Exit with WARNING status
EOF

chmod +x /usr/lib/check_mk_agent/local/900/test_warning

2. Execute the script manually

/usr/lib/check_mk_agent/local/900/test_warning
echo "Exit code: $?"

Output:

1 "Test Service" count=5 WARNING - This is a warning
Exit code: 1

3. Check agent output

cmk-agent-ctl dump | grep -A10 "<<<local:sep(0)>>>"

Result: No output from test_warning appears (cache not created)

4. Fix: Change exit code to 0

cat > /usr/lib/check_mk_agent/local/900/test_warning << 'EOF'
#!/bin/bash
echo '1 "Test Service" count=5 WARNING - This is a warning'
exit 0  # Always exit 0 for successful script execution
EOF

5. Check agent output again

cmk-agent-ctl dump | grep -A10 "<<<local:sep(0)>>>"

Result: Output now appears and is cached ✓

---

The Confusion

Two Different "Status Codes"

1. Script Exit Code (shell $?)

   • Indicates whether the script executed successfully
   • 0 = script ran without errors
   • Non-zero = script encountered an error

2. Monitoring Status (first field in output)

   • Indicates the monitoring state of the service
   • 0 = OK
   • 1 = WARNING
   • 2 = CRITICAL
   • 3 = UNKNOWN

The Undocumented Rule

For cached local checks: Script must always exit 0, with monitoring status in the output line.

# WRONG (prevents caching):
echo '1 "Service" metric=5 WARNING message'
exit 1

# CORRECT (enables caching):
echo '1 "Service" metric=5 WARNING message'
exit 0

---

Impact

User Experience Issues

1. Silent Failure: No error messages when cache doesn't work
2. Debugging Difficulty: Script works manually but not via agent
3. Wasted Time: Users spend hours troubleshooting permissions, paths, syntax
4. Incorrect Assumptions: Users assume caching is broken, not exit codes

Who Is Affected

• Users converting monitoring plugins to local checks
• Users migrating from Nagios/Icinga (where exit codes = monitoring states)
• Developers creating custom local checks
• Anyone using cached local checks (subdirectories with cache intervals)

---

Proposed Documentation Fix

Add to Section 3.7 ("Executing asynchronously and caching output")

New subsection: "Exit Codes and Caching Behavior"

Important: For cached local checks to work correctly, your script must always exit with code 0 (success), regardless of the monitoring state.

The monitoring status is determined by the first field in the output line (0=OK, 1=WARNING, 2=CRITICAL, 3=UNKNOWN), not by the script's exit code.

Script exit codes affect caching:

• exit 0 → Script executed successfully, output is cached
• exit 1/2/3 → Script "failed", output is discarded, cache not created

Example:

#!/bin/bash
# Check disk space
USAGE=85

if [ $USAGE -gt 80 ]; then
    # Service is WARNING (1), but script succeeds (exit 0)
    echo "1 \"Disk Space\" usage=${USAGE}% WARNING - Disk ${USAGE}% full"
    exit 0  # ← Always exit 0 for caching to work
else
    echo "0 \"Disk Space\" usage=${USAGE}% OK - Disk ${USAGE}% full"
    exit 0
fi

Common mistake:

# This will NOT be cached:
echo "2 \"Service\" - CRITICAL problem detected"
exit 2  # ← Agent discards output, doesn't cache

Add Troubleshooting Section

Troubleshooting: Cached local check not appearing in output

If your cached local check works when executed manually but doesn't appear in agent output:

1. Check script exit code:

   /usr/lib/check_mk_agent/local/900/your_check
   echo $?  # Must be 0

2. Verify output format:

   # Correct format:
   0 "Service Name" metric=value Status message

3. Check cache directory permissions:

   ls -la /var/lib/check_mk_agent/cache/

4. Review agent logs for errors

---

Additional Information

Environment

• Checkmk Version: Affects all versions with local check caching support
• Operating System: Linux (confirmed), likely affects AIX, FreeBSD, OpenWrt, Windows
• Agent Type: check_mk_agent (standard Linux agent)

Related Code

Agent script location: /usr/bin/check_mk_agent

Relevant section that handles local check caching (approximate):

# Pseudocode from agent script
for script in /usr/lib/check_mk_agent/local/[0-9]*/*; do
    output=$($script)
    exit_code=$?
    
    if [ $exit_code -eq 0 ]; then
        cache_output "$output"  # Only caches if exit 0
    else
        # Silently discards output
    fi
done

---

Suggested Fix Locations

1. Main Documentation:

   • https://docs.checkmk.com/latest/en/localchecks.html
   • Section 3.7 (add exit code clarification)

2. Example Scripts:

   • Update all example scripts to show exit 0
   • Add comments explaining the reason

3. Agent Code (Optional):

   • Add debug logging when non-zero exit prevents caching
   • Consider warning message: "Local check exited with code X, output not cached"

---

Keywords for Searchability

• local checks
• cached local checks
• asynchronous local checks
• exit code
• cache not working
• local check not appearing
• <<local:sep(0)>>
• check_mk_agent local
• 900 subdirectory

---

References

• Official Docs: https://docs.checkmk.com/latest/en/localchecks.html
• This troubleshooting session: [Link to discussion if applicable]

---

Community Impact

Based on this troubleshooting session, this issue likely affects many users but goes unreported because:

1. Users assume they made a configuration error
2. No clear error messages guide users to the root cause
3. Manual execution works, making the issue non-obvious
4. Users often abandon caching and move to non-cached local checks as a workaround

Adding this documentation would save significant community troubleshooting time.