[Bug]: Matrix notification sender issue

[byteonabeach]

Bug description

Hey,
I'm running Netdata on my home lab and trying to set up Matrix notifications for alerts. Everything seems configured right in health_alarm_notify.conf (homeserver URL, access token and room ID), but when I test it with /usr/libexec/netdata/plugins.d/alarm-notify.sh test, it fails with HTTP 404 and this Matrix error: {"errcode":"M_UNRECOGNIZED","error":"M_UNRECOGNIZED: Not Found"}.

What happens:

The curl command in debug looks like this (sanitized):

/usr/bin/curl -X POST --data '{...}' https://myhomeserver.example.com/_matrix/client/r0/rooms/%21roomid%3Amyhomeserver.address.tld/send/m.room.message?access_token=valid_token_here

And the response is always that M_UNRECOGNIZED not found thing.

I tested manually with curl using the right format (PUT instead of POST, add a txnId like timestamp, and put the token in Authorization header), and it works fine:

txnid=$(date +%s%N)
curl -X PUT -d '{"msgtype":"m.text", "body":"Test"}' \
"https://myhomeserver.example.com/_matrix/client/r0/rooms/%21roomid:myhomeserver.address.tld/send/m.room.message/${txnid}" \
-H "Authorization: Bearer valid_token_here"

Gets a proper event_id back.

Expected behavior

Notifications should post to the Matrix room without errors.

Steps to reproduce

1. Enabled Matrix in health_alarm_notify.conf like that:
   • SEND_MATRIX="YES"
   • MATRIX_HOMESERVER="https://myhomeserver.address.tld"
   • MATRIX_ACCESSTOKEN="valid_token_here"
   • DEFAULT_RECIPIENT_MATRIX="!roomid:myhomeserver.address.tld"
2. Bot account is joined to the room.
3. Ran the test script as the netdata user: sudo su -s /bin/bash netdata then /usr/libexec/netdata/plugins.d/alarm-notify.sh test
4. Or with debug: export NETDATA_ALARM_NOTIFY_DEBUG=1 and run again.

Screenshots

No response

Error Logs

No response

Desktop

Netdata version: v2.8.4
OS: Fedora 43
Matrix homeserver: Tuwunel 1.4.7

Additional context

That version of a send_matrix function fixed issue:

send_matrix() {
  local homeserver="${1}" rooms="${2}" httpcode sent=0 payload txnid

  [ "${SEND_MATRIX}" != "YES" ] && return 1
  [ -z "${MATRIX_ACCESSTOKEN}" ] && return 1

  accesstoken="${MATRIX_ACCESSTOKEN}"

  case "${status}" in
  WARNING) emoji="warning" ;;
  CRITICAL) emoji="red_circle" ;;
  CLEAR) emoji="white_check_mark" ;;
  *) emoji="white_circle" ;;
  esac

  txnid=$(date +%s%N)

  for room in ${rooms}; do
    webhook="$homeserver/_matrix/client/r0/rooms/$(urlencode $room)/send/m.room.message/${txnid}" 
    payload="$(
  cat <<EOF
  {
    "msgtype": "m.notice",
    "format": "org.matrix.custom.html",
      "formatted_body": "${emoji} ${host} ${status_message} - <b>${name//_/ }</b><br>${chart}<br><a href=\"${goto_url}\">${alarm}</a><br><i>${info}</i>",
      "body": "${emoji} ${host} ${status_message} - ${name//_/ } ${chart} ${goto_url} ${alarm} ${info}"
  }
EOF
    )"

    httpcode=$(docurl -X PUT --data "${payload}" -H "Authorization: Bearer ${accesstoken}" "${webhook}") 
    if [ "${httpcode}" == "200" ]; then
      info "sent Matrix notification to '${room}' for ${notification_description}"
      sent=$((sent + 1))
    else
      error "failed to send Matrix notification to '${room}' for ${notification_description}, with HTTP response status code ${httpcode}."
    fi
  done

  [ ${sent} -gt 0 ] && return 0

  return 1
}

[ktsaou]

Thanks for the detailed bug report and analysis. You correctly identified the root cause.

Root Cause Confirmed

The send_matrix() function was using an outdated Matrix API format:

• POST instead of PUT
• Missing transaction ID (txnid) in the URL path
• Access token in query parameter instead of Authorization header

Fix

PR netdata/netdata#21659 addresses all three issues:

- webhook="$homeserver/_matrix/client/r0/rooms/$(urlencode $room)/send/m.room.message?access_token=$accesstoken"
- httpcode=$(docurl -X POST --data "${payload}" "${webhook}")
+ txnid="nd_$(date +%s)_$$_$RANDOM"
+ webhook="$homeserver/_matrix/client/r0/rooms/$(urlencode $room)/send/m.room.message/$txnid"
+ httpcode=$(docurl -X PUT -H "Authorization: Bearer ${accesstoken}" --data "${payload}" "${webhook}")

Changes:

Aspect	Before	After
HTTP Method	POST	PUT
Transaction ID	None	nd_{epoch}_{pid}_{random}
Authentication	Query param	Authorization: Bearer header

Workaround (until release)

You can patch /usr/libexec/netdata/plugins.d/alarm-notify.sh manually with the changes above, or wait for the next release.