add silence annotations

[Spaceman1701]

Silence annotations are... annotations... for silences!

Just like alert annotations, silence annotations are an optional LabelSet which provides structured key->value data on silences. Annotations are added to the underlying cluster model for silences so they're synced properly between nodes in HA mode.

This is a simple change that adds a lot of value: with structured fields, it's possible to add both machine and human readble metadata for silences. In HRT, we use annotations to indicate the purpose of silences, associate them with machine processes, and encode conventions. The format is flexible so it's possible to encode any number of things.

[SoloJacobs]

I think a bit more test coverage would go here, most importantly a round trip test:
post silence (using amtool) -> get silence -> compare annotations.

cli/silence_add.go and api/v2/compat.go could also be tested.

I also assume that we expect amtool to not have an error, if you post to an older alertmanager version?

[Spaceman1701]

I think a bit more test coverage would go here, most importantly a round trip test: post silence (using amtool) -> get silence -> compare annotations.

cli/silence_add.go and api/v2/compat.go could also be tested.

Sure, I'll add a test case for the round trip, that's a good idea.

I also assume that we expect amtool to not have an error, if you post to an older alertmanager version?

Ah yeah, let me also make sure that works - I think if you set annotations and try to post to an old alertmanager it should be expected to break. If you don't, I think it's reasonable to make it work.

We don't really have a version skew policy, but I guess at some point we'll need to think about guidelines for which versions of amtool ought to be compatible with which versions of alertmanager...

[TheMeier]

amtool already checks if the versions differ

For a users perspective the optimal behaviour would be amtool printing a warning if annotations are passed but the server version does not handle them but work otherwise.

[Spaceman1701]

amtool already checks if the versions differ

For a users perspective the optimal behaviour would be amtool printing a warning if annotations are passed but the server version does not handle them but work otherwise.

Agreed this would be the nicest behavior. However, it's a bit of work to implement because I don't know what semantic version of alertmanager will start supporting silence annotations...

I could think of something to make this work, but given that this is a bit of a corner case (the user is using a new version of amtool on an old version of alertmanager and is trying to set annotations), I don't think we should worry too much.

[siavashs]

LGTM

[SoloJacobs]

I quite like the new test! Thanks for adding it. Regarding the amtool - alertmanager compatibility: Well at least there is a warning. So, LGTM 👍