e2e: color Slack alert by failed-agent tier (red vs yellow)

150bbbb→main· toothbrush·2w ago·1 file·+72 added/-17 removed

Every merge reddened main because the 4 chronically-flaky agents (cursor-cli, copilot-cli, codex, factoryai-droid) share one matrix with the reliable ones, so the Slack page fired on every run and the signal went dead. Classify failed jobs: a reliable agent (claude-code, opencode, gemini-cli, roger-roger) or Windows failing is RED (real regression); only flaky agents failing is YELLOW. Mixed failures are RED but list both tiers, so flaky-agent visibility is preserved. Switched the payload to attachments so the color bar renders.

Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com

Sessions

b75934be80e3View transcript

?\Make E2E Tests Reliable on MainClaude Code·Opus 4.8[1m]·2 steps

Changes

1

• .github/workflows

• Me2e.yml+72/-17

127 unmodified lines

128
129
130
131
132
131
132
133
134
135
136
137
138
139
140
136
137
138
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
2 unmodified lines

194
195
196
147
197
198
149
150
151
152
153
154
155
156
157
199
200
201
159
160
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218

127 unmodified lines

needs: [e2e-tests, e2e-windows]
    if: ${{ always() && (needs.e2e-tests.result == 'failure' || needs.e2e-windows.result == 'failure') && github.event_name == 'push' }}
    steps:
      - name: Get failed agents
        id: failed
      # Classify the failed jobs into two severities:
      #   RED    — a "reliable" agent (or Windows) failed. These pass on every
      #            healthy run, so a failure here is a real regression.
      #   YELLOW — only a known-flaky agent failed. Worth investigating, but not
      #            a `main` regression. Reported so we don't lose visibility.
      - name: Classify failures
        id: classify
        env:
          GH_TOKEN: ${{ github.token }}
        run: |
          failed=$(gh api repos/${{ github.repository }}/actions/runs/${{ github.run_id }}/jobs \
            --jq '[.jobs[] | select(.conclusion == "failure") | .name | if test("\(") then capture("\((?<agent>[^"]+)\)") | .agent else . end] | join(", ")')
          echo "agents=$failed" >> "$GITHUB_OUTPUT"
          set -euo pipefail
          # Reliable tier (RED). The e2e-windows reusable job is also RED but
          # is matched separately below since it has no "(agent)" suffix.
          reliable="claude-code opencode gemini-cli roger-roger"

failed_names=$(gh api --paginate \
            "repos/${{ github.repository }}/actions/runs/${{ github.run_id }}/jobs" \
            --jq '.jobs[] | select(.conclusion == "failure") | .name')

red=""
          yellow=""
          while IFS= read -r name; do
            [ -z "$name" ] && continue
            case "$name" in
              *e2e-windows*) red="$red windows"; continue ;;
            esac
            # "e2e-tests (cursor-cli)" -> "cursor-cli"
            agent="${name#*(}"; agent="${agent%)*}"
            case " $reliable " in
              *" $agent "*) red="$red $agent" ;; 
              *)            yellow="$yellow $agent" ;;
            esac
          done <<EOF
          $failed_names
          EOF

# Normalize to comma-separated lists. paste -d takes a delimiter
          # *set* used in rotation, so join with a single ',' then space it out.
          red=$(echo $red | tr ' ' '\n' | sed '/^$/d' | paste -sd, - | sed 's/,/, /g')
          yellow=$(echo $yellow | tr ' ' '\n' | sed '/^$/d' | paste -sd, - | sed 's/,/, /g')

if [ -n "$red" ]; then
            color="#d50200"
            header=":red_circle: *E2E Tests Failed* on \`main\`"
          else
            color="#daa038"
            header=":large_yellow_circle: *E2E flaky-agent failures* on \`main\`"
          fi

body=""
          [ -n "$red" ]    && body="${body}Reliable agents (regression): *${red}*\n"
          [ -n "$yellow" ] && body="${body}Flaky agents (investigate): *${yellow}*\n"

{
            echo "color=$color"
            echo "header=$header"
            echo "body=$body"
          } >> "$GITHUB_OUTPUT"

- name: Notify Slack of E2E failure
        uses: slackapi/slack-github-action@45a88b9581bfab2566dc881e2cd66d334e621e2c # v3.0.3
2 unmodified lines

webhook-type: incoming-webhook
          payload: |
            {
              "blocks": [\
              "attachments": [\
                {\
                  "type": "section",\
                  "text": {\
                    "type": "mrkdwn",\
                    "text": ":red_circle: *E2E Tests Failed* on `main`\n\nFailed agents: *${{ steps.failed.outputs.agents }}*\n<${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}|View run details>"\
                  }\
                },\
                {\
                  "type": "context",\
                  "elements": [\
                  "color": "${{ steps.classify.outputs.color }}",\
                  "blocks": [\
                    {\
                      "type": "mrkdwn",\
                      "text": "Commit: <${{ github.server_url }}/${{ github.repository }}/commit/${{ github.sha }}|${{ github.sha }}> by ${{ github.actor }}"\
                      "type": "section",\
                      "text": {\
                        "type": "mrkdwn",\
                        "text": "${{ steps.classify.outputs.header }}\n\n${{ steps.classify.outputs.body }}<${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}|View run details>"\
                      }\
                    },\
                    {\
                      "type": "context",\
                      "elements": [\
                        {\
                          "type": "mrkdwn",\
                          "text": "Commit: <${{ github.server_url }}/${{ github.repository }}/commit/${{ github.sha }}|${{ github.sha }}> by ${{ github.actor }}"\
                        }\
                      ]\
                    }\
                  ]\
                }\
```\
\
M.github/workflows/e2e.yml+72/-17