CITGM is in deep trouble

[targos]

We got a regression in the latest v22.5.0 that affects at least yarn and npm: nodejs/node#53902

It seems that citgm would have caught it, but for jobs end up green even when they have lots of failures.

From https://ci.nodejs.org/view/Node.js-citgm/job/citgm-smoker/3457/nodes=debian12-x64/

22:06:57 warn: @nearform/bubbleprof npm-install:| npm error Exit handler never called!
22:06:57 warn: @nearform/bubbleprof npm-install:| npm error This is an error with npm itself. Please report this error at:                                                                                                
22:06:57 warn:                                  | npm error   <https://github.com/npm/cli/issues>                                                                                                                         
22:06:57 warn:                                  | npm error A complete log of this run can be found in: /home/iojs/tmp/citgm_tmp/bf0d267e-49e6-467a-bf5b-dcf938d40bcc/home/.npm/_logs/2024-07-16T20_05_57_634Z-debug-0.log
22:06:58 error: failure             | The canary is dead: 
22:07:03 error: @nearform/bubbleprof done| done - the test suite for @nearform/bubbleprof version 7.0.1 failed
22:07:03 (node:783693) [DEP0174] DeprecationWarning: Calling promisify on a function that returns a Promise is likely a mistake.
22:07:03 (node:783693) [DEP0174] DeprecationWarning: Calling promisify on a function that returns a Promise is likely a mistake.
22:08:09 warn: @nearform/doctor npm-install:| npm error Exit handler never called!
22:08:09 warn: @nearform/doctor npm-install:| npm error This is an error with npm itself. Please report this error at:                                                                                                
22:08:09 warn:                              | npm error   <https://github.com/npm/cli/issues>                                                                                                                         
22:08:09 warn:                              | npm error A complete log of this run can be found in: /home/iojs/tmp/citgm_tmp/50a9288d-b4f4-4ba6-88c9-df6e8ba9bca3/home/.npm/_logs/2024-07-16T20_07_07_903Z-debug-0.log
22:08:10 error: failure             | The canary is dead: 
22:08:16 error: @nearform/doctor done| done - the test suite for @nearform/doctor version 8.1.0 failed

[ljharb]

This is also filed as npm/cli#7657, and nodejs/node#53462 was pointed to me as a possible cause.

[targos]

I think the cause is nodejs/node#53627, but this issue is not about the regression itself. It's about the fact that CITGM somehow reported successful runs instead of failures.

[targos]

@nodejs/build-infra I believe this is an issue with the Jenkins job. If I run node bin/citgm-all.js --includeTags binary-split, I can reproduce the regression and it correctly exits with code 1.

[richardlau]

Possibilities:

• The job runs citgm via eval:

# Using eval so we can use a complicated CITGM_COMMAND such as 'NODE_OPTIONS=--xzy citgm-all'
eval $CITGM_COMMAND "--nodedir=$npm_config_nodedir -v $CITGM_LOGLEVEL -x $PWD/report.xml -q $NPM_LOGLEVEL --tmpDir $temp"

• We have Do not fail the build on empty test results turned on
  

I'll turn off Do not fail the build on empty test results.

[richardlau]

Rerunning https://ci.nodejs.org/view/Node.js-citgm/job/citgm-smoker/3457/nodes=debian12-x64/ with Do not fail the build on empty test results disabled:
https://ci.nodejs.org/view/Node.js-citgm/job/citgm-smoker/3459/nodes=debian12-x64/console

[richardlau]

Rerunning https://ci.nodejs.org/view/Node.js-citgm/job/citgm-smoker/3457/nodes=debian12-x64/ with Do not fail the build on empty test results disabled: https://ci.nodejs.org/view/Node.js-citgm/job/citgm-smoker/3459/nodes=debian12-x64/console

With Do not fail the build on empty test results disabled, the CITGM run for 22.5.0 now fails due to having no test results.

[targos]

That's better, thanks ! Now do we need to investigate why it "thinks" there are no results ?

[richardlau]

https://ci.nodejs.org/view/Node.js-citgm/job/citgm-smoker/3459/nodes=debian12-x64/console

13:55:12 Recording test results
13:55:12 ERROR: Step ‘Publish JUnit test result report’ failed: No test report files were found. Configuration error?

which likely means smoker/report.xml wasn't generated:

we use -x to specify outputting to report.xml.

# Using eval so we can use a complicated CITGM_COMMAND such as 'NODE_OPTIONS=--xzy citgm-all'
eval $CITGM_COMMAND "--nodedir=$npm_config_nodedir -v $CITGM_LOGLEVEL -x $PWD/report.xml -q $NPM_LOGLEVEL --tmpDir $temp"

[targos]

Here's the report.xml I get with Node.js 22.5.0 using the following command: node bin/citgm-all.js --includeTags binary-split -x $PWD/report.xml

<testsuite name="citgm">
  <testcase name="binary-split-v1.0.5" time="12.196">
    <system-out><![CDATA[ > [email protected] test
 > standard --verbose && node test.js
 npm error Exit handler never called!
 npm error This is an error with npm itself. Please report this error at:
 npm error   <https://github.com/npm/cli/issues>
 npm error A complete log of this run can be found in: /var/folders/9n/05y6s4ls78515vmnbd86hktm0000gn/T/9987debd-ab8e-402c-a690-08daa9c9d7de/home/.npm/_logs/2024-07-20T09_32_11_815Z-debug-0.log
 sh: standard: command not found]]></system-out>
    <failure message="module test suite failed">Error: The canary is dead: </failure>
  </testcase>
  <testcase name="clinic" time="NaN">
    <system-out/>
    <skipped/>
  </testcase>
  <testcase name="debug" time="NaN">
    <system-out/>
    <skipped/>
  </testcase>
  <testcase name="ember-cli" time="NaN">
    <system-out/>
    <skipped/>
  </testcase>
  <testcase name="eslint" time="NaN">
    <system-out/>
    <skipped/>
  </testcase>
  <testcase name="glob" time="NaN">
    <system-out/>
    <skipped/>
  </testcase>
  <testcase name="got" time="NaN">
    <system-out/>
    <skipped/>
  </testcase>
  <testcase name="gulp" time="NaN">
    <system-out/>
    <skipped/>
  </testcase>
  <testcase name="jest" time="NaN">
    <system-out/>
    <skipped/>
  </testcase>
  <testcase name="q" time="NaN">
    <system-out/>
    <skipped/>
  </testcase>
  <testcase name="underscore" time="NaN">
    <system-out/>
    <skipped/>
  </testcase>
  <testcase name="watchify" time="NaN">
    <system-out/>
    <skipped/>
  </testcase>
  <testcase name="yeoman-generator" time="NaN">
    <system-out/>
    <skipped/>
  </testcase>
</testsuite>

And with Node.js 22.5.1:

<testsuite name="citgm">
  <testcase name="binary-split-v1.0.5" time="9.769">
    <system-out><![CDATA[ added 267 packages in 7s
 > [email protected] test
 > standard --verbose && node test.js
 TAP version 13
 # ldjson file
 ok 1 should be equal
 # custom matcher
 ok 2 should be equal
 ok 3 should be equal
 # long matcher
 ok 4 should be equal
 ok 5 should be equal
 ok 6 should be equal
 # matcher at index 0 check
 ok 7 should be equal
 ok 8 should be equal
 ok 9 should be equal
 # chunked input
 ok 10 should be equal
 # chunked input with long matcher
 ok 11 should be equal
 ok 12 should be equal
 # lookbehind in multi character matcher
 ok 13 should be equal
 ok 14 should be equal
 ok 15 should be equal
 1..15
 # tests 15
 # pass  15
 # ok]]></system-out>
  </testcase>
  <testcase name="clinic" time="NaN">
    <system-out/>
    <skipped/>
  </testcase>
  <testcase name="debug" time="NaN">
    <system-out/>
    <skipped/>
  </testcase>
  <testcase name="ember-cli" time="NaN">
    <system-out/>
    <skipped/>
  </testcase>
  <testcase name="eslint" time="NaN">
    <system-out/>
    <skipped/>
  </testcase>
  <testcase name="glob" time="NaN">
    <system-out/>
    <skipped/>
  </testcase>
  <testcase name="got" time="NaN">
    <system-out/>
    <skipped/>
  </testcase>
  <testcase name="gulp" time="NaN">
    <system-out/>
    <skipped/>
  </testcase>
  <testcase name="jest" time="NaN">
    <system-out/>
    <skipped/>
  </testcase>
  <testcase name="q" time="NaN">
    <system-out/>
    <skipped/>
  </testcase>
  <testcase name="underscore" time="NaN">
    <system-out/>
    <skipped/>
  </testcase>
  <testcase name="watchify" time="NaN">
    <system-out/>
    <skipped/>
  </testcase>
  <testcase name="yeoman-generator" time="NaN">
    <system-out/>
    <skipped/>
  </testcase>
</testsuite>

[RedYetiDev]

FWIW another fast api regression made it through CITGM: nodejs/node#54518