More efficient test for binary files in matrix.comp.

[ukmo-ccbunney]

Pull Request Summary

Provides a more efficient test for binary files when running matrix.comp

Description

Currently, matrix.comp recursively checks all files in a work directory using the command

 binaryfiles=`grep . -r * | grep 'Binary file' | sed -e "s/^Binary file //" -e "s/ matches$//"` 

This is very slow as:

1. It will unnecessarily read the entirety of any text files (. matches "any character")
2. It recurses into the CMake build* and exe directories which have a lot of files in.

A better solution is to use the standard linux file command to check whether individual files are "plain text" files or binary. This is not only a much quicker and more efficient test, but it will only be called when needed (as apposed on a complete recursive listing of the directory)

I assuming a fairly modern implementation of the file command that accepts the -i or --mime flags for robustness purposes. The test is encapsulated as a function at the top of the matrix.comp script to facilitate modification if needed.

Performance

Some results of running new vs old version of matrix.comp:

• ww3_tp2.2 (10 work directories):
  • original version: 3m 7s
  • new version: 9s
• ts1 (11 work directories):
  • original version 2m2s
  • new version: 7s
• tp2.9 (20 work directories):
  • 6m 9s
  • 20s

Note: When testing performance, it is important to not rerun matix.comp twice on the same regression test directory as the file system will likely have cached the files and the timings will be much quicker the second time.

I make performance comparisons by testing in two separate WW3 regtest trees (and testing the regtests against themselves)

Issue(s) addressed

Fixes #1031

Commit Message

More efficient test for binary files in matrix.comp

Check list

• Branch is up to date with the authoritative repository (NOAA-EMC) develop branch.
• Checked the checklist for a developer submitting to develop.
• If a version number update is required, checked the updating version number checklist.
• If a new feature was added, a regression test for testing the new feature is added.

Testing

• How were these changes tested? Comparison of regression test output; diffs out matrixCompFull.txt are same
• Are the changes covered by regression tests? (If not, why? Do new tests need to be added?) N/A
• Have the matrix regression tests been run (if yes, please note HPC and compiler)? N/A
• Please indicate the expected changes in the regression test output, (Note the list of known non-identical tests.) N/A

[ukmo-ccbunney]

As part of this PR, I wonder if others would also agree to the removal of the "do not compare build or exe directory" messages to STDOUT?

 testing case: ww3_tp2.9; run: ./work_PR2_UQ_a_MPI
do not compare build or exe directories build
do not compare build or exe directories build_SHRD
do not compare build or exe directories exe
do not compare build or exe directories build
do not compare build or exe directories build_SHRD
do not compare build or exe directories exe

these get printed multiple times for each test and dominate the output to the screen.

[JessicaMeixner-NOAA]

As part of this PR, I wonder if others would also agree to the removal of the "do not compare build or exe directory" messages to STDOUT?

 testing case: ww3_tp2.9; run: ./work_PR2_UQ_a_MPI
do not compare build or exe directories build
do not compare build or exe directories build_SHRD
do not compare build or exe directories exe
do not compare build or exe directories build
do not compare build or exe directories build_SHRD
do not compare build or exe directories exe

these get printed multiple times for each test and dominate the output to the screen.

@ukmo-ccbunney I'm okay with that

[MatthewMasarik-NOAA]

I'm also OK with removing the repeated prints of "do not compare build or exe directory".

[ukmo-ccbunney]

Timing results from running matrix.comp on a full regression test matrix:

• old code: 272m (~4.5 hours)
• new code 63m

So - not as fast a speedup as the results for individual regtests show above (not sure why?), but still a significant improvement!

[ukmo-ccbunney]

I'm also OK with removing the repeated prints of "do not compare build or exe directory".

I've pushed a commit to remove this message.

[JessicaMeixner-NOAA]

Thanks @ukmo-ccbunney ! I am running tests to do comparisons between the old and the new on our machines. Those should be finished soon and then I can submit the two matrix.comps for comparisons. I'll let you know how it goes!

[JessicaMeixner-NOAA]

This improved the speed from 80min to 53 min on hera for me. Thank you for this speed up @ukmo-ccbunney !

[ukmo-ccbunney]

This improved the speed from 80min to 53 min on hera for me. Thank you for this speed up @ukmo-ccbunney !

I'm impressed they were running in 80 minutes before! I think the regtests really stress the filesystem on our HPC... it takes over 4.5 hours for me to run them!

[JessicaMeixner-NOAA]

@ukmo-ccbunney well, i just had a set of matrix.comp take 4.5 hours.... with the updates, so .... I wonder if we need to rethink what we're comparing and how we're comparing it to relieve some of the stress. 4.5 hours is crazy, but I get that once in a blue moon these days. I usually have the tests get stuck on the ufs tests at the end if something does take a long time.

[ukmo-ccbunney]

@ukmo-ccbunney well, i just had a set of matrix.comp take 4.5 hours.... with the updates, so .... I wonder if we need to rethink what we're comparing and how we're comparing it to relieve some of the stress. 4.5 hours is crazy, but I get that once in a blue moon these days. I usually have the tests get stuck on the ufs tests at the end if something does take a long time.

Wow. 4.5 hours with this update? I wonder if that is just some spurious filesystem contention that you are experiencing.
This update has certainly helped speed thiings up a lot on my HPC, but every system is different!
I think it is definitely high time we looked at the matrix.comp script and what/how it is comparing.