Parse pytest warning summaries from colcon logs.

[nuclearsandwich]

The parser itself is actually part of the primary Jenkins configuration
and requires separate documentation and potentially being separated out as a warning
parser plugin.

Groovy-based parsers are not allowed to run on console output so this
scans the stderr.log files from colcon test output.

A test build with this parser enabled is here: https://ci.ros2.org/job/test_ci_linux/39/colcon-log-pytest-warnings

The regexp is

^\[[^]]+] .* \.\.\. (?<path>[^:]*):(?<lineno>\d+): (?<exname>[^:]*): (?<message>.*)$

and the Groovy script

import edu.hm.hafner.analysis.Severity

builder.setFileName(matcher.group("path"))
        .setLineStart(Integer.parseInt(matcher.group("lineno")))
        .setSeverity(Severity.WARNING_NORMAL)
        .setType(matcher.group("exname"))
        .setMessage(matcher.group("message"))

return builder.buildOptional();

The above assumes the colcon log is printing elapsed time and the test which triggered the warning is being printed before the ... by Pytest.

[dirk-thomas]

Even with the timestamp prefix removed) the proposed patch will only extract a small subset of cases from the following example snippets from stderr.log files from colcon test:

nightly_linux_debug - test_launch_ros

�[33m=============================== warnings summary ===============================�[0m
test/test_launch_ros/actions/test_node.py::TestNode::test_launch_node_with_parameter_descriptions
  /home/jenkins-agent/workspace/nightly_linux_debug/ws/src/ros2/launch_ros/test_launch_ros/test/test_launch_ros/actions/test_node.py:186: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
    parameters.append((name, yaml.load(value)))

-- Docs: https://docs.pytest.org/en/stable/warnings.html

nightly_win_deb - test_launch_ros

Warning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.9 it will stop working

Warning: The TerminalReporter.writer attribute is deprecated, use TerminalReporter._tw instead at your own risk.
See https://docs.pytest.org/en/stable/deprecations.html#terminalreporter-writer for more information.


============================== warnings summary ===============================
test/test_launch_ros/actions/test_composable_node_container.py::test_composable_node_container
  Warning: unclosed event loop <ProactorEventLoop running=False closed=False debug=False>

test/test_launch_ros/actions/test_node.py::TestNode::test_launch_node_with_parameter_descriptions
  Warning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.

-- Docs: https://docs.pytest.org/en/stable/warnings.html

nightly_win_deb - domain_coordinator

Warning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.9 it will stop working

Warning: The TerminalReporter.writer attribute is deprecated, use TerminalReporter._tw instead at your own risk.
See https://docs.pytest.org/en/stable/deprecations.html#terminalreporter-writer for more information.


============================== warnings summary ===============================
test/test_domain_coordinator.py::TestUniqueness::test_known_order
test/test_domain_coordinator.py::TestUniqueness::test_with_forced_collision
  Warning: unclosed <socket.socket fd=560, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0>

test/test_domain_coordinator.py::TestUniqueness::test_known_order
test/test_domain_coordinator.py::TestUniqueness::test_quickly
  Warning: unclosed <socket.socket fd=148, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0>

test/test_domain_coordinator.py::TestUniqueness::test_quickly
test/test_domain_coordinator.py::TestUniqueness::test_with_forced_collision
test/test_domain_coordinator.py::TestUniqueness::test_with_forced_collision
  Warning: unclosed <socket.socket fd=600, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0>

test/test_domain_coordinator.py::TestUniqueness::test_quickly
  Warning: unclosed <socket.socket fd=672, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 22229)>

test/test_domain_coordinator.py::TestUniqueness::test_quickly
  Warning: unclosed <socket.socket fd=668, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 22228)>

test/test_domain_coordinator.py::TestUniqueness::test_with_forced_collision
  Warning: unclosed <socket.socket fd=484, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('0.0.0.0', 22261)>

-- Docs: https://docs.pytest.org/en/stable/warnings.html

nightly_win_deb - tf2_ros_py

Warning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.9 it will stop working

Warning: The TerminalReporter.writer attribute is deprecated, use TerminalReporter._tw instead at your own risk.
See https://docs.pytest.org/en/stable/deprecations.html#terminalreporter-writer for more information.

[foonathan::memory] Allocator foonathan::memory::new_allocator (at 0000000000000000) leaked 3840 bytes.
[foonathan::memory] Allocator foonathan::memory::heap_allocator (at 0000000000000000) leaked 94152 bytes.

nightly_linux_debug - tf2_ros_py

[foonathan::memory] Allocator foonathan::memory::new_allocator (at (nil)) leaked 456 bytes.
[foonathan::memory] Allocator foonathan::memory::heap_allocator (at (nil)) leaked 71560 bytes.

[nuclearsandwich]

I hadn't encountered any of the multi-line examples when I first drafted this PR. We'd either need to create a mega-regexp that handles either the single-line or multi-line cases or try to move the analysis into a Java-based Jenkins plugin where we have a bit more flexibility.

Another option could be creating a Java-based analysis model plugin for the warnings. I think further work on this approach would require determining whether we should target pytest's warning summaries or warnings in their natural habitat. Pytest documentation suggests the warnings summary (as documented) can be disabled and they recommend doing so if there is existing support for warnings in our analysis suite however there currently isn't so either way we would have to supply the plugin.

[dirk-thomas]

@nuclearsandwich I propose to close this in favor of #509.