Ouster partial cloud issue

Ouster partial cloud issue

This issue has appeared before and was techincally solved by migrating from the ros-ouster driver to the official Ouster ROS2 driver. Additionally, the sensor firmware was updated.

Lets check Centos7 ethtool and ifconfig output for enp129s0f1

[seb_adm@core5 ~]$ sudo ethtool enp129s0f1
Settings for enp129s0f1:
        Supported ports: [ TP ]
        Supported link modes:   100baseT/Full 
                                1000baseT/Full 
                                10000baseT/Full 
        Supported pause frame use: Symmetric
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  100baseT/Full 
                                1000baseT/Full 
                                10000baseT/Full 
        Advertised pause frame use: No
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Speed: 1000Mb/s
        Duplex: Full
        Port: Twisted Pair
        PHYAD: 0
        Transceiver: internal
        Auto-negotiation: on
        MDI-X: Unknown
        Supports Wake-on: umbg
        Wake-on: g
        Current message level: 0x00000007 (7)
                               drv probe link
        Link detected: yes

[seb_adm@core5 ~]$ ifconfig enp129s0f1
enp129s0f1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.31.0.1  netmask 255.255.0.0  broadcast 172.31.255.255
        inet6 fe80::ae1f:6bff:fe8a:5219  prefixlen 64  scopeid 0x20<link>
        ether ac:1f:6b:8a:52:19  txqueuelen 1000  (Ethernet)
        RX packets 380812674  bytes 506625747397 (471.8 GiB)
        RX errors 3  dropped 1860  overruns 0  frame 3
        TX packets 56690544  bytes 84152309668 (78.3 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

The interface reports RX errors and dropped packages

Also, ouster's UI reports an ETHERNET_LINK_BAD warnning:

The ros2 driver is configured with the following parameters:

/**:
  ros__parameters:
    # sensor_hostname[required]: hostname or IP address of the sensor (IP4 or
    # IP6).
    sensor_hostname: '172.31.2.10'
    # udp_dest[optional]: hostname or multicast group IP where the sensor will
    # send UDP data packets.
    udp_dest: '172.31.0.1'
    # mtp_dest[optional]: hostname IP address for receiving data packets via
    # multicast, by default it is INADDR_ANY, so packets will be received on
    # first available network interface.
    mtp_dest: ''
    # mtp_main[optional]: if true, then configure and reinit the sensor,
    # otherwise start client with active configuration of sensor
    mtp_main: false
    # lidar_mode[optional]: resolution and rate; possible values: { 512x10,
    # 512x20, 1024x10, 1024x20, 2048x10, 4096x5 }. Leave empty to remain on
    # current the lidar mode.
    lidar_mode: '1024x10'
    # timestamp_mode[optional]: method used to timestamp measurements; possible
    # values:
    # - TIME_FROM_INTERNAL_OSC
    # - TIME_FROM_SYNC_PULSE_IN
    # - TIME_FROM_PTP_1588
    # - TIME_FROM_ROS_TIME: This option uses the time of reception of first
    #                       packet of a LidarScan as the timestamp of the IMU,
    #                       PointCloud2 and LaserScan messages.
    time: 'TIME_FROM_ROS_TIME'
    # ptp_utc_tai_offset[optional]: UTC/TAI offset in seconds to apply when
    # TIME_FROM_PTP_1588 timestamp mode is used.
    ptp_utc_tai_offset: -37.0
    # udp_profile_lidar[optional]: lidar packet profile; possible values:
    # - LEGACY: not recommended
    # - RNG19_RFL8_SIG16_NIR16
    # - RNG15_RFL8_NIR8
    # - RNG19_RFL8_SIG16_NIR16_DUAL
    # - FUSA_RNG15_RFL8_NIR8_DUAL
    udp_profile_lidar: 'LEGACY'
    # metadata[optional]: path to save metadata file to, if left empty a file
    # with the sensor hostname or ip will be crearted in ~/.ros folder.
    metadata: ''
    # lidar_port[optional]: port value should be in the range [0, 65535]. If you
    # use 0 as the port value then the first available port number will be
    # assigned.
    lidar_port: 0
    # imu_port[optional]: port value should be in the range [0, 65535]. If you
    # use 0 as the port value then the first available port number will be
    # assigned.
    imu_port: 0
    # sensor_frame[optional]: name to use when referring to the sensor frame.
    sensor_frame: lidar_ouster_top
    # lidar_frame[optional]: name to use when referring to the lidar frame.
    lidar_frame: lidar_ouster_sensor
    # imu_frame[optional]: name to use when referring to the imu frame.
    imu_frame: imu_ouster
    # point_cloud_frame[optional]: which frame of reference to use when
    # generating PointCloud2 or LaserScan messages, select between the values of
    # lidar_frame and sensor_frame.
    point_cloud_frame: lidar_ouster_top
    # proc_mask[optional]: use any combination of the 4 flags IMG, PCL, IMU and
    # SCAN to enable or disable their respective messages.
    proc_mask: PCL
    # scan_ring[optional]: use this parameter in conjunction with the SCAN flag
    # to select which beam of the LidarScan to use when producing the LaserScan
    # message. Choose a value the range [0, sensor_beams_count).
    scan_ring: 0
    # use_system_default_qos[optional]: if false, data are published with sensor
    # data QoS. This is preferable for production but default QoS is needed for
    # rosbag. See: https://github.com/ros2/rosbag2/issues/125
    use_system_default_qos: true
    point_type: xyzi

Usually these parameters should make the Ouster's packet size smaller, however with a ros2 topic bw on its topic, we see > 120 MB/s as if we are runnig the lidar at max resolution:

root@core5:/opt/ros_ws# ros2 topic bw /sensor/lidar/top/points
Subscribed to [/sensor/lidar/top/points]
147.58 MB/s from 33 messages
        Message size mean: 4.19 MB min: 4.19 MB max: 4.19 MB
136.37 MB/s from 63 messages
        Message size mean: 4.19 MB min: 4.19 MB max: 4.19 MB
127.07 MB/s from 89 messages
        Message size mean: 4.19 MB min: 4.19 MB max: 4.19 MB
115.42 MB/s from 100 messages
        Message size mean: 4.19 MB min: 4.19 MB max: 4.19 MB
118.12 MB/s from 100 messages
        Message size mean: 4.19 MB min: 4.19 MB max: 4.19 MB
115.53 MB/s from 100 messages
        Message size mean: 4.19 MB min: 4.19 MB max: 4.19 MB
118.99 MB/s from 100 messages
        Message size mean: 4.19 MB min: 4.19 MB max: 4.19 MB
129.72 MB/s from 100 messages
        Message size mean: 4.19 MB min: 4.19 MB max: 4.19 MB
133.35 MB/s from 100 messages
        Message size mean: 4.19 MB min: 4.19 MB max: 4.19 MB
123.17 MB/s from 100 messages
        Message size mean: 4.19 MB min: 4.19 MB max: 4.19 MB

The frequency of the topic is at 30 hz rather than 10 hz

^Croot@core5:/opt/ros_ws# ros2 topic hz /sensor/lidar/top/points
average rate: 30.180
        min: 0.004s max: 0.099s std dev: 0.03524s window: 31
average rate: 31.078
        min: 0.004s max: 0.099s std dev: 0.03469s window: 63
average rate: 30.709
        min: 0.004s max: 0.100s std dev: 0.03475s window: 93
average rate: 30.215
        min: 0.004s max: 0.101s std dev: 0.03562s window: 122
average rate: 30.153
        min: 0.004s max: 0.101s std dev: 0.03556s window: 152

Stopping and running the ros2 driver yields the same result.

What if we turn off and turn on the sensor from the Dataspeed dashboard? ...

After few connection hiccops, the server is able to ping the sensor. The UI took longer than expected to open

We have now more Active and logged warnings

Lets run the sensor driver and visualisation with foxglove...

Bandwith and topic frequency looks normal again

root@core5:/opt/ros_ws# ros2 topic bw /sensor/lidar/top/points
Subscribed to [/sensor/lidar/top/points]
43.80 MB/s from 10 messages
        Message size mean: 4.19 MB min: 4.19 MB max: 4.19 MB
42.85 MB/s from 20 messages
        Message size mean: 4.19 MB min: 4.19 MB max: 4.19 MB
42.55 MB/s from 30 messages
        Message size mean: 4.19 MB min: 4.19 MB max: 4.19 MB
42.39 MB/s from 40 messages
        Message size mean: 4.19 MB min: 4.19 MB max: 4.19 MB

root@core5:/opt/ros_ws# ros2 topic hz /sensor/lidar/top/points
average rate: 10.010
        min: 0.098s max: 0.101s std dev: 0.00081s window: 11
average rate: 10.007
        min: 0.098s max: 0.101s std dev: 0.00066s window: 22
average rate: 10.003
        min: 0.098s max: 0.101s std dev: 0.00063s window: 32
average rate: 10.003
        min: 0.098s max: 0.101s std dev: 0.00058s window: 43
root@core5:/opt/ros_ws# 

Lidar looks normal again??

What if we modify the ros2 driver parameters so the lidar runs at max resolution?

Looks okay, also topics stats looks okay:

Subscribed to [/sensor/lidar/top/points]
129.53 MB/s from 9 messages
        Message size mean: 12.58 MB min: 12.58 MB max: 12.58 MB
127.57 MB/s from 19 messages
        Message size mean: 12.58 MB min: 12.58 MB max: 12.58 MB
126.96 MB/s from 29 messages
        Message size mean: 12.58 MB min: 12.58 MB max: 12.58 MB
126.67 MB/s from 39 messages
        Message size mean: 12.58 MB min: 12.58 MB max: 12.58 MB
126.50 MB/s from 49 messages
        Message size mean: 12.58 MB min: 12.58 MB max: 12.58 MB

root@core5:/opt/ros_ws# ros2 topic hz /sensor/lidar/top/points
average rate: 10.050
        min: 0.096s max: 0.101s std dev: 0.00133s window: 12
average rate: 10.027
        min: 0.096s max: 0.101s std dev: 0.00114s window: 22
average rate: 10.021
        min: 0.096s max: 0.102s std dev: 0.00102s window: 33
average rate: 10.015
        min: 0.096s max: 0.102s std dev: 0.00095s window: 44
average rate: 10.012
        min: 0.092s max: 0.105s std dev: 0.00168s window: 54

Now, lest try to run all the sensors and see if the issue happens again ...

All good?? I'm even maximising the remote PC bandwith (~120 MB/s) by visualising the side lidars as well and the ouster points looks stable, for now...

What causes this issue in the first place????

Maybe nex time this problem happens, we can use Ouster's visualisation UI. We can techically run this from an remote machine that is connected to the network switch and an IP has been assigned by the server DHCP4