Coredump on exit with Pure Virtual Call in GazeboRosVideo

[rethink-imcmahon]

This is a tough issue to know where to post since it touches a few different repositories. baxter_simulator users always experience a coredump on exit (if they set ulimit to allow coredumps). The output looks something like this:

[gazebo-2] killing on exit
pure virtual method called
terminate called without an active exception
pure virtual method called
terminate called without an active exception
Aborted (core dumped)

Looking into the backtrace itself, I see gzclient, GazeboRosVideo and roscpp all have a hand in this. Also, the double pure virtual method call makes sense as the left and right arm simulated cameras are being destroyed. Here's the backtrace with as many debug symbols as I could get my hands on:

Core was generated by `gzclient __name:=gazebo_gui __log:=/data/users/imcmahon/.ros/log/9cd40d06-4d97-'.
Program terminated with signal SIGABRT, Aborted.
#0  0x00007f23c8298cc9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56  ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) set debug-file-directoy /usr/lib/debug/usr/bin/gzclient-2.2.3
No symbol "debug" in current context.
(gdb) bt full
#0  0x00007f23c8298cc9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
        resultvar = 0
        pid = 19889
        selftid = 19889
#1  0x00007f23c829c0d8 in __GI_abort () at abort.c:89
        save_stage = 2
        act = {__sigaction_handler = {sa_handler = 0x628ff48, sa_sigaction = 0x628ff48}, sa_mask = {__val = {
              103349392, 88715840, 139791731832151, 139788300582917, 0, 29, 139791658552616, 88701760, 88715840, 
              88715856, 139791731860757, 29, 139791659489421, 29, 139791668616960, 2}}, sa_flags = -933090864, 
          sa_restorer = 0x7f23c8c01fa6}
        sigs = {__val = {32, 0 <repeats 15 times>}}
#2  0x00007f23c8ba3645 in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#3  0x00007f23c8ba1836 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#4  0x00007f23c8ba1863 in std::terminate() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#5  0x00007f23c8ba233f in __cxa_pure_virtual () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#6  0x00007f232b514905 in ros::Subscription::removeCallback(boost::shared_ptr<ros::SubscriptionCallbackHelper> const&)
    () from /opt/ros/indigo/lib/libroscpp.so
No symbol table info available.
#7  0x00007f232b4c434a in ros::TopicManager::unsubscribe(std::string const&, boost::shared_ptr<ros::SubscriptionCallbackHelper> const&) () from /opt/ros/indigo/lib/libroscpp.so
No symbol table info available.
#8  0x00007f232b497e58 in ros::Subscriber::Impl::unsubscribe() () from /opt/ros/indigo/lib/libroscpp.so
No symbol table info available.
#9  0x00007f232b497ee3 in ros::Subscriber::Impl::~Impl() () from /opt/ros/indigo/lib/libroscpp.so
No symbol table info available.
#10 0x00007f232b498932 in boost::detail::sp_counted_impl_p<ros::Subscriber::Impl>::dispose() ()
   from /opt/ros/indigo/lib/libroscpp.so
No symbol table info available.
#11 0x00000000004975fe in boost::detail::sp_counted_base::release (this=0x53478e0)
    at /usr/include/boost/smart_ptr/detail/sp_counted_base_gcc_x86.hpp:146
No locals.
#12 0x00007f232c48ae8d in gazebo::GazeboRosVideo::~GazeboRosVideo() () from /opt/ros/indigo/lib/libgazebo_ros_video.so
No symbol table info available.
#13 0x00007f232c48b019 in gazebo::GazeboRosVideo::~GazeboRosVideo() () from /opt/ros/indigo/lib/libgazebo_ros_video.so
No symbol table info available.
#14 0x00000000004975fe in boost::detail::sp_counted_base::release (this=0x29b9590)
    at /usr/include/boost/smart_ptr/detail/sp_counted_base_gcc_x86.hpp:146
No locals.
#15 0x00007f23cc0763d4 in ~shared_count (this=0x548d028, __in_chrg=<optimized out>)
    at /usr/include/boost/smart_ptr/detail/shared_count.hpp:371
No locals.
#16 ~shared_ptr (this=0x548d020, __in_chrg=<optimized out>) at /usr/include/boost/smart_ptr/shared_ptr.hpp:328
No locals.
#17 _Destroy<boost::shared_ptr<gazebo::VisualPlugin> > (__pointer=0x548d020)
    at /usr/include/c++/4.8/bits/stl_construct.h:93
No locals.
#18 __destroy<boost::shared_ptr<gazebo::VisualPlugin>*> (__last=<optimized out>, __first=0x548d020)
    at /usr/include/c++/4.8/bits/stl_construct.h:103
No locals.
#19 _Destroy<boost::shared_ptr<gazebo::VisualPlugin>*> (__last=<optimized out>, __first=<optimized out>)
    at /usr/include/c++/4.8/bits/stl_construct.h:126
No locals.
#20 _Destroy<boost::shared_ptr<gazebo::VisualPlugin>*, boost::shared_ptr<gazebo::VisualPlugin> > (__last=0x548d030, 
    __first=0x548d020) at /usr/include/c++/4.8/bits/stl_construct.h:151
No locals.
#21 _M_erase_at_end (this=<optimized out>, __pos=0x548d020) at /usr/include/c++/4.8/bits/stl_vector.h:1352
#22 clear (this=0x57ca508) at /usr/include/c++/4.8/bits/stl_vector.h:1126
No locals.
#23 gazebo::rendering::Visual::Fini (this=0x57ca4f0) at /tmp/buildd/gazebo2-2.2.3/gazebo/rendering/Visual.cc:160
        iter = <optimized out>
#24 0x00007f23cc0447fb in gazebo::rendering::Scene::RemoveVisual (this=this@entry=0x2ff6f30, _vis=...)
    at /tmp/buildd/gazebo2-2.2.3/gazebo/rendering/Scene.cc:2699
        piter = <optimized out>
        iter = {_M_node = 0x549b850}
#25 0x00007f23cc0493d0 in gazebo::rendering::Scene::Clear (this=0x2ff6f30)
    at /tmp/buildd/gazebo2-2.2.3/gazebo/rendering/Scene.cc:189
No locals.
#26 0x00007f23cc0314e8 in gazebo::rendering::RenderEngine::RemoveScene (
    this=this@entry=0x8ca5c0 <SingletonT<gazebo::rendering::RenderEngine>::GetInstance()::t>, _name=...)
    at /tmp/buildd/gazebo2-2.2.3/gazebo/rendering/RenderEngine.cc:218
        iter = {_M_current = 0x4bbae20}
#27 0x00007f23cc031697 in gazebo::rendering::RenderEngine::Fini (
    this=this@entry=0x8ca5c0 <SingletonT<gazebo::rendering::RenderEngine>::GetInstance()::t>)
    at /tmp/buildd/gazebo2-2.2.3/gazebo/rendering/RenderEngine.cc:326
No locals.
#28 0x00007f23cc0373e1 in gazebo::rendering::fini ()
    at /tmp/buildd/gazebo2-2.2.3/gazebo/rendering/RenderingIface.cc:66
No locals.
#29 0x00007f23c312ac2e in gazebo::sensors::fini () at /tmp/buildd/gazebo2-2.2.3/gazebo/sensors/SensorsIface.cc:71
No locals.
#30 0x00007f23cb2c4fd1 in gazebo::shutdown () at /tmp/buildd/gazebo2-2.2.3/gazebo/gazebo.cc:248
        lock = {m = 0x7f23cb4dae80 <fini_mutex>, is_locked = true}
#31 0x00000000004aea89 in gazebo::gui::stop () at /tmp/buildd/gazebo2-2.2.3/gazebo/gui/GuiIface.cc:220
No locals.
#32 0x00000000004aeac9 in signal_handler () at /tmp/buildd/gazebo2-2.2.3/gazebo/gui/GuiIface.cc:63
No locals.
#33 <signal handler called>
No locals.
#34 0x00007f23c834f12d in poll () at ../sysdeps/unix/syscall-template.S:81
No locals.
#35 0x00007f23c29a1fe4 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
No symbol table info available.
#36 0x00007f23c29a20ec in g_main_context_iteration () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
No symbol table info available.
#37 0x00007f23ca2947a1 in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) ()
   from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
No symbol table info available.
#38 0x00007f23ca83abe6 in ?? () from /usr/lib/x86_64-linux-gnu/libQtGui.so.4
No symbol table info available.
#39 0x00007f23ca2660af in QEventLoop::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) ()
   from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
No symbol table info available.
#40 0x00007f23ca2663a5 in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) ()
   from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
No symbol table info available.
#41 0x00007f23ca26bb79 in QCoreApplication::exec() () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
No symbol table info available.
#42 0x00000000004af9d1 in gazebo::gui::run (_argc=5, _argv=0x7ffeb075f5b8)
    at /tmp/buildd/gazebo2-2.2.3/gazebo/gui/GuiIface.cc:210
        sigact = {__sigaction_handler = {sa_handler = 0x4aeac0 <signal_handler(int)>, 
            sa_sigaction = 0x4aeac0 <signal_handler(int)>}, sa_mask = {__val = {139791668603694, 5718847, 9222256, 
              140731858941055, 5282480, 9213480, 139791658771673, 140731858941055, 45, 9154016, 139791668287040, 
              9213480, 139791658771673, 44, 45, 9154016}}, sa_flags = 4804503, sa_restorer = 0x8bade0}
#43 0x0000000000494699 in main (_argc=<optimized out>, _argv=<optimized out>)
    at /tmp/buildd/gazebo2-2.2.3/gazebo/gui/main.cc:23
        result = 0

For reference, this is ticket #41 in baxter_simulator

[hsu]

How does one reproduce the issue for debugging purposes? thanks.

[rethink-imcmahon]

Right, that is a fundamental piece of information in every bug report. Sorry!

My test setup: Ubuntu 14.04 Trusty, ROS Indigo, Gazebo 2.2 (though I suspect this is an issue in ROS Jade and Gazebo 5.0 as well). Also, the baxter_simulator set of repositories. The simulator itself is a private repo until the next release, but I've added you as a member. If anyone else is interested in debugging this issue, just let me know here and I will add them as well. The instructions for installing it can be found here: http://sdk.rethinkrobotics.com/wiki/Simulator_Installation Any branch of the simulator will do.

To reproduce, after enabling coredumps in your terminal with ulimit -c unlimited and sourcing your ROS environment:
roslaunch baxter_gazebo baxter_world.launch
ctrl+c in the terminal after the simulation is done loading. Then as the Gazebo ROS video streams are being destroyed, you should see at least one coredump in your ~/.ros/ directory.

[davetcoleman]

@hsu any progress on this issue? I'm experiencing it in Gazebo7 as well

[davetcoleman]

I've created a branch of the RRBot example that includes a GazeboROSVideo plugin and demonstrates the bug in a simple use case.

I have a fix for the Video bug here: #547