Skip to content

Multi-threaded raycast crashes the process #11

@SwiftGust

Description

@SwiftGust

Hi,
I'm run cow_and_lady dataset and trying to use multi-threaded raycasting.
However, process keep crashes if ray_cast_num_thread param is any value positive integer.

In this log, ray_cast_num_thread was set to 6 but any positive value crashes the process.

Maximum possible thread is 24 for my desktop, fiesta is built with debug mode.

Following in Fiesta.h is modified slightly for output stream

template<class DepthMsgType, class PoseMsgType>
void Fiesta<DepthMsgType, PoseMsgType>::RaycastMultithread() {
     // TODO: when using vector, this is not needed
#ifdef HASH_TABLE
     set_free_.clear();
       set_occ_.clear();
#endif
     int tt = ++tot_;
     timing::Timer raycastingTimer("raycasting");

     if (parameters_.ray_cast_num_thread_==0) {
          RaycastProcess(0, cloud_.points.size(), tt);
     } else {
          int part = cloud_.points.size()/parameters_.ray_cast_num_thread_;
          
          std::list<std::thread> integration_threads = std::list<std::thread>();
          for (size_t i = 0; i < parameters_.ray_cast_num_thread_; ++i) {
               ROS_WARN_STREAM("part : " << part << " thread count : " << i);
               integration_threads.emplace_back(&Fiesta::RaycastProcess, this, i, part, tt);
          }
          for (std::thread &thread : integration_threads) {
               thread.join();
          }
     }
     raycastingTimer.Stop();
}

console output from cow & lady launched with GDB

$ roslaunch fiesta cow_and_lady.launch 
... logging to /home/seunghwan/.ros/log/e218fa0a-4c77-11ea-a841-00d8617c19f3/roslaunch-seunghwan-ryzen-desktop-14727.log
Checking log directory for disk usage. This may take awhile.
Press Ctrl-C to interrupt
Done checking log file disk usage. Usage is <1GB.

RLException: Invalid roslaunch XML syntax: not well-formed (invalid token): line 3, column 20
The traceback for the exception was written to the log file
seunghwan@seunghwan-ryzen-desktop:~/fast_ws/src/FIESTA$ roslaunch fiesta cow_and_lady.launch 
... logging to /home/seunghwan/.ros/log/f0c12334-4c77-11ea-a841-00d8617c19f3/roslaunch-seunghwan-ryzen-desktop-14761.log
Checking log directory for disk usage. This may take awhile.
Press Ctrl-C to interrupt
Done checking log file disk usage. Usage is <1GB.

started roslaunch server http://seunghwan-ryzen-desktop:41235/

SUMMARY
========

PARAMETERS
 * /fiesta/depth_filter_margin: 0
 * /fiesta/depth_filter_max_dist: 10.0
 * /fiesta/depth_filter_min_dist: 0.1
 * /fiesta/depth_filter_tolerance: 0.1
 * /fiesta/global_map: True
 * /fiesta/global_update: True
 * /fiesta/global_vis: True
 * /fiesta/lx: -10.0
 * /fiesta/ly: -10.0
 * /fiesta/lz: -1
 * /fiesta/max_ray_length: 5.0
 * /fiesta/min_ray_length: 0.5
 * /fiesta/p_hit: 0.7
 * /fiesta/p_max: 0.97
 * /fiesta/p_min: 0.12
 * /fiesta/p_miss: 0.35
 * /fiesta/p_occ: 0.8
 * /fiesta/radius_x: 3.0
 * /fiesta/radius_y: 3.0
 * /fiesta/radius_z: 1.5
 * /fiesta/ray_cast_num_thread: 6
 * /fiesta/reserved_size: 1000000
 * /fiesta/resolution: 0.05
 * /fiesta/rx: 10.0
 * /fiesta/ry: 10.0
 * /fiesta/rz: 3
 * /fiesta/slice_vis_level: 1.6
 * /fiesta/slice_vis_max_dist: 2.0
 * /fiesta/update_esdf_every_n_sec: 0.1
 * /fiesta/use_depth_filter: True
 * /fiesta/vis_lower_bound: 0
 * /fiesta/vis_upper_bound: 10
 * /fiesta/visualize_every_n_updates: 10
 * /rosdistro: melodic
 * /rosversion: 1.14.3

NODES
  /
    fiesta (fiesta/test_fiesta)
    rvizvisualisation (rviz/rviz)

auto-starting new master
process[master]: started with pid [14772]
ROS_MASTER_URI=http://localhost:11311

setting /run_id to f0c12334-4c77-11ea-a841-00d8617c19f3
process[rosout-1]: started with pid [14786]
started core service [/rosout]
process[fiesta-2]: started with pid [14793]
process[rvizvisualisation-3]: started with pid [14794]
GNU gdb (Ubuntu 8.1-0ubuntu3.2) 8.1.0.20180409-git
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/seunghwan/fast_ws/devel/lib/fiesta/test_fiesta...done.
Starting program: /home/seunghwan/fast_ws/devel/lib/fiesta/test_fiesta \~depth:=/camera/depth_registered/points \~transform:=/kinect/vrpn_client/estimated_transform __name:=fiesta __log:=/home/seunghwan/.ros/log/f0c12334-4c77-11ea-a841-00d8617c19f3/fiesta-2.log
[tcsetpgrp failed in terminal_inferior: Inappropriate ioctl for device]
[tcsetpgrp failed in terminal_inferior: Inappropriate ioctl for device]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[tcsetpgrp failed in terminal_inferior: Inappropriate ioctl for device]
[tcsetpgrp failed in terminal_inferior: Inappropriate ioctl for device]
[New Thread 0x7fffefb20700 (LWP 14811)]
[New Thread 0x7fffef31f700 (LWP 14812)]
[New Thread 0x7fffeeb1e700 (LWP 14813)]
[New Thread 0x7fffee31d700 (LWP 14814)]
12800000
[New Thread 0x7fffedb1c700 (LWP 14841)]
Pointcloud Size:	307200
[ WARN] [1581388869.453713184]: part : 51200 thread count : 0
[New Thread 0x7fffed31b700 (LWP 14856)]
[ WARN] [1581388869.454807722]: part : 51200 thread count : 1
[New Thread 0x7fffecb1a700 (LWP 14857)]
[ WARN] [1581388869.454914913]: part : 51200 thread count : 2
[New Thread 0x7fffe4f2a700 (LWP 14858)]
[ WARN] [1581388869.455101774]: part : 51200 thread count : 3
[New Thread 0x7fffcaa30700 (LWP 14859)]

Thread 8 "test_fiesta" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffecb1a700 (LWP 14857)]
std::deque<fiesta::ESDFMap::QueueElement, std::allocator<fiesta::ESDFMap::QueueElement> >::emplace_back<fiesta::ESDFMap::QueueElement> (
    this=this@entry=0x5555557ed220, __args#0=...) at /usr/include/c++/7/bits/deque.tcc:167
167		    _Alloc_traits::construct(this->_M_impl,
(gdb) trace
Tracepoint 1 at 0x55555557fd8f: file /usr/include/eigen3/Eigen/src/Core/DenseStorage.h, line 194.

slightly modified cow_and_lady.launch

<launch>
    <node pkg="fiesta" type="test_fiesta" name="fiesta" output="screen" launch-prefix="gdb -ex run --args"
          required="true">

        <param name="resolution" value="0.05"/>
        <param name="update_esdf_every_n_sec" value="0.1"/>

        <!--Hash Table only-->
        <param name="reserved_size" value="1000000"/>

        <!--Array only-->
        <param name="lx" value="-10.0"/>
        <param name="ly" value="-10.0"/>
        <param name="lz" value="-1"/>
        <param name="rx" value="+10.0"/>
        <param name="ry" value="+10.0"/>
        <param name="rz" value="+3"/>

        <!-- raycasting parameters -->
        <param name="min_ray_length" value="0.5"/>
        <param name="max_ray_length" value="5.0"/>
        <!--If hash table is used, make sure the value is 0. In array implementation, 0 means no other thread.-->
        <param name="ray_cast_num_thread" value="6"/>



        <!-- needed when using depth image-->
        <!--        <param name="center_x" value="323.120483"/>-->
        <!--        <param name="center_y" value="236.743210"/>-->
        <!--        <param name="focal_x" value="385.754486"/>-->
        <!--        <param name="focal_y" value="385.754486"/>-->

        <!-- probabilistic grid map -->
        <param name="p_hit" value="0.70"/>
        <param name="p_miss" value="0.35"/>
        <param name="p_min" value="0.12"/>
        <param name="p_max" value="0.97"/>
        <param name="p_occ" value="0.80"/>

        <!-- global / local -->
        <param name="global_map" value="true"/>
        <param name="global_update" value="true"/>
        <param name="global_vis" value="true"/>
        <param name="radius_x" value="3.0"/>
        <param name="radius_y" value="3.0"/>
        <param name="radius_z" value="1.5"/>

        <!--depth_filter -->
        <param name="use_depth_filter" value="true"/>
        <param name="depth_filter_tolerance" value="0.1"/>
        <param name="depth_filter_max_dist" value="10.0"/>
        <param name="depth_filter_min_dist" value="0.1"/>
        <!--unit: pixel-->
        <param name="depth_filter_margin" value="0"/>

        <!-- visulization -->
        <!--0 for no visulize-->
        <param name="visualize_every_n_updates" value="10"/>
        <param name="slice_vis_max_dist" value="2.0"/>
        <!-- relative to the lz if array is used, relative to the origin if hash table is used -->
        <param name="slice_vis_level" value="1.6"/>
        <param name="vis_lower_bound" value="0"/>
        <param name="vis_upper_bound" value="+10"/>

        <!-- subsribe source -->
        <remap from="~depth" to="/camera/depth_registered/points"/>
        <remap from="~transform" to="/kinect/vrpn_client/estimated_transform"/>
    </node>

    <node name="rvizvisualisation" pkg="rviz" type="rviz" output="log" args="-d $(find fiesta)/demo.rviz" />
</launch>

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions