ROS2 调试助手 StvLi/dsh-ros2

dsh-ros2

面向 DeepSeek Harness（DSH）的 ROS2 调试工具集与机器人状态视觉分析，以插件形式发布。中文版见 README_CN.md。

版本对应关系：npm 上的 dsh-ros2@0.1.0 就是本仓库当前版本（monorepo 布局）。
GitHub 的 v0.8.0 ~ v0.15.0 标签是已废弃的旧单体布局历史，从未发布到 npm。
版本号在 2026-08 monorepo 拆分时重新基线。详见 docs/versioning.md。

dsh-ros2 让 DSH 智能体在任何装了 ROS2 的主机上具备完整的机器人开发 / 调试能力，按四个能力层级组织：

| 层级 | 能力 | 安全边界 |
| L1 | 只读诊断：包 / 工作区 / 依赖检查，节点 / 话题 / 服务 / 动作 / 参数 / 接口枚举，话题单次采样，TF 树查询，全图拓扑 JSON，ros2doctor，bag 摘要，MoveIt 发现，机器人配置档加载，安全状态读取与 VLM 语义仲裁 | 纯只读，无需审批 |
| L2 | 审批门控的管理操作：colcon build（后台任务）、rosdep install、消息骨架生成、param set、有界 bag record、一键安装 ROS2、launch 管理、rosbag 回放、MoveIt 运动、零位姿标定、机器人注册与拓扑学习、safety_monitor 启动 / 人工门控锁定 / 解锁 | 写操作一律先询问（fail-closed） |
| L3 | 可视化：RViz2 / rqt 生命周期管理、截图、多模态视觉描述、xdotool 级窗口交互 | 本地会话操作；交互限定在应用允许列表内，超出范围先询问 |
| L4 | 实时视觉：并行 VLM ROS2 节点 + 图像话题采集（无头运行；vision_bringup 按话题周期性刷新桥接），外加 RViz2 离屏渲染（OGRE 内核 → /rviz/scene 话题） | 纯软件 / GPU 渲染，无需显示器 |

所有工具都在主机上直接执行 ros2 / colcon / rosdep 命令行命令；L1 从不改动任何东西，L2 一律先询问。

截图

| RViz2 离屏渲染（最新 lite_urdf，真实材质颜色） | 头部相机 | 左腕相机 | 右腕相机 |
| mesh render | head cam | wrist left | wrist right |
左：rviz_offscreen_node 用真实 rviz 栈（OGRE）渲染，把画面发布到 /rviz/scene 图像话题。右：ros2_image_snapshot 从实时相机话题抓取的三帧画面（1280×720）。完整测试记录见 docs/test-robot-state-vision.md。

功能

- 零侵入诊断：83 个工具覆盖大多数 ROS2 调试场景——从「包装了吗？」到「这个话题上现在有什么？」，一条命令，一个答案；
- 全图拓扑：ros2_graph 把节点、发布者、订阅者、服务、动作折叠成一份 JSON——几秒看清系统结构；
- 写入需审批：构建、安装依赖、生成消息骨架等操作都走 DSH 审批服务；失败即关闭，拒绝即失败；
- 可视化即服务：无头「看」——截图、多模态描述、窗口交互全部在本地完成，不依赖远程显示；
- 并行实时视觉：VLM 跑在独立的 ROS2 进程里（vlm_node，服务 /vlm/describe）；图像来自话题（sensor_msgs/Image / CompressedImage）；vision_bringup 为每个图像话题自动创建一个桥接，开箱即用无头模式；
- RViz2 离屏渲染（llvmpipe 上动态渲染约 22 Hz；有 GPU 时满帧 30 Hz）：真实 rviz 渲染内核（rviz_common + OGRE）在虚拟显示上渲染任意 .rviz 场景，并作为图像话题发布——不截图，不依赖 X11 窗口叠加。性能优化：open3d 低模网格（scripts/simplify_visual_meshes.py）+ 直接读取 OGRE 像素（省去 PNG 往返）+ 消除重复渲染 → llvmpipe 上动态渲染从 1.9 Hz 提升到约 22 Hz（11 倍），NVIDIA GPU 直通时满帧 30 Hz（v0.9.3），内存降低 2.5 倍；
- 实时安全框架（safety_monitor 节点 + robot_safety_ 工具，见安全框架）：分层防御——工具层安全门（执行前检查 /safety/state）→ 反应式监控（带迟滞的运动跟踪/停滞、关节反馈丢失、看门狗、可选力矩）→ 事件驱动的 VLM 语义仲裁 → 人工仲裁。锁存的 NORMAL/LOCKED 状态机（锁定后持续生效，直到人工解锁；非致命事件永不锁定）；每个阈值、话题、锁定动作都按机器人登记在配置档的 safety 段；几何预检（关节限位、速度、FK 自碰撞）是预留层；
- 内置技能（9 个）：每条高频路径配一个载体 —— ros2-diagnostics（何时用哪个工具、缩小范围的排查方法）、ros2-bringup-recovery（环境 → 启动 → 验证）、ros2-liveness-triage（一两次调用拿到频率和载荷）、ros2-tf-integrity（边、查询、缺失的广播者）、robot-registration / robot-retrieval（本体配置档与即时复用）、robot-state-vision-analysis（状态 → 渲染 → VLM → 交叉核对）、robot-motion-control（唯一一条 规划 → 校验 → 批准 → 执行 → 验证 路径）和 robot-safety-procedure（锁存、LOCKED、边界）。

快速开始

环境要求

- 一台装了 ROS2 的主机（Jazzy 已验证；Humble 应该也能用），ros2 在 PATH 里；
- Node ^22.19 || >=24（DSH 宿主的要求）；
- L4 视觉还需要 Python 3 的 rclpy，以及一个兼容 OpenAI 的 VLM 网关（比如 Gemini，或自建网关）。

安装插件（monorepo 拆分后共 9 个 npm 包）

装你需要的领域包 —— 或者装 dsh-ros2 聚合包，它挂载 core/profile/moveit/safety/vision（81 个工具 + 9 个技能）。想要这一族完整的 83 个工具，再补上可选的 dsh-ros2-state。所有包都已发布到 npm，版本 0.1.0（GitHub ↔ npm 的版本对应关系见 docs/versioning.md）。

用 DSH 插件命令行（推荐；走 npm 解析）：

full set (aggregate; pulls core/profile/moveit/safety/vision as deps)
dsh plugin --profile <profile> add dsh-ros2
or lean installs — e.g. diagnostics only:
dsh plugin --profile <profile> add dsh-ros2-core
dsh-ros2-common is a plain library, auto-installed as a dependency.

直接用 npm（同样的包；适合查看/安装可选附加包）：

the aggregate (mounts dsh-ros2-common/core/profile/moveit/safety/vision = 81 tools)
npm install dsh-ros2
optional add-ons — the sidecar data plane + its control-plane state client. dsh-ros2-state is NOT part of the aggregate's patch; its own 2 tools (state_get / state_snapshot) are what take the family from 81 to 83:
npm install dsh-ros2-state dsh-ros2-sidecar

包的分工：dsh-ros2（聚合包）· dsh-ros2-common（共享库）· dsh-ros2-core（核心诊断）· dsh-ros2-profile（配置档/拓扑）· dsh-ros2-moveit（MoveIt 运动）· dsh-ros2-safety（安全框架）· dsh-ros2-vision（视觉流水线）· dsh-ros2-state（状态客户端）· dsh-ros2-sidecar（数据面守护进程）。

提示词引导。 dsh-ros2-core 会贡献一段系统提示词，指引模型做 ROS2 相关工作时优先用这套工具链 —— 用 ros2_ 摸清计算图，用 robot_load/robot_topology 加载本体，通过 moveit_/robot_safety_ 规划并执行动作（需审批，且受 /safety/state 门控）—— 而不是临时敲 ros2 命令行或写私有的 rclpy 脚本。所以任何包含 core 的安装都会带上这段提示词，dsh-ros2 聚合包也不例外。
每次装配时，这一节都会根据当时已注册的工具重新生成，所以只装核心包时，不会被告知去调用它并未随附的 moveit_move 或 robot_safety_。它的注册与插件自身的生命周期绑定：禁用或移除插件，它就会从提示词中消失，因此卸载 dsh-ros2 后不会留下残留。

最小配置（按 bundle 配置，整对象替换）

拆分后，每个 bundle 各自携带自己的 run-seam 配置（相同的键按 id 重复出现），vision 服务商只存在于 dsh-ros2-vision 上：

fragment of a DSH profile patch (id-targeted config overrides)
- id: dsh-ros2-core # also: -profile / -moveit / -safety
config:
rosSetup: source /opt/ros/jazzy/setup.bash && # prepare the ROS2 environment
workspaceRoot: /home/you/ros2_ws # default cwd for colcon/rosdep
- id: dsh-ros2-vision
config:
rosSetup: source /opt/ros/jazzy/setup.bash &&
vision:
provider: gemini # mock | gemini | openai
apiKey: ${VLM_API_KEY} # ${ENV} 引用从环境变量解析，勿明文写 key
- id: dsh-ros2-safety
config:
safetyStrict: warn # 'warn' (default) | 'reject' (fail-closed); LOCKED always rejects

三分钟上手

ros2_topology --tf # WHOLE system (nodes/topics/services/actions/TF) in one call
ros2_graph # whole system topology in one shot
ros2_topic_list # all topics and types
ros2_topic_echo /joint_states # sample one frame of joint states
ros2_tf_list # TF tree edges
ros2_doctor # system health report

工具参考

工具分放在各领域包中：L1/L2/L3 → dsh-ros2-core；motion_validate/moveit_ → dsh-ros2-moveit；robot_/ros2_zero_pose_semantics → dsh-ros2-profile；robot_safety_ → dsh-ros2-safety；vision 工具 → dsh-ros2-vision。

L1 只读诊断

| 工具 | 底层命令 | 用途 |
| ros2_pkg_list | ros2 pkg list | 已安装的包（可选子串过滤） |
| ros2_pkg_prefix / ros2_pkg_executables | ros2 pkg prefix <pkg> / ros2 pkg executables [pkg] | 包的安装前缀 / 可执行文件（结构化输出） |
| ros2_colcon_list | colcon list | colcon 工作区中的包 |
| ros2_rosdep_check | rosdep check --from-paths src --ignore-src | 依赖健康状况（缺依赖算发现项，不算错误） |
| ros2_node_list | ros2 node list | 运行中的节点 |
| ros2_node_info | ros2 node info <node> [-v] | 单个节点的订阅者 / 发布者 / 服务 / 动作 |
| ros2_topic_list | ros2 topic list -t | 话题及其类型 |
| ros2_topic_find | ros2 topic find <type> | 承载某消息类型的话题 |
| ros2_topic_info | ros2 topic info <topic> [-v] | 话题元数据 / QoS |
| ros2_topic_echo | ros2 topic echo <topic> --once | 采样一条消息（能转 JSON 就转）；--qos-reliability / --qos-durability 覆盖参数可触达 TRANSIENT_LOCAL 锁存话题 |
| ros2_topic_hz | ros2 topic hz <topic> [--window N] | 测量发布频率（窗口内的平均/最小/最大/标准差）；自然终止即测量超时 |
| ros2_env_check | 解析后的 setup 探测 | 自检环境：当前 source 的是哪个 setup、路径是否存在、可见的包/节点 |
| ros2_workspace | source <ws>/install/setup.bash && | 在会话内切换/查看工作空间（use/show/reset）——不改配置，不重启。use 需要审批：source 文件会执行它，覆盖项随后作用于之后每条命令 |
| ros2_topic_bw / ros2_topic_delay | ros2 topic bw|delay <topic> | 测量话题带宽 / 端到端延迟；超时终止 |
| ros2_service_list | ros2 service list -t | 服务及其类型 |
| ros2_service_type / ros2_service_find | ros2 service type|find ... | 服务类型 / 按类型查找服务 |
| ros2_action_list | ros2 action list -t | 动作及其类型 |
| ros2_action_info | ros2 action info <action> | 动作类型与状态 |
| ros2_action_type | ros2 action type <action> | 动作类型 |
| ros2_param_list | ros2 param list <node> | 节点的参数 |
| ros2_param_get | ros2 param get <node> | 读取单个参数值 |
| ros2_param_dump | ros2 param dump <node> | 导出节点的全部参数 |
| ros2_interface_show | ros2 interface show <type> | 消息/服务/动作的完整字段定义 |
| ros2_interface_list / ros2_interface_prototype / ros2_interface_package | ros2 interface list|prototype|package ... | 全部接口类型 / 默认值原型 / 某个包内的类型 |
| ros2_graph | ros2 node list + 逐节点 node info | 折叠后的 JSON 拓扑图 |
| ros2_topology | 单个 rclpy 进程（不逐条 spawn 命令行） | 一次调用拿到全系统快照：节点及其 pub/sub/services，话题及其类型与 pub/sub 数量，服务、动作，以及可选的 TF 帧和参数。优先用它，而不是多次窄范围调用 |
| ros2_tf_list | 同一个 rclpy 快照 | 来自 /tf 和锁存的 /tf_static 的 TF 边（动态帧和静态帧都有） |
| ros2_tf_echo | 同一个 rclpy 快照 | 两个帧之间的变换，包含反向边 |
| ros2_doctor | ros2 doctor | 系统健康报告 |
| ros2_bag_info | ros2 bag info <path> | bag 摘要 |
| moveit_discover | 扫描 MoveIt 包 + 解析 SRDF + 探测 move_group | 发现 MoveIt2 配置包（任何带 SRDF 的包）、其规划组与命名位姿，以及 /move_action / /execute_trajectory / /compute_cartesian_path 是否在线。传 srdf 可直接解析指定文件——通用，不绑定某个包 |
| robot_safety_state | ros2 topic echo /safety/state --once | 读取锁存的安全状态（NORMAL / LOCKED + 严重级别 + 触发原因 + 详情）；监控离线时报告 monitor_running: false |
| robot_safety_arbitrate | ros2 run dsh_ros2_safety safety_vlm_arbitrate ... | 事件驱动的 VLM 语义仲裁（计划变更 / 异常后）：固定格式提示词 + 通过 /vlm/describe 获取最新离屏帧；任何非安全判定都会标记为需人工仲裁 |
| motion_validate | motion_validator.py --trajectory <file> --config <json> | 确定性的执行前校验（只读）：关节限位、NaN/Inf、名称与组覆盖、时间戳/时长、新鲜度、可选工作空间盒、指纹 + TTL —— 碰撞/奇异仍交给 MoveIt 规划 |

L2 管理（需审批）

每个 L2 工具都执行写操作，先通过 DSH 审批服务询问用户（服务不可用或拒绝时按失败关闭处理）。只读辅助工具 ros2_jobs_list / ros2_job_status 无需审批。

| 工具 | 背后的命令 | 说明 |
| ros2_colcon_build | colcon build [--packages-select ...] [--symlink-install] | 作为后台任务运行（ctx.jobs）；返回 jobId，用 ros2_job_status 跟踪 |
| ros2_rosdep_install | rosdep install --from-paths src --ignore-src -y | dryRun 用 --simulate 预览 |
| ros2_interface_create | 写入 <root>/<pkg>/<msg\|srv\|action>/<Name>. | 骨架生成器；绝不覆盖已有文件 |
| ros2_param_set | ros2 param set <node> <value> | JSON 数字/布尔按类型处理，其他值按字符串处理 |
| ros2_bag_record | ros2 bag record <topics...> --output <dir> | 限时录制：duration 秒后自动停止 |
| ros2_jobs_list | ctx.jobs.list | 本智能体的后台任务（只读） |
| ros2_job_status | ctx.jobs.get | 按 id 查询单个任务的状态（只读） |
| ros2_install | FishROS 一键安装器（交互式 PTY 会话） | ROS2 缺失时：check 探测（已安装 / 已安装未 source / 不存在）；start（需审批）启动安装器；send / status / stop 驱动并观察交互菜单 |
| ros2_bag_play | ros2 bag play <path> [--topics ...] [--rate X] [--loop] [--start-offset S] | 把 rosbag 回放到对应话题（需审批；向计算图发布消息）；前台运行 timeoutMs |
| ros2_topic_pub | ros2 topic pub <topic> <type> "<yaml>" [-r Hz] [-n N\|--once\|-t sec] | 发布消息（需审批；会改动计算图）。限次发布 + QoS 覆盖（--qos-reliability / --qos-durability） |
| ros2_run | ros2 run <pkg> <executable> [args] | 运行任意已安装的 ROS2 可执行文件（需审批）：前台（限时）或后台任务 |
| ros2_process_cleanup | pgrep -f '[p]attern' + kill | 杀掉匹配模式的残留 ROS2 进程（自身安全）；需审批 |
| ros2_service_call | ros2 service call <svc> <type> "<yaml>" | 调用服务（需审批）；从 repr 解析响应 |
| ros2_action_send_goal | ros2 action send_goal <action> <type> "<yaml>" [--feedback] | 发送 action 目标（需审批）；返回 goal id + 状态 |
| ros2_daemon | ros2 daemon status|stop|start | 守护进程状态（L1）/ 停止-启动（L2 审批）—— 重新发现过期的计算图 |
| ros2_param_delete | ros2 param delete <node> | 删除节点参数（需审批） |
| ros2_lifecycle | ros2 lifecycle get|list|set <node> [state] | 生命周期状态获取/列出（L1）或设置（L2 审批） |
| ros2_component | ros2 component list / load <container> <pkg> <type> | 组件容器列出（L1）/ 加载（L2 审批） |
| ros2_launch | ros2 launch <pkg> <launch_file> [args] | 以后台任务方式启动 launch 文件（需审批；返回 jobId，用 DSH 任务控制停止） |
| ros2_zero_pose_semantics | 发布零位 → 离屏渲染 → VLM → 确认 | 交互式校准零位语义（通用）：analyze 渲染全零位姿，让 VLM 从三个方面判断姿态（手臂：lateral_raise/hanging，肘部：forward/upward，手掌/相机安装：up/forward/down）；confirm 把用户认可的组合同（或 customText 自由文本描述）写入 ~/.dsh-ros2/zero-pose.yaml，供技能使用 |
| robot_register | 收集 URDF/TF/cameras/MoveIt/zero-pose → 写入 ~/.dsh-ros2/robots/<name>.yaml | 首次接触时注册机器人本体配置档（需审批），之后可即时复用 |
| robot_load | 读取 ~/.dsh-ros2/robots/<name>.yaml | 以结构化 JSON 加载已注册的机器人配置档（快速路径 —— 不做发现）；名字留空则列出全部配置档 |
| robot_topology | 聚合快照 + 渐进式节点学习（严格 schema） | 机器人通信拓扑的取舍：snapshot（需审批）记录节点/话题/服务列表（轻量，不啰嗦）；learn（需审批）记录一个重要节点的角色/描述 + pub/sub/srv/act；show（只读）读回这些内容 |
| moveit_move | 统一：/move_action + /execute_trajectory | 一个工具，五种核心模式（需审批）：joint_abs、joint_rel、pose_abs、pose_rel（坐标系 ee/world）、trajectory。通用：只用标准 moveit_msgs + SRDF。单一运动路径：规划 → 确定性校验（motion_validator，robot 配置档提供完整限位）→ 人工审批（展示校验摘要）→ 执行 → 验证。以 /safety/state 为闸门（LOCKED 一律拒绝；监控掉线按 safetyStrict 处理） |
| robot_safety_start | ros2 run dsh_ros2_safety safety_monitor --profile <yaml> | 以后台任务方式启动通用安全监控（需审批）；所有机器人相关数值都来自配置档的 safety 段 |
| robot_safety_lock / robot_safety_unlock | ros2 service call /safety/set_lock|unlock ... | 需人工确认的显式加锁 / 解锁（调用前需 L2 审批）；锁会一直保持，直到人工解锁（恢复流程：解锁 → 重新回原点 → 继续运行） |
| moveit_status | 探测 move_group 接口 + 采样 /joint_states | 运行时状态：在线探测 + 当前关节状态 + SRDF 规划坐标系（只读） |

L3 可视化

GUI 生命周期 + 截图 + 多模态视觉（「先看再动」）+ xdotool 级交互（「边看边动」）。截图在 X11 上用 Pillow 的 ImageGrab（无需额外安装命令行工具）；视觉服务商可插拔；交互需要 xdotool（sudo apt install xdotool）。

| 工具 | 用途 |
| ros2_gui_start | 在主机显示器上启动 RViz2（带 -d config）/ rqt_graph / rqt；会话会被跟踪 |
| ros2_gui_list | 已跟踪的会话 + X11 窗口（wmctrl -lG） |
| ros2_gui_close | 关闭会话（SIGTERM） |
| ros2_screenshot | 截取整个屏幕或单个窗口，保存为 PNG |
| ros2_vision_describe | 用配置好的多模态模型描述图片（Gemini / OpenAI / mock）。路径必须是真实的 PNG/JPEG/GIF/WebP —— 上传前会先嗅探内容 —— 且不超过 20 MiB；文件过大或不是普通文件会被拒绝，且不会被读取 |
| ros2_gui_observe | 确保 GUI 在运行 → 截图 → 返回多模态描述（「看到它」工作流） |
| ros2_gui_interact | 统一的 xdotool 交互：action=click（点击/滚动，button 4/5 = 滚动）、action=drag（按下-拖动-释放：RViz2 旋转/平移/缩放）、action=key（组合键如 ctrl+shift+r，或输入文本）。限定在应用允许列表内（默认 rviz2 / rqt / rqt_graph，可通过 guiAllowedApplications 配置）；任何其他目标 —— 包括省略 windowTitle、从而作用于当前焦点窗口的情况 —— 都需要审批 |

交互配方（面向模型）：用 ros2_gui_interact {action: "drag", windowTitle: "rviz2", button: 1, toX: <dx>, toY: <dy>} 旋转 RViz2 视图，用 action: "drag", button: 3 缩放，用 ros2_gui_interact {action: "key", keys: "ctrl+shift+r"} 重新加载显示配置。当 wmctrl 无法枚举窗口时（例如显示器上没有窗口管理器），窗口相对交互会报「window not found」—— 此时改用绝对屏幕坐标，这会请求审批，因为它作用于当前焦点窗口。

为什么要有允许列表：action=key 会向目标窗口输入任意文本，所以不加限制的目标就是主机级输入注入通道（往终端里打字就能执行命令）。允许列表只是便利性控制 —— 任何进程都能改自己的窗口标题，所以标题匹配并不能证明身份；审批关卡才是边界，而且它默认拒绝（没有审批服务 = 拒绝）。见 docs/safety.md。

L4 实时视觉（在 ROS2 上并行跑 VLM，无头图像话题）

感知部分与机器人控制栈保持一致：VLM 跑在独立的 ROS2 进程里（vlm_node，服务 /vlm/describe + 缓存话题 /vlm/description），图像来自 sensor_msgs/Image 话题，绝不用 X11 截图——无头环境也能跑。采集（ros2_image_snapshot）不需要自定义包（纯 rclpy）；只有 ros2_vlm_analyze/ros2_vision_analyze 需要 dsh_ros2_vlm ROS2 包（vlm/）——构建/运行见 docs/architecture.md §4。流水线挂掉时，ros2_vlm_analyze/ros2_vision_analyze 会返回明确的 VLM_UNAVAILABLE + 降级提示，快照 JPEG 可以直接交给智能体自带的多模态模型读取。

| 工具 | 用途 |
| ros2_image_snapshot | 从话题抓一帧（raw/compressed）存成 JPEG——纯 rclpy 脚本，不需要自定义 ROS2 包；话题没数据时用 --v4l 走 ffmpeg 兜底；JPEG 可直接交给智能体自带的多模态模型 |
| ros2_vlm_analyze | 通过并行 VLM 分析图像文件或桥接的最新帧（useBridge） |
| ros2_vision_topics | 列出活跃的图像话题及其自动桥接服务名 |
| ros2_vision_doctor | 一次性流水线自检：vlm 工作空间是否构建 / vlm_node+vision_bringup 是否在跑 / 网关是否可达 / 可见图像话题 / apiKey 解析状态（config/env/secrets/missing）+ 构建启动指引 |
| ros2_vision_set_key | 把用户提供的 VLM API Key 存进 ~/.dsh-ros2/secrets.json（0600，放在仓库外——绝不提交、绝不上传、绝不回显）；工具返回 VLM_API_KEY_REQUIRED 时用它 |
| ros2_vision_analyze | 通过自动桥接分析任意话题的最新帧（ros2_vision_analyze {topic, prompt}） |

build + launch the vision pipeline (auto bridge per image topic)
Build the workspace OUTSIDE /tmp: a reboot wipes /tmp, and a rosSetup that
sources a deleted workspace is the misconfiguration documented in
docs/feedback-env-recovery.md. Use any persistent path (~/vlm_ws here).
mkdir -p ~/vlm_ws/src && ln -s <repo>/vlm ~/vlm_ws/src/dsh_ros2_vlm
cd ~/vlm_ws && colcon build --symlink-install && source install/setup.bash
VLM_API_KEY=... ros2 run dsh_ros2_vlm vlm_node & # parallel VLM process
ros2 run dsh_ros2_vlm vision_bringup & # discover topics, one bridge each

RViz2 离屏渲染（dsh_ros2_rviz_offscreen）

真正的 rviz 渲染栈（rviz_common + OGRE + rviz_default_plugins）在 Xvfb 下离屏加载 .rviz 场景，发布到 /rviz/scene 图像话题——从渲染内核读取，不用 X 截图，不依赖窗口叠放。

build (needs a colcon workspace like vlm_ws — keep it out of /tmp)
ln -s <repo>/offscreen ~/vlm_ws/src/dsh_ros2_rviz_offscreen
cd ~/vlm_ws && colcon build --symlink-install && source install/setup.bash
run (config_path points at a .rviz scene file)
xvfb-run -a -s "-screen 0 1280x800x24" ros2 run dsh_ros2_rviz_offscreen rviz_offscreen_node \
--ros-args -p config_path:=/tmp/robot_scene.rviz -p topic:=/rviz/scene \
-p width:=800 -p height:=600 -p rate:=5.0

机器人本体网格渲染要点（踩坑汇总，细节见 docs/architecture.md §4.4）：

1. Jazzy 的 RobotModel 属性：用 Description Source: Topic + Description Topic: <topic>（旧的 Robot Description: 会被忽略 → Links 为空）；
2. 网格路径：在 URDF 里用绝对路径，或加 file:// 前缀（裸路径在 resource_retriever 里会失败）；发布描述的节点必须一直活着（transient-local，晚到的订阅者否则收不到）；
3. URDF 必须按名字绑定 TF：URDF 里的 link 名必须和实时 TF 帧名完全一致（直接发布机器人真实的 /robot_description 就行）。名字对不上 → 每个 link 的变换查找都失败 → 所有网格堆在固定帧原点；
4. 视距：Orbit 的 Distance 取 1.5–2.0 m，接近 RViz 的近距离全身视角（> 5 m 机器人会缩成中心一个小点）；
5. 健康信号：启动约 3 s 后节点会打印 FM: ... frames=N 和 transformHasProblems(...)=0 —— 说明网格已正确绑定 TF。

MoveIt2 运动 —— 五种基本模式，一个工具

设计意图。 用 MoveIt 让机器人运动，本质上只有五种操作：绝对设置关节、相对微调关节、把末端执行器放到绝对位姿、按相对增量移动它，或执行预先规划好的轨迹。dsh-ros2 没有堆一堆按名字命名的工具，而是把它们抽象成一个工具 moveit_move，带一个 mode 参数 —— 接口因此保持精简、可预测、可脚本化，并且适用于任何* MoveIt 包（它读取 SRDF，只说标准 moveit_msgs）。

| 工具 | 作用 |
| moveit_discover (L1) | 读取任意 MoveIt 包的 SRDF：规划组、命名位姿、链末端；在线探测标准接口（/move_action、/execute_trajectory 等） |
| moveit_status (L1) | 运行时探测：接口在线状态 + 当前 /joint_states 采样 + SRDF 规划帧 |
| moveit_move (L2，需审批) | 一个工具，五种模式：joint_abs（关节 "j1:=v1 j2:=v2"）、joint_rel（deltaJoints = 当前值 + 增量）、pose_abs（位姿 "x y z rx ry rz"，在规划帧下）、pose_rel（deltaPose "dx dy dz drx dry drz"，帧为 ee/world）、trajectory（执行保存好的轨迹 JSON） |

moveit_move {mode: "joint_abs", group: "right_arm", joints: "right_shoulder_roll:=0.5"}
moveit_move {mode: "joint_rel", group: "right_arm", deltaJoints: "right_elbow_pitch:=-0.2"}
moveit_move {mode: "pose_rel", group: "right_arm", deltaPose: "0.05 0 0 0 0 0"}
moveit_move {mode: "pose_abs", group: "right_arm", pose: "0.5 0 0.8 0 0 0"}
moveit_move {mode: "trajectory", group: "right_arm", trajectory: "/tmp/traj.json"}

工作原理。 moveit_discover/moveit_status 告诉你能动什么（groups、joints、EE 链接取自 SRDF 链末端、在线状态）；moveit_move 把任意模式转成标准 MoveGroup 目标（/move_action），经 /execute_trajectory 执行，并配合 planOnly + trajectoryOut 保存规划好的轨迹，供 mode: "trajectory" 稍后运行（规划与执行分离）。不绑定任何具体 MoveIt 包，只有 SRDF 路径是关键（自动扫描包解析，或显式指定 srdf/package）。

安全框架

设计意图。 两层策略，把实时判断与语义判断分开（完整约定见 docs/safety-handover.md，这是给下游机器人适配智能体的交接文档——通用框架/接口放这里，本体相关的数据源/方案/算法放下游）：

tool-layer safety gate (pre-execution: /safety/state LOCKED; monitor-down per safetyStrict)
→ reactive monitors (in-execution: motion tracking/stall + hysteresis,
joint-feedback loss, watchdog, optional torque; detect at control frequency,
response budget ≤100 ms)
→ VLM semantic arbitration (post plan-change / anomaly, seconds — pulled up only when needed) → human arbitration (non-safe verdicts always escalate; human-gated unlock)

- safety_monitor node (dsh_ros2_safety package): subscribes the joint
feedback (+ optional commanded stream / torque stream), runs the checkers on
a control_frequency timer, and latches LOCKED on any CRITICAL event — the latch persists until a human unlocks (no auto-reset into the same danger). Publishes /safety/state (transient-local), /safety/event, /safety/heartbeat, /safety/lock_active; services /safety/get_state, /safety/unlock, /safety/set_lock.
- Latched lock, non-fatal never locks: any CRITICAL event latches LOCKED
(the latch persists until a human unlocks — no auto-reset into the same danger); watchdog separates critical (down → lock) from observed (down → WARNING only); single-frame noise is filtered by M-of-K hysteresis; a WARNING never latches. Tool-layer fail-closed on monitor-down is per safetyStrict: 'reject' (default 'warn' proceeds with a warning).
- Per-robot registration: robot_register writes a generic safety
section (URDF-derived velocity/effort limits) and auto-launches the monitor;
robot_profile.py safety set <key> <json> updates any threshold/topic/list
(schema-validated). Torque checking auto-disables when no effort feedback
exists. Reserved layers (interfaces registered, not implemented): the
geometric pre-check (joint-limit / velocity / FK self-collision on the command
path — motion.max_velocity / max_acceleration), the computed-torque feedforward input (torque.feedforward_topic), YOLO-style lightweight triggers, and the non-ROS estop path (estop).
- Tool-layer gate: moveit_move consults /safety/state before executing
— LOCKED always rejected (SAFETY_LOCKED); if the monitor is unreachable, safetyStrict: 'reject' (fail-closed) or 'warn' (default) proceed-with- warning. robot_safety_arbitrate runs the fixed-format VLM arbitration and flags any non-safe verdict for human arbitration.
- Forensics: a ring buffer of joint/torque samples is dumped to
forensics.dump_dir on every CRITICAL lock, for post-hoc / VLM diagnosis.

build the safety package (same colcon workspace as vlm/ and offscreen/)
ln -s <repo>/safety ~/vlm_ws/src/dsh_ros2_safety
cd ~/vlm_ws && colcon build --symlink-install && source install/setup.bash

The safety_core pure logic ships with a fault-injection self test
(python3 packages/safety/safety/scripts/safety_core.py --selftest, 12 scenarios) — no ROS2 needed to verify the state machine.

Robot profiles & communication topology

Design intent. A robot's body (URDF links/joints, cameras, MoveIt groups,
zero-pose semantics) and its comms graph are costly to rediscover every time.
The plugin persists them as a structured profile (~/.dsh-ros2/robots/<name>.yaml):
register once on first contact, then load instantly forever after. For the
comms graph, full verbosity does not scale (a complex robot has hundreds of
topics/services), so the design is a trade-off: an aggregate snapshot
(light node/topic/service lists) plus progressive learning of only the
important nodes as you actually work with them.

| Tool | Role |
| robot_register (L2) | Collect body info (URDF links/joints, TF root, cameras, MoveIt SRDF groups, auto-links the zero-pose calibration and a generic safety section with URDF-derived limits) into the profile; auto-launches the safety monitor (startSafety: false to skip) |
| robot_load (L1) | Load the profile as structured JSON — the fast path, no rediscovery; empty name lists all |
| robot_topology (L1/L2) | Comms graph: snapshot (L2, aggregate lists), learn (L2, one important node's role/description + pub/sub/srv/act, strict schema), show (L1, read back), diagnose (L1 — knowledge-augmented diagnosis: cross-references the learned knowledge base + snapshot against the LIVE graph: missing / new / drift / topic_drift), search (L1 — efficient retrieval in the knowledge archive: reverse-lookup by topic / keyword match on name/role/description/connections) |
| ros2_zero_pose_semantics (L2) | Calibrate zero pose via render + VLM + user confirm (arm/elbow/palm combos or free text); the profile auto-includes it |

Two bundled skills complete the workflow: robot-registration (first-contact
flow: ask name/URDF → collect → register → verify, plus the topology baseline
snapshot) and robot-retrieval (load the profile instantly and bring up
renders/diagnostics/motion from it, including the learned topology and zero-pose
semantics).

How it works. Everything is plain structured YAML under ~/.dsh-ros2/ —
robots/<name>.yaml (body + topology) and zero-pose.yaml (calibration).
robot_register snapshots the body; robot_topology snapshot snapshots the
aggregate layer; robot_topology learn appends one node at a time (idempotent
merge) as you discover what matters. robot_load and robot_topology show read
them back instantly — one call instead of N discovery calls.

The knowledge base is consumed, not just stored. robot_topology diagnose
(L1, read-only) is the entry point for knowledge-augmented diagnosis: it
loads the learned nodes + snapshot and cross-references them against the LIVE
ros2 graph —

- missing: learned nodes that are offline now (controllers/publishers down) — highest priority;
- new: live nodes not in the knowledge base (learn candidates);
- matched[].drift: per learned node, expected pub/sub/srv/act vs actual (a missing topic = a gone connection; a new topic = the node changed);
- topic_drift: aggregate snapshot topics vs live topics.

The ros2-diagnostics and robot-retrieval skills both start diagnosis with
diagnose and close the loop by learning important new nodes — so the knowledge base improves with every session and every diagnosis gets faster.

Bundled skills

| Skill | Content |
| ros2-diagnostics | Which tool to use when, narrow-down methodology, debugging "topic has no data" / message-mismatch / TF problems |
| ros2-bringup-recovery | Journey "it won't come up": ros2_env_check → ros2_workspace use (in-session, no restart) → launch → job tracking → doctor verification |
| ros2-liveness-triage | Journey "is it alive / why stale": ros2_topology {rates} → ros2_topic_sample → read publisher count / rate / QoS, with the timeout semantics that cause false alarms |
| ros2-tf-integrity | Journey "is the TF tree right": frames beside the graph, /tf + latched /tf_static edges, inverse-edge lookups, localizing a missing edge to its broadcaster |
| robot-state-vision-analysis | Full pipeline: status → offscreen render → VLM → cross-check (includes Jazzy Description Source/Topic, URDF↔TF frame-name matching, file:// meshes, view distance, FM frames signal, calibrated zero-pose semantics) |
| robot-registration | First-contact flow: ask name/URDF → collect body info + zero-pose calibration → robot_register → topology baseline snapshot |
| robot-retrieval | Instant profile load (robot_load) and bring-up of renders / diagnostics / motion; read & progressively learn the comms topology (robot_topology) |
| robot-motion-control | Journey "can it move / how": moveit_status → group discovery → planOnly → motion_validate → the single approval- and /safety/state-gated execution path → pass/fail verification |
| robot-safety-procedure | Journey "is it safe": robot_safety_state first, LOCKED as a stop signal, VLM arbitration (uncertain ≠ safe), the six layers and their explicit boundaries |

Configuration

| Key | Type | Default | Meaning |
| rosSetup | string | '' | Shell prefix to prepare the environment, e.g. source /opt/ros/jazzy/setup.bash && |
| timeoutMs | number | 15000 | Per-command timeout |
| rosLogDir | string | '' | Override ROS_LOG_DIR (helps when ~/.ros/log is not writable) |
| workspaceRoot | string | '' | cwd for colcon / rosdep when a tool omits cwd |
| includeStderr | boolean | false | Attach trailing stderr to successful results |
| display | string | '' | DISPLAY override for GUI/screenshot tools |
| screenshotDir | string | '' | Screenshot output dir (default $TMPDIR/dsh-ros2) |
| screenshotCommand | string | '' | Custom screenshot command; {output} is replaced with the PNG path |
| guiAllowedApplications | string[] | [] | Applications ros2_gui_interact may drive without approval; empty = rviz2 / rqt / rqt_graph |
| vision.provider | string | 'mock' | mock \| gemini \| openai |
| vision.apiKey | string | '' | Your API key (user-supplied; never logged) |
| vision.model | string | '' | Model override (e.g. gemini-2.5-flash, gpt-4o-mini) |
| vision.baseUrl | string | '' | API base URL override (OpenAI-compatible endpoints) |

rosLogDir also covers ROS2 Python CLIs spawned by tools (topic echo/pub, ros2 run); additionally runCommand auto-falls back to a writable dir when ~/.ros/log is not writable.

Project layout

dsh-ros2/ # pnpm monorepo (workspace root, private)
├── pnpm-workspace.yaml # packages/*
├── tsconfig.base.json
├── packages/
│ ├── common/ # dsh-ros2-common (not a bundle): runner / parse / toolkit + scripts/robot_profile.py (zero-copy)
│ ├── core/ # dsh-ros2-core (61 tools): L1 diagnostics + L2 management + L3 GUI + diagnostics/bring-up/liveness/TF skills + gui.ts + pty_session.py
│ ├── profile/ # dsh-ros2-profile (4 tools): robot_register/load/topology + zero-pose calibration + registration/retrieval skills
│ ├── moveit/ # dsh-ros2-moveit (4 tools): discover/status/motion_validate/moveit_move + moveit_.py + motion_validator.py + robot-motion-control skill
│ ├── safety/ # dsh-ros2-safety (5 tools): robot_safety_ + safety/ ROS2 pkg + safetyStrict config + robot-safety-procedure skill
│ ├── vision/ # dsh-ros2-vision (7 tools): vision tools + vlm/ + offscreen/ ROS2 pkgs + vision provider service + state-vision skill
│ └── dsh-ros2/ # aggregate bundle (empty apply, backward compat)
├── docs/ # architecture.md · safety.md / safety-handover.md / safety-todo.md / safety-gpt-review.md · test-*.md · plugin-split-plan.md ├── .github/workflows/ # CI: Node 22/24 → workspace typecheck/test/build + per-package tarball validation └── CHANGELOG.md # version history (Keep a Changelog)

Plugin split (9 packages)

Since v0.15.0 the plugin is a pnpm monorepo of 9 npm packages (per
docs/plugin-split-plan.md, ISP-tightened): 83 tools +
9 skills (the four original skills preserved with unchanged names and behavior).
Install the domain
bundles you need (or the dsh-ros2 aggregate for the full set):

- dsh-ros2-common is a plain library (not a cordis bundle) — shared runner, parsers, toolkit and scripts/robot_profile.py (zero-copy).
- Cross-package runtime contracts stay unchanged: /vlm/describe,
/safety/state, /safety/set_lock …; safetyStrict semantics unchanged.
- dsh-ros2-vision fixes the npm publish defect: its files include
vlm/ + offscreen/.
- The vision provider is an optional cordis service (dshRos2.vision);
ros2_gui_observe (core) and ros2_zero_pose_semantics (profile) degrade to
VISION_UNAVAILABLE when it is absent.

Troubleshooting / FAQ

- ~/.ros/log Permission denied: ROS2 cannot write its log dir. Set rosLogDir (e.g. /tmp/ros-log); runCommand also falls back automatically.
- A flood of RTPS_TRANSPORT_SHM / FastDDS SHM warnings in stderr: harmless noise when SHM transport is unavailable (common in containers/restricted environments); tools drop it by default.
- ros2 topic echo returns empty: check ros2_topic_info -v for publisher count and QoS; sample transient-local topics with --qos-durability transient_local.
- Offscreen render: "all parts piled at origin": URDF link names do not match TF frame names (see mesh essentials #3); check the node log FM transformHasProblems()=1 first.
- RobotModel shows no meshes (Links empty): on Jazzy you must use Description Source/Topic; the legacy Robot Description: does nothing.
- Could not load resource ... Unable to open file: mesh paths need absolute paths or a file:// prefix.
- Description missing after the publisher exits: the URDF publisher must stay alive (transient-local only re-sends once to late subscribers; gone when the process exits).
- vision_bringup missed some image topics (2/4 measured): fixed — it now refreshes discovery every --refresh seconds (default 10), auto-spawning bridges for topics that appear later and stopping bridges for topics that disappear.

Development

Install with pnpm 11.x (the repo pins packageManager: pnpm@11.22.0; CI installs the matching
pnpm via pnpm/action-setup@v6). Node ^22.19 || >=24 is required.
bash
pnpm install
pnpm run typecheck # tsc --noEmit
pnpm run test # vitest (425 cases; plus 11 sidecar + 15 zero-pose + 38 robot-profile + 16 private-path Python checks)
pnpm run build # tsc -> lib/ + lib/types/
pnpm run audit # supply-chain gate: runtime graph, high+ (the exact command CI runs)
pnpm run audit:all # ... plus the dev-only tooling that CI's gate deliberately excludes

CI (.github/workflows/ci.yml): on push to main / PRs, runs typecheck/test/build on Node 22 and 24, and validates that the pnpm pack artifact contains the patch layer (cordis.patch.yml) and the build output.

Release workflow (npm & GitHub Releases): see PUBLISH.md.

Roadmap

- [x] vision_bringup polling/refresh discovery (auto-bridge late topics, stop bridges for gone topics);
- [x] Zero-pose semantics: generic calibration flow (ros2_zero_pose_semantics, render + VLM + user confirm, 3-axis combos) linked into robot profiles;
- [x] npm publishing (9 packages @ 0.1.0 on npmjs, published 2026-08-30; dsh-ros2-state/dsh-ros2-sidecar added 2026-09; see docs/versioning.md);
- [ ] More ROS2 distros (Humble / Rolling) compatibility validation.

Docs

| Doc | Content |
| docs/architecture.md | Design overview, four tiers, L4 vision & offscreen rendering architecture, performance evolution, safety model |
| docs/compatibility.md | Compatibility baseline |
| docs/safety.md | Safety boundary: six layers (agent permission / human approval / motion validation / execution monitoring / post-execution verification / physical robot safety), fail-closed & degradation policy, "DSH is not a functional-safety system" |
| docs/safety-handover.md | Handover for downstream robot-adaptation agents: generic framework/interfaces vs body-specific data sources/algorithms, profile safety schema, interfaces |
| docs/safety-todo.md | GPT review decisions + batches: 0.14.1 done (deterministic validation), GUI allowlist landed (§17), still pending (audit JSONL / C++ real-time node ...) |
| docs/test-robot-state-vision.md | End-to-end real-robot tests: pipeline, realtime, mesh/TF binding fix & verification, 4-channel joint analysis (with images) |
| docs/test-gpu-passthrough.md | GPU passthrough verification: hardware, troubleshooting, results, usage |
| CHANGELOG.md | Version history (Keep a Changelog) |
| docs/versioning.md | GitHub tag ↔ npm package version correspondence (monorepo re-baseline; old single-package tags v0.8–v0.15 were never on npm) |

Contributing

Issues and PRs welcome (in Chinese or English). Please keep pnpm run typecheck && pnpm run test && pnpm run build green and update the relevant docs.

Acknowledgments

- DeepSeek Harness — plugin host framework;
- ROS2 / RViz2 community — rendering and toolchain foundations.

License

MIT