Dev vlm open loop use training demo

[datvo06]

WIP

[datvo06]

@yichao-liang I ran into this problem when the solver fails to solve.
The problem is that when the approach fails to solve (line 227-230), the code enters the exception handler where it sets succeed_in_solving = False. Then later at line 242-243, it checks if succeed_in_solving is True to decide between two execution paths.
However, when the approach fails and succeed_in_solving is False, it tries to use policy at line 258, but policy is only defined in two places:
Line 223-224: Only if CFG.demonstrator is NOT "oracle" or "oracle_process_planning"
Line 236: Only if CFG.keep_failed_demos is True

Should I add --keep_failed_demos True?

(.venv) ➜  predicators git:(dev_vlm_open_loop_use_training_demo) ✗ python predicators/main.py --env burger --approach vlm_open_loop --seed 0 \
--num_train_tasks 1 --num_test_tasks 1 --bilevel_plan_without_sim True \
--make_failure_videos --sesame_task_planner fdopt --debug \
--vlm_model_name gemini-1.5-pro-latest --vlm_open_loop_use_training_demos True
Gym has been unmaintained since 2022 and does not support NumPy 2.0 amongst other critical functionality.
Please upgrade to Gymnasium, the maintained drop-in replacement of Gym, or contact the authors of your software and request that they upgrade.
Users of this version of Gym should be able to simply replace 'import gym' with 'import gymnasium as gym' in the vast majority of cases.
See the migration guide at https://gymnasium.farama.org/introduction/migration_guide/ for additional information.
pybullet build time: Sep  1 2025 21:46:14
/Users/nguyendat/Marc/basis/predicators/.venv/lib/python3.13/site-packages/gym_sokoban/__init__.py:2: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
  import pkg_resources
INFO: Running command: python predicators/main.py --env burger --approach vlm_open_loop --seed 0 --num_train_tasks 1 --num_test_tasks 1 --bilevel_plan_without_sim True --make_failure_videos --sesame_task_planner fdopt --debug --vlm_model_name gemini-1.5-pro-latest --vlm_open_loop_use_training_demos True
INFO: Full config:
INFO: namespace(num_train_tasks=1, num_test_tasks=1, num_online_learning_cycles=10, online_learning_max_transitions=inf, online_learning_early_stopping=False, skip_test_until_last_ite_or_early_stopping=False, max_initial_demos=inf, allow_interaction_in_demo_tasks=True, max_num_steps_interaction_request=100, pretty_print_when_loading=False, test_env_seed_offset=10000, test_task_json_dir=None, segmenter='option_changes', demonstrator='oracle', render_state_dpi=150, approach_wrapper=None, allow_exclude_goal_predicates=False, allow_state_allclose_comparison_despite_simulator_state=False, env_include_bbox_features=False, cover_multistep_action_limits=[-inf, inf], cover_multistep_degenerate_oracle_samplers=False, cover_multistep_max_tb_placements=100, cover_multistep_max_hr_placements=100, cover_multistep_thr_percent=0.4, cover_multistep_bhr_percent=0.4, cover_multistep_bimodal_goal=False, cover_multistep_goal_conditioned_sampling=False, cover_blocks_change_color_when_cover=False, bumpy_cover_num_bumps=2, bumpy_cover_spaces_per_bump=1, bumpy_cover_right_targets=False, bumpy_cover_bumpy_region_start=0.8, bumpy_cover_init_bumpy_prob=0.25, regional_bumpy_cover_include_impossible_nsrt=False, blocks_num_blocks_train=[3, 4], blocks_num_blocks_test=[5, 6], blocks_holding_goals=False, blocks_block_size=0.045, blocks_high_towers_are_unstable=False, balance_num_blocks_train=[2, 4], balance_num_blocks_test=[4, 6], balance_holding_goals=False, balance_block_size=0.045, balance_wierd_balance=False, grow_plant_same_color_as_cup=False, grow_weak_pour_terminate_condition=False, grow_place_option_no_sampler=False, grow_num_cups_train=[2], grow_num_cups_test=[2, 3], grow_num_jugs_train=[2], grow_num_jugs_test=[2], laser_zero_reflection_angle=False, laser_use_debug_line_for_beams=False, ants_ants_attracted_to_points=False, playroom_num_blocks_train=[3], playroom_num_blocks_test=[3], cluttered_table_num_cans_train=5, cluttered_table_num_cans_test=10, cluttered_table_can_radius=0.01, cluttered_table_collision_angle_thresh=0.7853981633974483, cluttered_table_place_goal_conditioned_sampling=True, repeated_nextto_num_dots=15, repeated_nextto_nextto_thresh=0.5, painting_initial_holding_prob=0.5, painting_lid_open_prob=0.3, painting_num_objs_train=[2, 3], painting_num_objs_test=[3, 4], painting_max_objs_in_goal=inf, painting_goal_receptacles='box_and_shelf', painting_raise_environment_failure=True, rnt_painting_num_objs_train=[8, 9, 10], rnt_painting_num_objs_test=[11, 12, 13], rnt_painting_max_objs_in_goal=2, tools_num_items_train=[2], tools_num_items_test=[2, 3], tools_num_contraptions_train=[2], tools_num_contraptions_test=[3], sandwich_ingredients_train={'bread': [2], 'patty': [1], 'ham': [1], 'egg': [1], 'cheese': [1], 'lettuce': [1], 'tomato': [1], 'green_pepper': [1]}, sandwich_ingredients_test={'bread': [2], 'patty': [1], 'ham': [1], 'egg': [1], 'cheese': [1], 'lettuce': [1], 'tomato': [1], 'green_pepper': [1]}, pybullet_draw_debug=False, pybullet_camera_width=335, pybullet_camera_height=180, pybullet_sim_steps_per_action=20, pybullet_max_ik_iters=100, pybullet_ik_tol=0.001, pybullet_robot='fetch', pybullet_birrt_num_attempts=10, pybullet_birrt_num_iters=100, pybullet_birrt_smooth_amt=50, pybullet_birrt_extend_num_interp=10, pybullet_control_mode='position', pybullet_max_vel_norm=0.05, pybullet_robot_ee_orns=defaultdict(<function GlobalSettings.<lambda> at 0x17d3b5300>, {'pybullet_blocks': {'fetch': (0.7071, 0.0, -0.7071, 0.0), 'panda': (0.7071, 0.7071, 0.0, 0.0)}, 'pybullet_balance': {'fetch': (0.7071, 0.0, -0.7071, 0.0), 'panda': (0.7071, 0.7071, 0.0, 0.0)}}), pybullet_ik_validate=True, ikfast_max_time=0.05, ikfast_max_candidates=100, ikfast_max_attempts=inf, ikfast_max_distance=inf, ikfast_norm=inf, pddl_blocks_procedural_train_min_num_blocks=3, pddl_blocks_procedural_train_max_num_blocks=4, pddl_blocks_procedural_train_min_num_blocks_goal=2, pddl_blocks_procedural_train_max_num_blocks_goal=3, pddl_blocks_procedural_test_min_num_blocks=5, pddl_blocks_procedural_test_max_num_blocks=6, pddl_blocks_procedural_test_min_num_blocks_goal=2, pddl_blocks_procedural_test_max_num_blocks_goal=5, pddl_blocks_procedural_new_pile_prob=0.5, pddl_blocks_fixed_train_indices=[1, 2, 3, 4, 5], pddl_blocks_fixed_test_indices=[6, 7, 8, 9, 10], pddl_delivery_procedural_train_min_num_locs=5, pddl_delivery_procedural_train_max_num_locs=10, pddl_delivery_procedural_train_min_want_locs=2, pddl_delivery_procedural_train_max_want_locs=4, pddl_delivery_procedural_train_min_extra_newspapers=0, pddl_delivery_procedural_train_max_extra_newspapers=1, pddl_delivery_procedural_test_min_num_locs=31, pddl_delivery_procedural_test_max_num_locs=40, pddl_delivery_procedural_test_min_want_locs=20, pddl_delivery_procedural_test_max_want_locs=30, pddl_delivery_procedural_test_min_extra_newspapers=0, pddl_delivery_procedural_test_max_extra_newspapers=10, pddl_easy_delivery_procedural_train_min_num_locs=3, pddl_easy_delivery_procedural_train_max_num_locs=5, pddl_easy_delivery_procedural_train_min_want_locs=1, pddl_easy_delivery_procedural_train_max_want_locs=2, pddl_easy_delivery_procedural_train_min_extra_newspapers=0, pddl_easy_delivery_procedural_train_max_extra_newspapers=1, pddl_easy_delivery_procedural_test_min_num_locs=4, pddl_easy_delivery_procedural_test_max_num_locs=6, pddl_easy_delivery_procedural_test_min_want_locs=2, pddl_easy_delivery_procedural_test_max_want_locs=3, pddl_easy_delivery_procedural_test_min_extra_newspapers=0, pddl_easy_delivery_procedural_test_max_extra_newspapers=1, pddl_spanner_procedural_train_min_nuts=1, pddl_spanner_procedural_train_max_nuts=3, pddl_spanner_procedural_train_min_extra_spanners=0, pddl_spanner_procedural_train_max_extra_spanners=2, pddl_spanner_procedural_train_min_locs=2, pddl_spanner_procedural_train_max_locs=4, pddl_spanner_procedural_test_min_nuts=10, pddl_spanner_procedural_test_max_nuts=20, pddl_spanner_procedural_test_min_extra_spanners=0, pddl_spanner_procedural_test_max_extra_spanners=10, pddl_spanner_procedural_test_min_locs=20, pddl_spanner_procedural_test_max_locs=30, pddl_forest_procedural_train_min_size=8, pddl_forest_procedural_train_max_size=10, pddl_forest_procedural_test_min_size=10, pddl_forest_procedural_test_max_size=12, pddl_gripper_procedural_train_min_num_rooms=3, pddl_gripper_procedural_train_max_num_rooms=5, pddl_gripper_procedural_train_min_num_balls=1, pddl_gripper_procedural_train_max_num_balls=2, pddl_gripper_procedural_test_min_num_rooms=3, pddl_gripper_procedural_test_max_num_rooms=5, pddl_gripper_procedural_test_min_num_balls=1, pddl_gripper_procedural_test_max_num_balls=2, pddl_ferry_procedural_train_min_num_locs=3, pddl_ferry_procedural_train_max_num_locs=5, pddl_ferry_procedural_train_min_num_cars=1, pddl_ferry_procedural_train_max_num_cars=2, pddl_ferry_procedural_test_min_num_locs=3, pddl_ferry_procedural_test_max_num_locs=5, pddl_ferry_procedural_test_min_num_cars=1, pddl_ferry_procedural_test_max_num_cars=2, pddl_miconic_procedural_train_min_buildings=1, pddl_miconic_procedural_train_max_buildings=2, pddl_miconic_procedural_train_min_floors=3, pddl_miconic_procedural_train_max_floors=5, pddl_miconic_procedural_train_min_passengers=1, pddl_miconic_procedural_train_max_passengers=2, pddl_miconic_procedural_test_min_buildings=1, pddl_miconic_procedural_test_max_buildings=2, pddl_miconic_procedural_test_min_floors=3, pddl_miconic_procedural_test_max_floors=5, pddl_miconic_procedural_test_min_passengers=1, pddl_miconic_procedural_test_max_passengers=2, stick_button_num_buttons_train=[1, 2], stick_button_num_buttons_test=[3, 4], stick_button_disable_angles=True, stick_button_holder_scale=0.1, screws_num_screws_train=[15, 20], screws_num_screws_test=[25, 30], doors_room_map_size=5, doors_min_obstacles_per_room=0, doors_max_obstacles_per_room=3, doors_min_room_exists_frac=0.25, doors_max_room_exists_frac=0.75, doors_birrt_num_attempts=10, doors_birrt_num_iters=100, doors_birrt_smooth_amt=50, doors_draw_debug=False, doorknobs_target_value=0.75, test_doors_room_map_size=10, narrow_passage_open_door_refine_penalty=0, narrow_passage_door_width_padding_lb=0.0001, narrow_passage_door_width_padding_ub=0.015, narrow_passage_passage_width_padding_lb=0.0005, narrow_passage_passage_width_padding_ub=0.02, narrow_passage_birrt_num_attempts=10, narrow_passage_birrt_num_iters=100, narrow_passage_birrt_smooth_amt=50, exit_garage_clear_refine_penalty=0, exit_garage_min_num_obstacles=2, exit_garage_max_num_obstacles=3, exit_garage_rrt_extend_fn_threshold=0.001, exit_garage_rrt_num_control_samples=100, exit_garage_rrt_num_attempts=3, exit_garage_rrt_num_iters=100, exit_garage_rrt_sample_goal_eps=0.1, exit_garage_motion_planning_ignore_obstacles=False, exit_garage_raise_environment_failure=False, coffee_num_cups_train=[1, 2], coffee_num_cups_test=[2, 3], coffee_jug_init_rot_amt=2.0943951023931953, coffee_rotated_jug_ratio=0.5, coffee_twist_sampler=True, coffee_combined_move_and_twist_policy=False, coffee_move_back_after_place_and_push=False, coffee_jug_pickable_pred=False, coffee_render_grid_world=False, coffee_simple_tasks=False, coffee_machine_have_light_bar=True, coffee_machine_has_plug=False, coffee_use_pixelated_jug=False, coffee_plug_break_after_plugged_in=False, coffee_fill_jug_gradually=False, satellites_num_sat_train=[2, 3], satellites_num_obj_train=[3, 4], satellites_num_sat_test=[3, 4], satellites_num_obj_test=[4, 5], sokoban_gym_name='Sokoban-v0', kitchen_use_perfect_samplers=False, kitchen_goals='all', kitchen_render_set_of_marks=False, kitchen_use_combo_move_nsrts=False, kitchen_randomize_init_state=False, sticky_table_num_tables=5, sticky_table_place_smooth_fall_prob=0.6, sticky_table_place_sticky_fall_prob=0.0, sticky_table_pick_success_prob=1.0, sticky_table_tricky_floor_place_sticky_fall_prob=0.5, sticky_table_place_ball_fall_prob=1.0, sticky_table_num_sticky_tables=1, grid_row_num_cells=100, float_water_level_doesnt_raise=False, domino_debug_layout=False, domino_some_dominoes_are_connected=False, domino_initialize_at_finished_state=True, domino_use_domino_blocks_as_target=False, domino_use_grid=False, domino_include_connected_predicate=False, domino_prune_actions=False, domino_train_num_dominos=[2], domino_test_num_dominos=[3], domino_train_num_targets=[1], domino_test_num_targets=[1, 2], domino_train_num_pivots=[0], domino_test_num_pivots=[0], domino_train_num_pos_x=3, domino_train_num_pos_y=2, domino_test_num_pos_x=4, domino_test_num_pos_y=3, burger_render_set_of_marks=True, burger_no_move_task_type='more_stacks', burger_dummy_render=False, burger_num_test_start_holding=5, circuit_light_doesnt_need_battery=False, fan_fans_blow_opposite_direction=False, fan_known_controls_relation=True, fan_combine_switch_on_off=False, fan_train_num_pos_x=3, fan_train_num_pos_y=3, fan_test_num_pos_x=9, fan_test_num_pos_y=4, fan_train_num_walls_per_task=[1], fan_test_num_walls_per_task=[3, 4], boil_use_constant_delay=False, boil_use_normal_delay=True, boil_use_cmp_delay=False, boil_goal='simple', boil_goal_simple_human_happy=False, boil_use_derived_predicates=True, boil_require_jug_full_to_heatup=False, boil_goal_require_burner_off=True, boil_add_jug_reached_capacity_predicate=False, boil_num_jugs_train=[1], boil_num_jugs_test=[1, 2], boil_num_burner_train=[1], boil_num_burner_test=[1], random_options_max_tries=100, option_model_terminate_on_repeat=True, option_model_use_gui=False, gnn_num_message_passing=3, gnn_layer_size=16, gnn_learning_rate=0.001, gnn_weight_decay=0, gnn_num_epochs=25000, gnn_batch_size=128, gnn_do_normalization=False, gnn_use_validation_set=True, gnn_option_policy_solve_with_shooting=True, gnn_option_policy_shooting_variance=0.1, gnn_option_policy_shooting_max_samples=100, metacontroller_max_samples=100, pg3_heuristic='policy_guided', pg3_search_method='hill_climbing', pg3_task_planning_heuristic='lmcut', pg3_gbfs_max_expansions=100, pg3_hc_enforced_depth=0, pg3_max_policy_guided_rollout=50, pg3_plan_compare_inapplicable_cost=0.99, pg3_add_condition_allow_new_vars=True, pg3_max_analogies=5, pg3_init_policy=None, pg3_init_base_env=None, nsrt_rl_reward_epsilon=0.01, nsrt_rl_pos_reward=0, nsrt_rl_neg_reward=-1, nsrt_rl_option_learner='dummy_rl', nsrt_rl_valid_reward_steps_threshold=10, pretrained_model_prompt_cache_dir='pretrained_model_cache', llm_openai_max_response_tokens=700, llm_use_cache_only=False, llm_model_name='text-curie-001', llm_temperature=0.5, llm_num_completions=1, pretrained_model_service_provider='openai', vlm_model_name='gemini-1.5-pro-latest', vlm_temperature=0.0, vlm_num_completions=1, vlm_include_cropped_images=False, use_hardcoded_vlm_atom_proposals=False, vlm_double_check_output=False, vlm_open_loop_use_training_demos=True, vlm_open_loop_no_image=False, sesame_task_planner='fdopt', sesame_task_planning_heuristic='lmcut', sesame_allow_noops=True, sesame_check_expected_atoms=True, sesame_use_necessary_atoms=True, sesame_use_visited_state_set=False, sesame_grounder='naive', sesame_check_static_object_changes=False, sesame_static_object_change_tol=0.001, bilevel_plan_without_sim=True, planning_filter_unreachable_nsrt=True, planning_check_dr_reachable=True, no_repeated_arguments_in_grounding=False, log_dir='logs', results_dir='results', eval_trajectories_dir='eval_trajectories', approach_dir='saved_approaches', data_dir='saved_datasets', video_dir='videos', image_dir='images', video_fps=2, failure_video_mode='longest_only', terminate_on_goal_reached=True, keep_failed_demos=False, terminate_on_goal_reached_and_option_terminated=False, offline_data_planning_timeout=-1, offline_data_task_planning_heuristic='default', offline_data_max_skeletons_optimized=-1, offline_data_num_replays=500, offline_data_bilevel_plan_without_sim=None, teacher_dataset_num_examples=1, min_data_for_nsrt=0, min_perc_data_for_nsrt=0, data_orderings_to_search=1, strips_learner='cluster_and_intersect', disable_harmlessness_check=False, enable_harmless_op_pruning=False, precondition_soft_intersection_threshold_percent=0.8, backchaining_check_intermediate_harmlessness=False, pnad_search_without_del=False, pnad_search_timeout=10.0, compute_sidelining_objective_value=False, clustering_learner_true_pos_weight=10, clustering_learner_false_pos_weight=1, cluster_and_intersect_prederror_max_groundings=10, cluster_and_search_inner_search_max_expansions=2500, cluster_and_search_inner_search_timeout=30, cluster_and_search_score_func_max_groundings=10000, cluster_and_search_var_count_weight=0.1, cluster_and_search_precon_size_weight=0.01, cluster_and_intersect_prune_low_data_pnads=False, cluster_and_intersect_min_datastore_fraction=0.0, cluster_and_intersect_soft_intersection_for_preconditions=False, find_best_matching_pnad_skip_if_effect_not_subset=True, exogenous_process_learner='cluster_and_intersect', exogenous_process_learner_do_intersect=False, only_learn_exogenous_processes=False, learn_process_parameters=False, pause_after_process_learning_for_inspection=False, learnable_delay_distribution='cmp', process_learner_check_false_positives=False, cluster_and_search_process_learner_parallel_condition=True, cluster_and_search_process_learner_parallel_pnad=False, cluster_and_search_process_learner_llm_select_condition=False, cluster_and_search_process_learner_llm_select_atoms=False, cluster_and_search_vi_steps=200, cluster_search_max_workers=-1, cluster_and_inverse_planning_candidates='top_consistent', cluster_and_inverse_planning_top_consistent_method='percentage', cluster_and_inverse_planning_top_consistent_num=-1, cluster_and_inverse_planning_top_p_percent=3, cluster_and_inverse_planning_top_consistent_max_cost=3, cluster_process_learner_top_n_conditions=-1, process_scoring_method='data_likelihood', process_condition_search_complexity_weight=0.0001, process_param_learning_num_steps=200, process_param_learning_patience=None, process_param_learning_batch_size=16, process_condition_search_prune_with_fp_count=False, process_learning_learn_strength=True, process_learning_process_per_physical_core=True, process_learning_init_at_previous_results=False, predicate_invent_neural_symbolic_predicates=False, cluster_learning_one_effect_per_process=False, use_derived_predicate_in_heuristic=True, process_planning_heuristic_weight=1.0, build_exogenous_process_index_for_planning=True, process_planning_use_abstract_policy=False, process_planning_max_policy_guided_rollout=10, use_torch_gpu=False, use_wandb=False, learning_rate=0.001, weight_decay=0, mlp_regressor_max_itr=10000, mlp_regressor_hid_sizes=[32, 32], mlp_regressor_clip_gradients=False, mlp_regressor_gradient_clip_value=5, mlp_classifier_hid_sizes=[32, 32], mlp_classifier_balance_data=True, cnn_regressor_max_itr=500, cnn_regressor_conv_channel_nums=[3, 3], cnn_regressor_conv_kernel_sizes=[5, 3], cnn_regressor_linear_hid_sizes=[32, 8], cnn_regressor_clip_gradients=True, cnn_regressor_gradient_clip_value=5, neural_gaus_regressor_hid_sizes=[32, 32], neural_gaus_regressor_max_itr=1000, mlp_classifier_n_iter_no_change=5000, implicit_mlp_regressor_max_itr=10000, implicit_mlp_regressor_num_negative_data_per_input=5, implicit_mlp_regressor_num_samples_per_inference=100, implicit_mlp_regressor_temperature=1.0, implicit_mlp_regressor_inference_method='derivative_free', implicit_mlp_regressor_derivative_free_num_iters=3, implicit_mlp_regressor_derivative_free_sigma_init=0.33, implicit_mlp_regressor_derivative_free_shrink_scale=0.5, implicit_mlp_regressor_grid_num_ticks_per_dim=100, pytorch_train_print_every=1000, sampler_learner='neural', max_rejection_sampling_tries=100, sampler_mlp_classifier_max_itr=10000, sampler_mlp_classifier_n_reinitialize_tries=1, sampler_learning_use_goals=False, sampler_disable_classifier=False, sampler_learning_regressor_model='neural_gaussian', sampler_learning_max_negative_data=100000, option_learning_action_converter='identity', interactive_num_ensemble_members=10, interactive_query_policy='threshold', interactive_score_function='entropy', interactive_score_threshold=0.05, interactive_random_query_prob=0.5, interactive_num_requests_per_cycle=10, predicate_classifier_model='mlp', predicate_mlp_classifier_max_itr=100000, predicate_mlp_classifier_n_reinitialize_tries=1, predicate_mlp_classifier_init='default', predicate_knn_classifier_n_neighbors=1, online_nsrt_learning_requests_per_cycle=10, online_learning_max_novelty_count=0, online_nsrt_learning_number_of_tasks_to_try=1, online_nsrt_learning_requests_per_task=3, active_sampler_learning_model='myopic_classifier_mlp', active_sampler_learning_feature_selection='all', active_sampler_learning_knn_neighbors=3, active_sampler_learning_use_teacher=True, active_sampler_learning_num_samples=100, active_sampler_learning_score_gamma=0.5, active_sampler_learning_fitted_q_iters=5, active_sampler_learning_explore_pursue_goal_interval=5, active_sampler_learning_object_specific_samplers=False, active_sampler_learning_n_iter_no_change=5000, active_sampler_learning_num_lookahead_samples=5, active_sampler_learning_explore_length_base=2, active_sampler_learning_num_ensemble_members=10, active_sampler_learning_exploration_sample_strategy='epsilon_greedy', active_sampler_learning_exploration_epsilon=0.5, active_sampler_learning_replay_buffer_size=1000000, active_sampler_learning_batch_size=64, use_epsilon_annealing=True, min_epsilon=0.05, skill_competence_model='optimistic', skill_competence_model_num_em_iters=3, skill_competence_model_max_train_iters=1000, skill_competence_model_learning_rate=0.01, skill_competence_model_lookahead=1, skill_competence_model_optimistic_window_size=5, skill_competence_model_optimistic_recency_size=5, skill_competence_default_alpha_beta=(10.0, 1.0), skill_competence_initial_prediction_bonus=0.5, refinement_estimator='oracle', refinement_estimation_num_skeletons_generated=8, refinement_data_num_skeletons=8, refinement_data_skeleton_generator_timeout=20, refinement_data_low_level_search_timeout=5, refinement_data_failed_refinement_penalty=5, refinement_data_include_execution_cost=True, refinement_data_low_level_execution_cost=0.05, cnn_refinement_estimator_crop=False, cnn_refinement_estimator_crop_bounds=(320, 400, 100, 650), cnn_refinement_estimator_downsample=2, bridge_policy='learned_ldl', glib_min_goal_size=1, glib_max_goal_size=1, glib_num_babbles=10, greedy_lookahead_max_num_trajectories=100, greedy_lookahead_max_traj_length=2, greedy_lookahead_max_num_resamples=10, active_sampler_explore_use_ucb_bonus=True, active_sampler_explore_bonus=0.1, active_sampler_explore_task_strategy='planning_progress', active_sampler_explorer_replan_frequency=100, active_sampler_explorer_planning_progress_max_tasks=10, active_sampler_explorer_planning_progress_max_replan_tasks=5, active_sampler_explorer_skip_perfect=True, active_sampler_learning_init_cycles_to_pursue_goal=1, bilevel_planning_explorer_enumerate_plans=False, exploit_bilevel_planning_explorer_fallback_explorer='RandomOptions', grammar_search_grammar_use_single_feature=True, grammar_search_grammar_includes_givens=True, grammar_search_grammar_includes_negation=True, grammar_search_grammar_includes_foralls=True, grammar_search_grammar_use_diff_features=False, grammar_search_grammar_use_euclidean_dist=False, grammar_search_grammar_use_skip_grammar=True, grammar_search_use_handcoded_debug_grammar=False, grammar_search_forall_penalty=1, grammar_search_pred_selection_approach='score_optimization', grammar_search_pred_clusterer='oracle', grammar_search_true_pos_weight=10, grammar_search_false_pos_weight=1, grammar_search_bf_weight=1, grammar_search_operator_complexity_weight=0.0, grammar_search_pred_complexity_weight=0.0001, grammar_search_max_predicates=200, grammar_search_predicate_cost_upper_bound=6, grammar_search_prune_redundant_preds=True, grammar_search_score_function='expected_nodes_created', grammar_search_additional_bonus_for_matching_plan=0, grammar_search_heuristic_based_weight=10.0, grammar_search_max_demos=inf, grammar_search_max_nondemos=50, grammar_search_energy_based_temperature=10.0, grammar_search_task_planning_timeout=1.0, grammar_search_search_algorithm='hill_climbing', grammar_search_hill_climbing_depth=0, grammar_search_parallelize_hill_climbing=False, grammar_search_gbfs_num_evals=1000, grammar_search_off_demo_count_penalty=1.0, grammar_search_on_demo_count_penalty=10.0, grammar_search_suspicious_state_penalty=10.0, grammar_search_expected_nodes_upper_bound=100000.0, grammar_search_expected_nodes_optimal_demo_prob=0.99999, grammar_search_expected_nodes_backtracking_cost=1000.0, grammar_search_expected_nodes_allow_noops=True, grammar_search_classifier_pretty_str_names=['?x', '?y', '?z', '?w'], grammar_search_vlm_atom_proposal_prompt_type='options_labels_whole_traj_diverse', grammar_search_vlm_atom_label_prompt_type='per_scene_naive', grammar_search_vlm_atom_proposal_use_debug=False, grammar_search_parallelize_vlm_labeling=True, grammar_search_select_all_debug=False, grammar_search_invent_geo_predicates_only=False, grammar_search_early_termination_heuristic_thresh=0.0, grammar_search_clustering_gmm_num_components=10, handmade_demo_filename='', vlm_trajs_folder_name='', vlm_predicate_vision_api_generate_ground_atoms=False, vlm_test_time_atom_label_prompt_type='per_scene_naive', save_eval_trajs=True, rgb_observation=False, render_init_state=False, use_counterfactual_dataset_path_name=False, use_classification_problem_setting=False, classification_has_counterfactual_support=True, dino_model_name='dinov2_vits14', distance_function='dtw', vlm_predicator_oracle_base_predicates=False, vlm_predicator_oracle_learned_predicates=False, vlm_predicator_use_grammar=True, vlm_predicator_num_proposal_batches=1, get_arg_specific_settings=<classmethod(<function GlobalSettings.get_arg_specific_settings at 0x17d3b53a0>)>, perceiver='trivial', horizon=100, max_num_steps_option_rollout=1000, sesame_propagate_failures='immediately', offline_data_method='demo', option_model_name='oracle', sesame_max_skeletons_optimized=8, sesame_max_samples_per_step=10, grammar_search_expected_nodes_max_skeletons=-1, grammar_search_diff_features_const_multiplier=1e-06, grammar_search_euclidean_feature_names=[('x', 'y', 'x', 'y')], grammar_search_euclidean_const_multiplier=1e-06, cover_num_blocks=2, cover_num_targets=2, cover_block_widths=[0.1, 0.07], cover_target_widths=[0.05, 0.03], cover_initial_holding_prob=0.75, env='burger', approach='vlm_open_loop', excluded_predicates='', included_options='', seed=0, option_learner='no_learning', explorer='no_explore', execution_monitor='trivial', timeout=10, make_test_videos=False, make_failure_videos=True, make_test_images=False, make_failure_images=False, make_interaction_videos=False, make_demo_videos=False, make_demo_images=False, make_cogman_videos=False, video_not_break_on_exception=False, load_approach=False, restart_learning=False, load_data=False, load_atoms=False, save_atoms=False, skip_until_cycle=-1, experiment_id='', load_experiment_id='', log_file='', use_gui=False, loglevel=10, crash_on_failure=False, excluded_objects_in_state_str='')
INFO: Git commit hash: 860e917161af6f5bf8b1f170ac01ae357399d859
/Users/nguyendat/Marc/basis/predicators/.venv/lib/python3.13/site-packages/gym/spaces/box.py:127: UserWarning: WARN: Box bound precision lowered by casting to float32
  logger.warn(f"Box bound precision lowered by casting to {self.dtype}")
INFO: Solving task 0...
INFO: [CogMan] Reset called.
WARNING: WARNING: Approach failed to solve with error: Plan not found with FD! Error: gtimeout: failed to run command ‘/Users/nguyendat/Marc/basis/predicators/external/downward/fast-downward.py/fast-downward.py’: Not a directory
INFO: Executing policy...
ERROR: main.py crashed
Traceback (most recent call last):
  File "/Users/nguyendat/Marc/basis/predicators/predicators/main.py", line 772, in <module>
    main()
    ~~~~^^
  File "/Users/nguyendat/Marc/basis/predicators/predicators/main.py", line 181, in main
    offline_dataset = create_offline_dataset(env, train_tasks, preds, approach)
  File "/Users/nguyendat/Marc/basis/predicators/predicators/main.py", line 140, in create_offline_dataset
    return create_dataset(env, train_tasks, options, preds)
  File "/Users/nguyendat/Marc/basis/predicators/predicators/datasets/__init__.py", line 28, in create_dataset
    return create_demo_data(env,
                            train_tasks,
                            known_options,
                            annotate_with_gt_ops=False)
  File "/Users/nguyendat/Marc/basis/predicators/predicators/datasets/demo_only.py", line 42, in create_demo_data
    dataset = _generate_demonstrations(
        env,
    ...<2 lines>...
        train_tasks_start_idx=0,
        annotate_with_gt_ops=annotate_with_gt_ops)
  File "/Users/nguyendat/Marc/basis/predicators/predicators/datasets/demo_only.py", line 258, in _generate_demonstrations
    policy,
    ^^^^^^
UnboundLocalError: cannot access local variable 'policy' where it is not associated with a value
Traceback (most recent call last):
  File "/Users/nguyendat/Marc/basis/predicators/predicators/main.py", line 775, in <module>
    raise _err
  File "/Users/nguyendat/Marc/basis/predicators/predicators/main.py", line 772, in <module>
    main()
    ~~~~^^
  File "/Users/nguyendat/Marc/basis/predicators/predicators/main.py", line 181, in main
    offline_dataset = create_offline_dataset(env, train_tasks, preds, approach)
  File "/Users/nguyendat/Marc/basis/predicators/predicators/main.py", line 140, in create_offline_dataset
    return create_dataset(env, train_tasks, options, preds)
  File "/Users/nguyendat/Marc/basis/predicators/predicators/datasets/__init__.py", line 28, in create_dataset
    return create_demo_data(env,
                            train_tasks,
                            known_options,
                            annotate_with_gt_ops=False)
  File "/Users/nguyendat/Marc/basis/predicators/predicators/datasets/demo_only.py", line 42, in create_demo_data
    dataset = _generate_demonstrations(
        env,
    ...<2 lines>...
        train_tasks_start_idx=0,
        annotate_with_gt_ops=annotate_with_gt_ops)
  File "/Users/nguyendat/Marc/basis/predicators/predicators/datasets/demo_only.py", line 258, in _generate_demonstrations
    policy,
    ^^^^^^
UnboundLocalError: cannot access local variable 'policy' where it is not associated with a value

[yichao-liang]

Try running this command to see if it works: python scripts/local/launch_simp.py -c mara/causal_predicator_baselines.yaml. When the baseline is uncommented.

You got the error because we’ve only implemented the process model for the domains listed in the YAML file, so burger isn’t included.

[datvo06]

@yichao-liang it works!

INFO: [CogMan] Reset called.
DEBUG: Loaded model response from pretrained_model_cache/OpenRouter-google/gemini-2.5-pro_most_likely_1_None_-769831615136490837/prompt.txt.
INFO: Executing policy...
DEBUG: Executing option Pick(robot, domino_1)
DEBUG: Executing option Place(robot, domino_1, domino_0, pos_y1_x1, rot_90)
DEBUG: Executing option Pick(robot, domino_2)
DEBUG: Executing option Place(robot, domino_2, domino_1, pos_y1_x2, rot_90)
DEBUG: Executing option Push(robot, domino_0)
DEBUG: [CogMan] State at the exception: {AdjacentTo(pos_y0_x3:loc, domino_0:domino), DominoAtPos(domino_0:domino, pos_y0_x2:loc), AdjacentTo(pos_y1_x1:loc, domino_3:domino), MovableBlock(domino_1:domino), AdjacentTo(pos_y1_x2:loc, domino_0:domino), DominoAtRot(domino_3:domino, rot_180:rot), Toppled(domino_3:domino), AdjacentTo(pos_y0_x1:loc, domino_0:domino), AdjacentTo(pos_y0_x0:loc, domino_3:domino), AdjacentTo(pos_y1_x2:loc, domino_1:domino), HandEmpty(robot:robot), PosClear(pos_y1_x3:loc), DominoAtPos(domino_1:domino, pos_y1_x1:loc), PosClear(pos_y2_x3:loc), AdjacentTo(pos_y0_x2:loc, domino_3:domino), AdjacentTo(pos_y2_x1:loc, domino_1:domino), PosClear(pos_y1_x0:loc), Toppled(domino_0:domino), DominoAtRot(domino_2:domino, rot_0:rot), StartBlock(domino_0:domino), MovableBlock(domino_2:domino), DominoAtRot(domino_0:domino, rot_-90:rot), PosClear(pos_y2_x0:loc), PosClear(pos_y2_x2:loc), DominoAtRot(domino_1:domino, rot_90:rot), AdjacentTo(pos_y0_x1:loc, domino_1:domino), PosClear(pos_y0_x0:loc), Upright(domino_1:domino), Toppled(domino_2:domino), PosClear(pos_y1_x2:loc), AdjacentTo(pos_y1_x0:loc, domino_1:domino), PosClear(pos_y2_x1:loc), PosClear(pos_y0_x3:loc), DominoAtPos(domino_3:domino, pos_y0_x1:loc)}
INFO: Approach failed at execution time with error: Option plan exhausted!
DEBUG: Final abstract state:
{AdjacentTo(pos_y0_x3:loc, domino_0:domino), DominoAtPos(domino_0:domino, pos_y0_x2:loc), AdjacentTo(pos_y1_x1:loc, domino_3:domino), MovableBlock(domino_1:domino), AdjacentTo(pos_y1_x2:loc, domino_0:domino), DominoAtRot(domino_3:domino, rot_180:rot), Toppled(domino_3:domino), AdjacentTo(pos_y0_x1:loc, domino_0:domino), AdjacentTo(pos_y0_x0:loc, domino_3:domino), AdjacentTo(pos_y1_x2:loc, domino_1:domino), HandEmpty(robot:robot), PosClear(pos_y1_x3:loc), DominoAtPos(domino_1:domino, pos_y1_x1:loc), PosClear(pos_y2_x3:loc), AdjacentTo(pos_y0_x2:loc, domino_3:domino), AdjacentTo(pos_y2_x1:loc, domino_1:domino), Toppled(domino_0:domino), PosClear(pos_y1_x0:loc), DominoAtRot(domino_2:domino, rot_0:rot), StartBlock(domino_0:domino), MovableBlock(domino_2:domino), DominoAtRot(domino_0:domino, rot_-90:rot), PosClear(pos_y2_x0:loc), PosClear(pos_y2_x2:loc), DominoAtRot(domino_1:domino, rot_90:rot), AdjacentTo(pos_y0_x1:loc, domino_1:domino), PosClear(pos_y0_x0:loc), Upright(domino_1:domino), Toppled(domino_2:domino), PosClear(pos_y1_x2:loc), AdjacentTo(pos_y1_x0:loc, domino_1:domino), PosClear(pos_y2_x1:loc), PosClear(pos_y0_x3:loc), DominoAtPos(domino_3:domino, pos_y0_x1:loc)}
DEBUG: Final state:
############################################## STATE ##############################################
type: direction      dir
-----------------  -----
left                   1
right                  2
straight               0

type: domino           x        y         z          rot          tilt     r     g     b    is_held
--------------  --------  -------  --------  -----------  ------------  ----  ----  ----  ---------
domino_0        0.816522  1.25374  0.415472  -1.54819      1.5708       0.56  0.93  0.56          0
domino_1        0.699969  1.35849  0.47499    1.58753      4.11255e-05  0.6   0.8   1             0
domino_2        0.799021  1.14747  0.407473  -0.00885297   1.57077      0.6   0.8   1             0
domino_3        0.705071  1.22735  0.407486  -2.96403     -1.57079      1     0.75  0.8           0

type: loc       xx     yy
-----------  -----  -----
pos_y0_x0    0.603  1.252
pos_y0_x1    0.701  1.252
pos_y0_x2    0.799  1.252
pos_y0_x3    0.897  1.252
pos_y1_x0    0.603  1.35
pos_y1_x1    0.701  1.35
pos_y1_x2    0.799  1.35
pos_y1_x3    0.897  1.35
pos_y2_x0    0.603  1.448
pos_y2_x1    0.701  1.448
pos_y2_x2    0.799  1.448
pos_y2_x3    0.897  1.448

type: robot           x        y         z    fingers    tilt      wrist
-------------  --------  -------  --------  ---------  ------  ---------
robot          0.749646  1.34545  0.942412  0.0263505  1.5708  0.0201273

type: rot      angle
-----------  -------
rot_-135        -135
rot_-45          -45
rot_-90          -90
rot_0              0
rot_135          135
rot_180          180
rot_45            45
rot_90            90
###################################################################################################

INFO: Task 48 / 50: Policy/Env encountered an exception
INFO: [main.py] Solving task 49/50...
DEBUG: [main.py] Solving task w. goal: {Toppled(domino_3:domino)}
INFO: [CogMan] Reset called.
DEBUG: Loaded model response from pretrained_model_cache/OpenRouter-google/gemini-2.5-pro_most_likely_1_None_-769831615136490837/prompt.txt.
INFO: Executing policy...
DEBUG: Executing option Pick(robot, domino_1)
DEBUG: Executing option Place(robot, domino_1, domino_0, pos_y1_x1, rot_90)
DEBUG: Executing option Pick(robot, domino_2)
DEBUG: Executing option Place(robot, domino_2, domino_1, pos_y1_x2, rot_90)
DEBUG: Executing option Push(robot, domino_0)
DEBUG: [CogMan] State at the exception: {AdjacentTo(pos_y1_x3:loc, domino_3:domino), MovableBlock(domino_1:domino), AdjacentTo(pos_y1_x2:loc, domino_1:domino), InFrontDirection(domino_2:domino, domino_1:domino, straight:direction), InFront(domino_1:domino, domino_2:domino), HandEmpty(robot:robot), PosClear(pos_y0_x1:loc), AdjacentTo(pos_y1_x1:loc, domino_2:domino), PosClear(pos_y1_x3:loc), DominoAtRot(domino_0:domino, rot_90:rot), DominoAtPos(domino_1:domino, pos_y1_x1:loc), AdjacentTo(pos_y2_x2:loc, domino_3:domino), DominoAtPos(domino_3:domino, pos_y2_x3:loc), DominoAtRot(domino_3:domino, rot_90:rot), AdjacentTo(pos_y1_x3:loc, domino_2:domino), AdjacentTo(pos_y1_x1:loc, domino_0:domino), AdjacentTo(pos_y2_x1:loc, domino_1:domino), PosClear(pos_y1_x0:loc), Toppled(domino_0:domino), InFrontDirection(domino_1:domino, domino_2:domino, straight:direction), StartBlock(domino_0:domino), AdjacentTo(pos_y2_x2:loc, domino_2:domino), MovableBlock(domino_2:domino), PosClear(pos_y2_x0:loc), DominoAtRot(domino_2:domino, rot_90:rot), InFront(domino_2:domino, domino_1:domino), AdjacentTo(pos_y0_x2:loc, domino_2:domino), DominoAtPos(domino_2:domino, pos_y1_x2:loc), AdjacentTo(pos_y2_x2:loc, domino_0:domino), PosClear(pos_y2_x2:loc), Upright(domino_2:domino), DominoAtRot(domino_1:domino, rot_90:rot), AdjacentTo(pos_y0_x1:loc, domino_1:domino), DominoAtPos(domino_0:domino, pos_y2_x1:loc), Upright(domino_1:domino), PosClear(pos_y0_x0:loc), PosClear(pos_y0_x2:loc), AdjacentTo(pos_y1_x0:loc, domino_1:domino), Upright(domino_3:domino), AdjacentTo(pos_y2_x0:loc, domino_0:domino), PosClear(pos_y0_x3:loc)}
INFO: Approach failed at execution time with error: Option plan exhausted!
DEBUG: Final abstract state:
{AdjacentTo(pos_y1_x3:loc, domino_3:domino), MovableBlock(domino_1:domino), AdjacentTo(pos_y1_x2:loc, domino_1:domino), InFrontDirection(domino_2:domino, domino_1:domino, straight:direction), InFront(domino_1:domino, domino_2:domino), HandEmpty(robot:robot), PosClear(pos_y0_x1:loc), AdjacentTo(pos_y1_x1:loc, domino_2:domino), PosClear(pos_y1_x3:loc), DominoAtRot(domino_0:domino, rot_90:rot), DominoAtPos(domino_1:domino, pos_y1_x1:loc), AdjacentTo(pos_y2_x2:loc, domino_3:domino), DominoAtPos(domino_3:domino, pos_y2_x3:loc), DominoAtRot(domino_3:domino, rot_90:rot), AdjacentTo(pos_y1_x3:loc, domino_2:domino), AdjacentTo(pos_y1_x1:loc, domino_0:domino), AdjacentTo(pos_y2_x1:loc, domino_1:domino), Toppled(domino_0:domino), PosClear(pos_y1_x0:loc), InFrontDirection(domino_1:domino, domino_2:domino, straight:direction), StartBlock(domino_0:domino), AdjacentTo(pos_y2_x2:loc, domino_2:domino), MovableBlock(domino_2:domino), PosClear(pos_y2_x0:loc), DominoAtRot(domino_2:domino, rot_90:rot), InFront(domino_2:domino, domino_1:domino), AdjacentTo(pos_y0_x2:loc, domino_2:domino), DominoAtPos(domino_2:domino, pos_y1_x2:loc), AdjacentTo(pos_y2_x2:loc, domino_0:domino), Upright(domino_2:domino), PosClear(pos_y2_x2:loc), DominoAtRot(domino_1:domino, rot_90:rot), AdjacentTo(pos_y0_x1:loc, domino_1:domino), DominoAtPos(domino_0:domino, pos_y2_x1:loc), Upright(domino_1:domino), PosClear(pos_y0_x0:loc), PosClear(pos_y0_x2:loc), AdjacentTo(pos_y1_x0:loc, domino_1:domino), Upright(domino_3:domino), AdjacentTo(pos_y2_x0:loc, domino_0:domino), PosClear(pos_y0_x3:loc)}
DEBUG: Final state:
############################################ STATE ############################################
type: direction      dir
-----------------  -----
left                   1
right                  2
straight               0

type: domino           x        y         z      rot          tilt     r     g     b    is_held
--------------  --------  -------  --------  -------  ------------  ----  ----  ----  ---------
domino_0        0.699871  1.44804  0.407485  1.57205   1.57087      0.56  0.93  0.56          0
domino_1        0.696872  1.34677  0.474993  1.59107   0.000253717  0.6   0.8   1             0
domino_2        0.795135  1.35655  0.47499   1.61614   2.78092e-05  0.6   0.8   1             0
domino_3        0.896993  1.448    0.47499   1.5708   -9.27772e-05  1     0.75  0.8           0

type: loc       xx     yy
-----------  -----  -----
pos_y0_x0    0.603  1.252
pos_y0_x1    0.701  1.252
pos_y0_x2    0.799  1.252
pos_y0_x3    0.897  1.252
pos_y1_x0    0.603  1.35
pos_y1_x1    0.701  1.35
pos_y1_x2    0.799  1.35
pos_y1_x3    0.897  1.35
pos_y2_x0    0.603  1.448
pos_y2_x1    0.701  1.448
pos_y2_x2    0.799  1.448
pos_y2_x3    0.897  1.448

type: robot           x        y         z    fingers    tilt    wrist
-------------  --------  -------  --------  ---------  ------  -------
robot          0.750158  1.34886  0.944475  0.0263529  1.5708   3.1445

type: rot      angle
-----------  -------
rot_-135        -135
rot_-45          -45
rot_-90          -90
rot_0              0
rot_135          135
rot_180          180
rot_45            45
rot_90            90
###############################################################################################

INFO: Task 49 / 50: Policy/Env encountered an exception
INFO: [main.py] Solving task 50/50...
DEBUG: [main.py] Solving task w. goal: {Toppled(domino_3:domino)}
INFO: [CogMan] Reset called.
DEBUG: Loaded model response from pretrained_model_cache/OpenRouter-google/gemini-2.5-pro_most_likely_1_None_-769831615136490837/prompt.txt.
INFO: Executing policy...
DEBUG: Executing option Pick(robot, domino_1)
DEBUG: Executing option Place(robot, domino_1, domino_0, pos_y1_x1, rot_90)
DEBUG: Executing option Pick(robot, domino_2)
DEBUG: Executing option Place(robot, domino_2, domino_1, pos_y1_x2, rot_90)
DEBUG: Executing option Push(robot, domino_0)
DEBUG: [CogMan] State at the exception: {AdjacentTo(pos_y0_x3:loc, domino_3:domino), AdjacentTo(pos_y0_x3:loc, domino_0:domino), DominoAtPos(domino_0:domino, pos_y0_x2:loc), MovableBlock(domino_1:domino), AdjacentTo(pos_y1_x2:loc, domino_0:domino), DominoAtRot(domino_3:domino, rot_180:rot), AdjacentTo(pos_y0_x1:loc, domino_0:domino), AdjacentTo(pos_y1_x2:loc, domino_1:domino), InFrontDirection(domino_2:domino, domino_1:domino, straight:direction), InFront(domino_1:domino, domino_2:domino), HandEmpty(robot:robot), PosClear(pos_y0_x1:loc), AdjacentTo(pos_y1_x1:loc, domino_2:domino), DominoAtRot(domino_0:domino, rot_90:rot), DominoAtPos(domino_1:domino, pos_y1_x1:loc), PosClear(pos_y2_x3:loc), AdjacentTo(pos_y1_x2:loc, domino_3:domino), AdjacentTo(pos_y1_x3:loc, domino_2:domino), AdjacentTo(pos_y2_x1:loc, domino_1:domino), PosClear(pos_y1_x0:loc), Toppled(domino_0:domino), InFrontDirection(domino_1:domino, domino_2:domino, straight:direction), StartBlock(domino_0:domino), AdjacentTo(pos_y2_x2:loc, domino_2:domino), MovableBlock(domino_2:domino), PosClear(pos_y2_x0:loc), DominoAtRot(domino_2:domino, rot_90:rot), InFront(domino_2:domino, domino_1:domino), AdjacentTo(pos_y0_x2:loc, domino_2:domino), DominoAtPos(domino_2:domino, pos_y1_x2:loc), PosClear(pos_y2_x2:loc), AdjacentTo(pos_y2_x3:loc, domino_3:domino), DominoAtRot(domino_1:domino, rot_90:rot), AdjacentTo(pos_y0_x1:loc, domino_1:domino), DominoAtPos(domino_3:domino, pos_y1_x3:loc), PosClear(pos_y0_x0:loc), Upright(domino_1:domino), Toppled(domino_2:domino), AdjacentTo(pos_y1_x0:loc, domino_1:domino), PosClear(pos_y2_x1:loc), Upright(domino_3:domino), PosClear(pos_y0_x3:loc)}
INFO: Approach failed at execution time with error: Option plan exhausted!
DEBUG: Final abstract state:
{AdjacentTo(pos_y0_x3:loc, domino_3:domino), AdjacentTo(pos_y0_x3:loc, domino_0:domino), DominoAtPos(domino_0:domino, pos_y0_x2:loc), MovableBlock(domino_1:domino), AdjacentTo(pos_y1_x2:loc, domino_0:domino), DominoAtRot(domino_3:domino, rot_180:rot), AdjacentTo(pos_y0_x1:loc, domino_0:domino), AdjacentTo(pos_y1_x2:loc, domino_1:domino), InFrontDirection(domino_2:domino, domino_1:domino, straight:direction), InFront(domino_1:domino, domino_2:domino), HandEmpty(robot:robot), PosClear(pos_y0_x1:loc), AdjacentTo(pos_y1_x1:loc, domino_2:domino), DominoAtRot(domino_0:domino, rot_90:rot), DominoAtPos(domino_1:domino, pos_y1_x1:loc), PosClear(pos_y2_x3:loc), AdjacentTo(pos_y1_x2:loc, domino_3:domino), AdjacentTo(pos_y1_x3:loc, domino_2:domino), AdjacentTo(pos_y2_x1:loc, domino_1:domino), Toppled(domino_0:domino), PosClear(pos_y1_x0:loc), InFrontDirection(domino_1:domino, domino_2:domino, straight:direction), StartBlock(domino_0:domino), AdjacentTo(pos_y2_x2:loc, domino_2:domino), MovableBlock(domino_2:domino), PosClear(pos_y2_x0:loc), DominoAtRot(domino_2:domino, rot_90:rot), InFront(domino_2:domino, domino_1:domino), AdjacentTo(pos_y0_x2:loc, domino_2:domino), DominoAtPos(domino_2:domino, pos_y1_x2:loc), PosClear(pos_y2_x2:loc), AdjacentTo(pos_y2_x3:loc, domino_3:domino), DominoAtRot(domino_1:domino, rot_90:rot), AdjacentTo(pos_y0_x1:loc, domino_1:domino), DominoAtPos(domino_3:domino, pos_y1_x3:loc), PosClear(pos_y0_x0:loc), Upright(domino_1:domino), Toppled(domino_2:domino), AdjacentTo(pos_y1_x0:loc, domino_1:domino), PosClear(pos_y2_x1:loc), Upright(domino_3:domino), PosClear(pos_y0_x3:loc)}
DEBUG: Final state:
############################################ STATE ############################################
type: direction      dir
-----------------  -----
left                   1
right                  2
straight               0

type: domino           x        y         z      rot          tilt     r     g     b    is_held
--------------  --------  -------  --------  -------  ------------  ----  ----  ----  ---------
domino_0        0.797307  1.25142  0.407487  1.57034   1.57079      0.56  0.93  0.56          0
domino_1        0.695     1.35724  0.47499   1.58839   5.08402e-05  0.6   0.8   1             0
domino_2        0.837501  1.39069  0.474889  1.61613   0.211793     0.6   0.8   1             0
domino_3        0.89715   1.35     0.47499   3.14159  -4.03105e-05  1     0.75  0.8           0

type: loc       xx     yy
-----------  -----  -----
pos_y0_x0    0.603  1.252
pos_y0_x1    0.701  1.252
pos_y0_x2    0.799  1.252
pos_y0_x3    0.897  1.252
pos_y1_x0    0.603  1.35
pos_y1_x1    0.701  1.35
pos_y1_x2    0.799  1.35
pos_y1_x3    0.897  1.35
pos_y2_x0    0.603  1.448
pos_y2_x1    0.701  1.448
pos_y2_x2    0.799  1.448
pos_y2_x3    0.897  1.448

type: robot           x        y         z    fingers    tilt    wrist
-------------  --------  -------  --------  ---------  ------  -------
robot          0.750327  1.34532  0.942012  0.0263529  1.5708  3.14408

type: rot      angle
-----------  -------
rot_-135        -135
rot_-45          -45
rot_-90          -90
rot_0              0
rot_135          135
rot_180          180
rot_45            45
rot_90            90
###############################################################################################

INFO: Task 50 / 50: Policy/Env encountered an exception
INFO: Tasks solved: 0 / 50
INFO: Average time for successes: inf seconds
INFO: Test results: defaultdict(<class 'float'>, {'num_solved': 0, 'num_total': 50, 'avg_suc_time': inf, 'avg_ref_cost': inf, 'min_num_samples': 0, 'max_num_samples': 0.0, 'min_skeletons_optimized': 0, 'max_skeletons_optimized': 0.0, 'num_solve_timeouts': 0, 'num_solve_failures': 0, 'num_execution_timeouts': 0, 'num_execution_failures': 49, 'avg_num_samples': 0.0, 'avg_num_skeletons_optimized': 0.0, 'avg_num_nodes_expanded': 0.0, 'avg_num_nodes_created': 0.0, 'avg_num_nsrts': 0.0, 'avg_num_preds': 0.0, 'avg_plan_length': 0.0, 'avg_num_failures_discovered': 0.0, 'num_offline_transitions': 0, 'num_online_transitions': 0.0, 'query_cost': 0.0, 'learning_time': 2.8250040486454964e-05})
INFO: Wrote out test results to results/pybullet_domino__vlm_open_loop__0______domino-vlm_plan__None.pkl
INFO: 

ONLINE LEARNING CYCLE 0

INFO: Did not receive any interaction requests, terminating
INFO: 

Main script terminated in 165.90235 seconds

[yichao-liang]

Sorry for the delay in getting back. Could we please separate the PR to two, with one just on extending VLM planning and the other on formatting, dependencies, and related fixes? Thanks!