Hi! When I run the nuplan_framework.ipynb notebook, I can get all the models to train but not shows up when I launch Nuboard. I get some messages when I train and simulate that maybe suggest the error?

"The agent on node Blade-15 failed to be restarted 5 times. There are 3 possible problems if you see this error.

1. The dashboard might not display correct information on this node.
2. Metrics on this node won't be reported.
3. runtime_env APIs won't work. Check out the dashboard_agent.log to see the detailed failure messages." dashboard_agent.log

This is what appears when I try to launch Nuboard. It seems that there are two experiments, but no data shows up at all.

I know that Nuboard is under development, but I was wondering if I was supposed to be able to see anything at all? If not, where would I be able to find the raw data from simulation and training?

When I have installed all the packages from requirements.txt and run the nuplan_planner_tutorial. I met a problem. When I try to run launch simulation(Within the notebook) block, it seems like my kernel is always dying, because it has the error called: Error: Canceled future for execute_request message before replies were done. Is there someone knows how to solve this problem? Thank you in advance!

Hi,

Training the vector model results in the following error: RuntimeError: mat1 and mat2 shapes cannot be multiplied (0x128 and 134x128) in "lanegcn_utils.py", line 561

I managed to find out that the reason for this is, that for some scenarios the VectorMap.traffic_light_data is an empty list. I further noticed that this is the case if the number of lane_segments returned in get_neighbor_vector_map in "vector_map_feature_builder", line 133 is equal to 0.

For instance check scenario token 1e00d42ba8095cb3 in log 2021.09.15.11.49.23_veh-28_00693_01062

If model.vector_map_feature_radius is increased it works though.

The downloaded file from the newest dataset（nuplan-v1.1） seems to be the old one. The Md5 of nuplan Map in the version v1.1 is the same as the version v1.1, which is not as the page described.

Ex. What is a lane connector? or How many wheels do your vehicles have? Hi developer: I use the v0.6 devkit and successfully finished training and simulation. But Nuboard cannot display the simulation result, including overview score, histogram and scenarios visualization.

However, the v0.6 nuboard can display v0.3 simulation result(only score, no scenario visualization either)

here's my config:

• training

experiment_name=vector_experiment
py_func=train
+training=training_vector_model
worker=ray_distributed
scenario_builder=nuplan_mini
scenario_builder.data_root=***
lightning.trainer.params.max_epochs=60
data_loader.params.batch_size=3
data_loader.params.num_workers=8
data_loader.params.pin_memory=True
scenario_filter.limit_total_scenarios=10
+lightning.distributed_training.scale_lr=1e-4
optimizer=adamw
lightning.trainer.params.gradient_clip_val=0.3
cache.use_cache_without_dataset=True
cache.cache_path=***
lightning.trainer.checkpoint.resume_training=False

• simulation

+simulation=closed_loop_nonreactive_agents
experiment_name='simulation_vector_experiment'
scenario_builder.data_root=***
scenario_builder=nuplan_mini
scenario_filter=all_scenarios
scenario_filter.expand_scenarios=False
scenario_filter.num_scenarios_per_type=1
planner=ml_planner
model=vector_model
worker=ray_distributed
planner.ml_planner.model_config=${model}
planner.ml_planner.checkpoint_path=***/nuplan/exp/exp/vector_experiment/2022.10.09.15.25.35/best_model/epoch_13-step_349.ckpt

• nuboard

simulation_path="[***/nuplan/exp/exp/simulation_vector_experiment/2022.10.09.15.57.59]"
scenario_builder.data_root=***

I want to get the future tracked objects, so I should get a scenario first and ues get_future_tracked_objects() fuction. But the class AbstractPlanner only gives me the initialization and current_input, how can a get a scenario?

Problem: The simulation result on frame 0 looks normal, but the trajectory predicted on frame 1 is abnormal.

### train.py import os import hydra import tempfile from pathlib import Path from nuplan.planning.script.run_training import main as main_train

CONFIG_PATH = '../nuplan-devkit/nuplan/planning/script/config/training' CONFIG_NAME = 'default_training'

SAVE_DIR = Path(tempfile.gettempdir()) / 'tutorial_nuplan_framework' # optionally replace with persistent dir EXPERIMENT = 'vector' # vector or raster or others #LOG_DIR = str(SAVE_DIR / EXPERIMENT) LOG_DIR = str(SAVE_DIR + '/' + EXPERIMENT)

hydra.core.global_hydra.GlobalHydra.instance().clear() hydra.initialize(config_path=CONFIG_PATH)

cfg = hydra.compose(config_name=CONFIG_NAME, overrides=[ f'group={str(SAVE_DIR)}', f'cache_dir={str(SAVE_DIR)}/cache', f'experiment_name={EXPERIMENT}', 'log_config=true', 'py_func=train', '+training=training_vector_model', # vector model that consumes ego, agents and map vector layers and regresses the ego's trajectory 'resume_training=false', # load the model from the last epoch and resume training 'worker=single_machine_thread_pool', # ray_distributed, sequential, single_machine_thread_pool 'scenario_builder=nuplan_mini', # use nuplan or nuplan_mini database 'scenario_builder.nuplan.scenario_filter.limit_scenarios_per_type=500000', # Choose 500 scenarios to train with 'scenario_builder.nuplan.scenario_filter.subsample_ratio=1', # subsample scenarios from 20Hz (1.0) to 0.2Hz (0.01), 10Hz (0.5), 5Hz (0.25) 'lightning.trainer.params.accelerator=ddp', # ddp is not allowed in interactive environment, using ddp_spawn instead - this can bottleneck the data pipeline, it is recommended to run training outside the notebook 'lightning.trainer.params.precision=16', 'lightning.trainer.params.auto_scale_batch_size=false', 'lightning.trainer.params.auto_lr_find=false', 'lightning.trainer.params.gradient_clip_val=0.0', 'lightning.trainer.params.gradient_clip_algorithm=norm', 'lightning.trainer.params.accumulate_grad_batches=64', 'lightning.trainer.overfitting.enable=false', # run an overfitting test instead of traning 'lightning.optimization.optimizer.learning_rate=2e-4', 'lightning.trainer.params.max_epochs=25', 'lightning.trainer.params.gpus=8', 'data_loader.params.batch_size=3', 'data_loader.params.num_workers=48', ])

main_train(cfg)

simulation.py

import os import hydra import tempfile from pathlib import Path from nuplan.planning.script.run_simulation import main as main_simulation

CONFIG_PATH = '../nuplan-devkit/nuplan/planning/script/config/simulation' CONFIG_NAME = 'default_simulation'

SAVE_DIR = Path(tempfile.gettempdir()) / 'tutorial_nuplan_framework' # optionally replace with persistent dir

EXPERIMENT = 'vector'

last_experiment = sorted(os.listdir(LOG_DIR))[-1] train_experiment_dir = sorted(Path(LOG_DIR).iterdir())[-1] checkpoint = sorted((train_experiment_dir / 'checkpoints').iterdir())[-1]

MODEL_PATH = str(checkpoint).replace("=", "=")

PLANNER = 'ml_planner' # [simple_planner, ml_planner] #CHALLENGE = 'challenge_1_open_loop_boxes' # [challenge_1_open_loop_boxes, challenge_3_closed_loop_nonreactive_agents, challenge_4_closed_loop_reactive_agents] CHALLENGE = 'challenge_3_closed_loop_nonreactive_agents' # [challenge_1_open_loop_boxes, challenge_3_closed_loop_nonreactive_agents, challenge_4_closed_loop_reactive_agents] #CHALLENGE = 'challenge_4_closed_loop_reactive_agents' # [challenge_1_open_loop_boxes, challenge_3_closed_loop_nonreactive_agents, challenge_4_closed_loop_reactive_agents] DATASET_PARAMS = [ 'scenario_builder=nuplan_mini', # use nuplan mini database 'scenario_builder/nuplan/scenario_filter=all_scenarios', # initially select all scenarios in the database 'scenario_builder.nuplan.scenario_filter.scenario_types=[nearby_dense_vehicle_traffic, ego_at_pudo, ego_starts_unprotected_cross_turn, ego_high_curvature]', # select scenario types 'scenario_builder.nuplan.scenario_filter.limit_scenarios_per_type=10', # use 10 scenarios per scenario type 'scenario_builder.nuplan.scenario_filter.subsample_ratio=0.5', # subsample 20s scenario from 20Hz to 1Hz (0.05) ]

hydra.core.global_hydra.GlobalHydra.instance().clear() # reinitialize hydra if already initialized hydra.initialize(config_path=CONFIG_PATH)

cfg = hydra.compose(config_name=CONFIG_NAME, overrides=[ f'experiment_name={EXPERIMENT}', f'group={SAVE_DIR}', 'log_config=true', 'planner=ml_planner', 'model=vector_model', 'planner.model_config=${model}', # hydra notation to select model config f'planner.checkpoint_path={MODEL_PATH}', # this path can be replaced by the checkpoint of the model trained in the previous section f'+simulation={CHALLENGE}', *DATASET_PARAMS, ])

main_simulation(cfg)

parent_dir = Path(SAVE_DIR) / EXPERIMENT results_dir = list(parent_dir.iterdir())[0] # get the child dir nuboard_file_2 = [str(file) for file in results_dir.iterdir() if file.is_file() and file.suffix == '.nuboard'][0]

Question: I found that subsample_ratio affects the simulation. What does subsample_ratio means? What are the proper values of subsample_ratio during training and simulation?

Hi. I wonder about the dataset structure I can download them about v1.1 via https://www.nuscenes.org/nuplan#code

• Maps 1ea
• Mini Split 1ea
• Train Split 8ea boston 1 / pittsburgh 1 / Las vegas 6
• val split 1ea
• test split 1ea

With https://nuplan-devkit.readthedocs.io/en/latest/dataset_setup.html, the structure is explained, but I couldn't follow it exactly with downloaed files.

What is the exp? What is the trainval?

Also I tried to run some tutorial. But when "nuplan_scenario_visualization.ipynb" run, it failed.

File /mnt/nuplan/nuplan-devkit/nuplan/database/nuplan_db/query_session.py:21, in execute_many(query_text, query_parameters, db_file) 18 cursor = connection.cursor() 20 try: ---> 21 cursor.execute(query_text, query_parameters) 23 for row in cursor: 24 yield row

OperationalError: near "NULLS": syntax error

Could you help me about the problem I got?

The training crashes right after the last epoch when I run it from a python script.

The only error I get is free(): invalid pointer

I am using the vector model, nuplan_mini, and ddp accelerator

Can you give me a hint where to search for an error?

Hi! I am trying to use the data of nuplan directly, including the map data. But I do not know the meaning of fields in the dataset, such as 'left_has_reflectors' which is from SemanticMapLayer.LANE of the map. Is there any documents for helping use the dataset? Thanks a lot!

Hi developers: I run the docker container, but it takes 13 hours and still doesn't finished. The information displayed is shown in the figure below. How long does it take through a complete simulation within docker?

My command: create the image

docker build --network host -f Dockerfile.submission . -t nuplan-evalservice-server:test.contestant

run the container

docker run --name nuplan-evalservice-server -d -v ./:/nuplan_devkit -p 9902:9902 nuplan-evalservice-server:test.contestant

Dear Motional nuplan Team,

Firstly congratulations to the work you have done!!! Working with nuplan is very interesting.

I have followed the instructions and set up the devkit. I am trying to run the tutorials. I am having issues at the last part of the tutorials. Nuboard is not getting launched. I am getting 500: Internal Server Error.

I am posting the error message from the log below. I would be very thankful to you, if you can help me fix this issue. Anyone else who came across the same issues, kindly share the solution to resolve it.

Thanks in advance.

ERROR:tornado.application:Uncaught exception GET / (127.0.0.1) HTTPServerRequest(protocol='http', host='localhost:5006', method='GET', uri='/', version='HTTP/1.1', remote_ip='127.0.0.1') Traceback (most recent call last): File "/home/divya/miniconda3/envs/nuplan/lib/python3.9/site-packages/tornado/web.py", line 1713, in _execute result = await result File "/home/divya/miniconda3/envs/nuplan/lib/python3.9/site-packages/bokeh/server/views/doc_handler.py", line 54, in get session = await self.get_session() File "/home/divya/miniconda3/envs/nuplan/lib/python3.9/site-packages/bokeh/server/views/session_handler.py", line 145, in get_session session = await self.application_context.create_session_if_needed(session_id, self.request, token) File "/home/divya/miniconda3/envs/nuplan/lib/python3.9/site-packages/bokeh/server/contexts.py", line 242, in create_session_if_needed self._application.initialize_document(doc) File "/home/divya/miniconda3/envs/nuplan/lib/python3.9/site-packages/bokeh/application/application.py", line 192, in initialize_document h.modify_document(doc) File "/home/divya/miniconda3/envs/nuplan/lib/python3.9/site-packages/bokeh/application/handlers/function.py", line 143, in modify_document self._func(doc) File "/home/divya/nuplan-devkit/nuplan/planning/nuboard/nuboard.py", line 113, in main_page overview_tab = OverviewTab(doc=self._doc, experiment_file_data=experiment_file_data) File "/home/divya/nuplan-devkit/nuplan/planning/nuboard/tabs/overview_tab.py", line 33, in init super().init(doc=doc, experiment_file_data=experiment_file_data) File "/home/divya/nuplan-devkit/nuplan/planning/nuboard/base/base_tab.py", line 55, in init self.planner_checkbox_group.on_click(self._click_planner_checkbox_group) File "/home/divya/miniconda3/envs/nuplan/lib/python3.9/site-packages/bokeh/core/has_props.py", line 360, in getattr self._raise_attribute_error_with_matches(name, properties) File "/home/divya/miniconda3/envs/nuplan/lib/python3.9/site-packages/bokeh/core/has_props.py", line 368, in _raise_attribute_error_with_matches raise AttributeError(f"unexpected attribute {name!r} to {self.class.name}, {text} attributes are {nice_join(matches)}") AttributeError: unexpected attribute 'on_click' to CheckboxGroup, possible attributes are active, align, aspect_ratio, classes, context_menu, css_classes, disabled, flow_mode, height, height_policy, inline, js_event_callbacks, js_property_callbacks, labels, margin, max_height, max_width, min_height, min_width, name, resizable, sizing_mode, styles, stylesheets, subscribed_events, syncable, tags, visible, width or width_policy ERROR:tornado.access:500 GET / (127.0.0.1) 17.33ms

Hello, developers! Thank you for your data and dev-kit!

In "2021.10.15.02.36.56_veh-53_02020_02442.db" of Singapore split, the ego vehicle ran a route like this:

But the corresponding velocity_x among the way is like:

While the velocity_y is like:

To me, velocity_y is like noise rather than normal speed.

Here is my code:

velocity_x = []
velocity_y = []
for i in range(len(scenarios_list)):
    scenario = get_default_scenario_from_token(
        NUPLAN_DATA_ROOT, log_db, scenarios_list[i]["token"].hex(), NUPLAN_MAPS_ROOT, NUPLAN_MAP_VERSION
    )
    velocity_x.append(scenario.initial_ego_state.agent.velocity.x)
    velocity_y.append(scenario.initial_ego_state.agent.velocity.y)

Bumps setuptools from 62.3.3 to 65.5.1.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Bumps setuptools from 59.5.0 to 65.5.1.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Describe the bug

The simulation history sampling can counterintuitive. This can be best explained considering the following example.

Example

Assume a model with past_trajectory_sampling.time_horizon=1.9 and past_trajectory_sampling.num_poses=4. (Note, setting past_trajectory_sampling.time_horizon=2.0 is not possible, as described in this issue. Ideally, the sampled past poses should be at the following time steps: [-0.475s, -0.95s, -1.425s, -1.9s]. However, as the logs themselves are recorded with a fixed sample interval of 0.1s, these time steps are between the recorded frames.

Expected Behavior

In my opinion the following behavior would be intuitive: The time steps between the recorded frames are rounded to the nearest frame resulting in the following time steps: [-0.5s, -1.0s, -1.4s, -1.9s]. That way, the returned frames match the intended trajectory sampling as good as possible.

Actual Behavior

However, the following time steps are actually sampled: [-0.4s, -0.8s, -1.2s, -1.6s]. Apparently, this can drastically impact the performance of a planner that was trained with a past interval of 2.0s, as it will implicitly assume that the frames passed to it in simulation are sampled with the same interval.

Explanation

In simulation, the feature builders have access to the SimulationHistoryBuffer which contains a list of past observations and ego states. In order to extract the frames that correspond to the feature builders past trajectory sampling from this list, this function is used. This function used a fixed step size for all time steps which is calculated from the desired past horizon and the number of samples and then rounded down (see here). In the above example, this would result in 0.475s being rounded down to 0.4s.

Takeaway

While I see that a fixed sample interval may make sense, I suggest to reconsider this decision as it may have undesired side effects such as described in the example above. Maybe it would also be ok, to raise an error or a warning if the user sets the past_trajectory_sampling in a way that will not be applicable in simulation

Workaround

In order to not be affected by this issue, the following must hold for the past_trajectory_sampling past_trajectory_sampling.time_horizon = k * past_trajectory_sampling.num_poses * 0.1s, where k is a natural number. Also, past_trajectory_sampling.time_horizon must not exceed 1.9s as described in this issue.