Hi, thanks for your great work. I have a question about loss. When i trained model for my data, the loss < 0 at epoch_0, is this normal? Config: baseline.yaml in the project

Hi,

Thank you for sharing this great work!

When I ran the training code, I got NaN for all losses after about 1 epoch of training. This problem is reproduced whenever I run the training code. (I have tested it three times.)

I followed the same environment setting with anaconda, and also used the same hyper-parameters. (The only difference is that our PyTorch version is 1.7.1 and yours is 1.7.0, and all other modules are the same as yours.)

Please share your idea about this problem, if you have any. Thanks!

Congratulations on your great work!

I want to follow your work for future research and I have some questions about your released code below:

In the fiery.py file of your code, can you provide more details about the get_geometry function and the projection_to_birds_eye_view function? I'm so confused about how they actually work, especially, the code shown in the red box below.

Thank you very much. Looking forward to your reply!

Hello, recently I found your great work and I want to try the "Visualisation" part locally to check the results, but after I run the command of python visualise.py --checkpoint ${CHECKPOINT_PATH} my terminal pop out an error like the following:

I try to solve it by searching on google but it does not help, could you help me if you know how to solve it. Many thanks.

I dont see anywhere that the checkpoint is getting saved and while resuming the training, I am getting an error that "size mismatch for model.temporal_model.model.1.aggregation.0.conv.weight"

If I want to get the result of Fiery Static of Setting2 in Table I of your paper, should I use the config in "configs/single_timeframe.yml"? When I train the network using this config file from scratch, the IOU is 39.2 when I use the "evaluate.py". However, in the paper, the result is 35.8. Is there another parameter needed to be modified, when I want the network to be one frame as input and the segmentation result of the present frame as output?

Hi,

Would you be able to explain how the panoptic_metrics function works? (Code linked here: https://github.com/wayveai/fiery/blob/master/fiery/metrics.py#L137) Especially, I wonder why 'void' is included for 'combine_mask', and why 'background' should be changed from 0 to 1.

Also, It is hard to understand the code under the comment "# hack for bincounting 2 arrays together". (Code linked here: https://github.com/wayveai/fiery/blob/master/fiery/metrics.py#L168)

Thank you!

Thanks for your great work. I have a little question about future ego pose during inference? I may find a little tricky because flow prediction is a module before motion planning. In real cases, the flow prediction module has no chance of getting future ego pose. But the code may show future ego pose is irreplacable in inference. When I turn to None, the inference doesn't work.

Hello and many thanks for your work and sharing your code.

I have a question regarding the way you compute your IoU metric and how it compares against Lift-splat.

You use stat_scores_multiple_classes from PLmetrics to compute the iou. Correct me if I am wrong, but by default the threshold of this method is 0.5

On the other hand, in get_batch_iou of LFS they use a threshold of 0: pred = (preds > 0) https://github.com/nv-tlabs/lift-splat-shoot/blob/master/src/tools.py

Wouldn't this have an impact on the evaluation results ,and thus, on how you compare to them ?

Hi, I couldn't understand how the self.downsample parameter was set (why 8 and 16 and how it affects upsampling_in_channels) and why delete_unused_layers is required in the encoder model. I tried to search the efficientnet-pytorch implementation and couldn't find any reference for this operation. Could you explain briefly why this is required? Thank you!

Thanks for your excellent work! I have some questions about instance_flow. warped_instance_seg = {} # t0,f01-->t1; t1,f12-->t2; t2,f23-->t3 # t1,f10-->t0; t2,f21-->t1 for t in range(1, seq_len): warped_inst_t = warp_features(instance_img[t].unsqueeze(0).unsqueeze(1).float(), # 1, 1, 200, 200 future_egomotion_inv[t - 1].unsqueeze(0), mode='nearest', spatial_extent=spatial_extent) warped_instance_seg[t] = warped_inst_t[0, 0] In your paper, "Finally, we obtain feature flow labels by comparing the position of the instance centers of gravity between two consecutive timesteps".I think the code should convert t to t-1, not t-1 to t. How can it get the feature flow? I'm really confuesd about it. I'm looking forward your replying.

Hi. Thanks for your great work. I followed your instructions in README.md to extract nuscenes dataset. I ran evaluate.py with official pretrained checkpoint (https://github.com/wayveai/fiery/releases/download/v1.0/fiery.ckpt) but got the output as follows: iou 53.5 & 28.6 pq 39.8 & 18.0 sq 69.4 & 66.3 rq 57.4 & 27.1 Is there something wrong? It seems to be much lower than the results you got.

Dear author, I just run the code without no change, during the training ,I got the total sum loss with the value <0 .

It looks so weird. Is that caused by the setting of the "uncertainty" ? Is that normal? Really thanks.

Thanks for your great work. I'd like to reproduce the training process, but I encountered an error. That is when I use multi-GPU distributed training process, the logging information seems normal, but afterwards the remote server stuck and connection reset and finally the process is killed. My remote server is an independent machine with 4xRTX3090. Is there any issues with the pytorch lightning distributed training that may cause my failure?