Thanks for your excellent work!!! I noticed your "outlook attention" is very similar with the "Involution"(https://github.com/d-li14/involution).I just want to know the main difference. As I see, the main difference is that you use a extra linear projection on the input iteself and you use a extra softmax to generate the attention weight. However I did not find the detailed comparation between these two methods in your paper.

Hi Creators, Thanks for making a new SOTA models & also for open-sourcing it.

I was trying the colab notebook and it's throwing an error!!

After going through the Usage and adding those params. It gave me different and wrong results compared to demo colab Notebook.

This is a great job that proposes a new attention way. However, I want to figure out its ability when comparing all the things in the same condition.

Could you please release an ablation study that compares the outlook and attention under the same training policy, hyperparameters (network width, depth), and architectures (for example ViT or LV-ViT)?

So that we can better know the effectiveness of the outlook.

Hello, thanks very much for sharing the code for your tremendous research!

For semantic segmentation, did you just run evaluation with multiple square tiles to handle the non-square resolution of Cityscapes? Can you share any details, like decoder head architecture?

Hello,

I am finetuning a model with an image size of 512 and --token-label-size 24. Is the label size enough for 512 image size? Should I use a higher label size? How do I really know the correct label size?

Thank you in advance!

I was trying to train VOLO on some custom data using the following command

!python3 main.py -data_dir "/content/dataset" --dataset "ImageFolder" --train-split "train" --val-split "valid" --num-classes 3 --epochs 100 --batch-size 64

Unfortunately, I keep on getting the following error:

Traceback (most recent call last):
  File "main.py", line 948, in <module>
    main()
  File "main.py", line 664, in main
    optimizers=optimizers)
  File "main.py", line 783, in train_one_epoch
    loss = loss_fn(output, target)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/timm/loss/cross_entropy.py", line 35, in forward
    loss = torch.sum(-target * F.log_softmax(x, dim=-1), dim=-1)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py", line 1768, in log_softmax
    ret = input.log_softmax(dim)
AttributeError: 'tuple' object has no attribute 'log_softmax'

Any advice on how I can go about fixing this or what is causing this error to occur?

Hi,

when I used volo_d1 pretrained model for my training. I loaded this model successfully. But I received this error when I do the training step. Could you please give me some hint? Here is the input before forward() function I traced. The original one is resnet18, it worked fine. d1_224.txt resnet18.txt

Thank you in advance.

Best regards, Hui Yu

When I use the code to use the pre-training model for training, I find that the data becomes tuple data when passing through the model, resulting in the model can not continue training。 model = volo_d2() load_pretrained_weights( model,'./path/to/pretrained/weights/d2_224_85.2.pth.tar', use_ema=False, strict=False,num_classes=2) print(model) ...... def train(train_loader,model,criterion, optimizer, epoch,args,scheduler=None): print("train--------------") avg_loss=0 avg_acc=0 model.train() for batch_idx,(image,target) in enumerate(train_loader): # measure data loading time image,target=Variable(image.cuda()),Variable(target.cuda()) print(type(image)) image=image.cuda() target=target.cuda() optimizer.zero_grad() logits =model(image)

    #m = [t.cpu().numpy() for t in logits]
    #m = [o.cpu().detach() for o in m]
    #logits = torch.tensor(m)
    #logits = torch.tensor([item.cpu().detach().numpy() for item in logits]).cuda()

    print(type(logits))

preds=logits.max(1, keepdim=True)[1] # get the index of the max log-probability

AttributeError: 'tuple' object has no attribute 'max'

in class OutlookAttention, there is self.v = nn.Linear(dim, dim, bias=qkv_bias) and the input of this class is x whose shape is B, H, W, C = x.shape. My quesion is how this code v = self.v(x).permute(0, 3, 1, 2) # B, C, H, W can go well without exception because matrix multiplication [B, H, W, C] * [dim, dim] will do here. And also in the original paper, Algorithm 1 implements v_pj = nn.Linear(C, C). But in your codes, C is replaced with dim. Thanks!

When I run the main program with "python main. py . /data", the following error occurs:

D:\Python36\lib\site-packages\torchvision\transforms\transforms.py:258: UserWarning: Argument interpolation should be of type InterpolationMode instead of int. Please, use InterpolationMode enum. "Argument interpolation should be of type InterpolationMode instead of int. " Traceback (most recent call last): File "main.py", line 948, in main() File "main.py", line 664, in main optimizers=optimizers) File "main.py", line 746, in train_one_epoch for batch_idx, (input, target) in enumerate(loader): File "D:\Python36\lib\site-packages\tlt\data\loader.py", line 105, in iter yield input, target UnboundLocalError: local variable 'input' referenced before assignment

Please give me more advice and look forward to your reply. thank you very much.

A problem happen when I load the pretrained weight you provided.

UnpicklingError Traceback (most recent call last) in 9 # as we interpolate the position embeding for different image size. 10 load_pretrained_weights(model, "/home/featurize/work/checkpoints/archive/data.pkl", use_ema=False, ---> 11 strict=False, num_classes=1000)

/cloud/volo/utils/utils.py in load_pretrained_weights(model, checkpoint_path, use_ema, strict, num_classes) 140 num_classes=1000): 141 '''load pretrained weight for VOLO models''' --> 142 state_dict = load_state_dict(checkpoint_path, model, use_ema, num_classes) 143 model.load_state_dict(state_dict, strict=strict) 144

/cloud/volo/utils/utils.py in load_state_dict(checkpoint_path, model, use_ema, num_classes) 92 if checkpoint_path and os.path.isfile(checkpoint_path): 93 # checkpoint = torch.load(checkpoint_path, map_location='cpu') ---> 94 checkpoint = torch.load(checkpoint_path) 95 state_dict_key = 'state_dict' 96 if isinstance(checkpoint, dict):

/environment/python/versions/miniconda3-4.7.12/lib/python3.7/site-packages/torch/serialization.py in load(f, map_location, pickle_module, **pickle_load_args) 591 return torch.jit.load(opened_file) 592 return _load(opened_zipfile, map_location, pickle_module, **pickle_load_args) --> 593 return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args) 594 595

/environment/python/versions/miniconda3-4.7.12/lib/python3.7/site-packages/torch/serialization.py in _legacy_load(f, map_location, pickle_module, **pickle_load_args) 760 "functionality.") 761 --> 762 magic_number = pickle_module.load(f, **pickle_load_args) 763 if magic_number != MAGIC_NUMBER: 764 raise RuntimeError("Invalid magic number; corrupt file?")

UnpicklingError: A load persistent id instruction was encountered, but no persistent_load function was specified.

First, thanks for your contribution, this paper inspired me a lot. However, I still have some questions as follows, I hope you can answer:

1. about the code, I think the output shape of Unfold operation is not right in your code, even if it has no error
2. about the ablation study, it would be better to compare the dynamic convolution with your outlook attention. they are very similar exactly with each other, the only difference is the weights generation method. I am very interested in this.
3. according to your paper, I modified your code with my own understanding: https://github.com/xingshulicc/Vision-In-Transformer-Model/blob/main/outlook_attention.py.
   Hoping you can give me some advice on my code. Thanks again.

Greetings! Thanks for all your inspiring and excellent VOLO work!!! In reading this paper, I get trouble in comprehending the formulation (8), which depicts the complexity of Outlooker Attention. I tried to inference the cost from the pytorch-like code provided aforementioned, however cannot get to the formulation (8). Would you mind providing any insight about the calculation process? Thanks a lot.

I download pre trained model with the link in the document. but when I try to use it, it can't report an error:

[email protected]:~/workspace/DeepLearning/VOLO/pretrained_models$ tar -xf d1_384_85.2.pth.tar tar: This does not look like a tar archive tar: Skipping to next header tar: Exiting with failure status due to previous errors

Hi,

Congratulations on your excellent work and many thanks for making the code public. I have trained a model using the base settings and no token labels:

CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 ./distributed_train.sh 8 /path/to/imagenet --model volo_d1 --img-size 224 -b 128 --lr 1.6e-3 --drop-path 0.1 --apex-amp

which reached best accuracy 81.72% after 310 epochs. I believe the expected best acc should be about 83.8% which is quite higher than what I get at the moment.

Can you see any issue with the command used to train the model? Any help would be really appreciated.

Best, Michael

AMP not enabled. Training in float32. Using native Torch DistributedDataParallel. Scheduled epochs: 310 /pytorch/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:312: operator(): block: [0,0,0], thread: [15,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed. Traceback (most recent call last): File "main.py", line 948, in main() File "main.py", line 664, in main optimizers=optimizers) File "main.py", line 782, in train_one_epoch output = model(input) File "/root/miniconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl result = self.forward(*input, **kwargs) File "/root/miniconda3/lib/python3.7/site-packages/torch/nn/parallel/distributed.py", line 610, in forward self._sync_params() File "/root/miniconda3/lib/python3.7/site-packages/torch/nn/parallel/distributed.py", line 1048, in _sync_params authoritative_rank, File "/root/miniconda3/lib/python3.7/site-packages/torch/nn/parallel/distributed.py", line 979, in _distributed_broadcast_coalesced self.process_group, tensors, buffer_size, authoritative_rank RuntimeError: CUDA error: device-side assert triggered terminate called after throwing an instance of 'std::runtime_error' what(): NCCL error in: /pytorch/torch/lib/c10d/../c10d/NCCLUtils.hpp:136, unhandled cuda error, NCCL version 2.7.8

Dear authors: Thanks for your wonderful work! the result in Semantic Segmentation task looks nice, thus, could the code and config about the Semantic Segmentation be published? Thanks!