The official implementation of the IEEE S&P`22 paper "SoK: How Robust is Deep Neural Network Image Classification Watermarking".

Overview

Watermark-Robustness-Toolbox - Official PyTorch Implementation

contact Python 3.6 PyTorch 1.3.1 cuDNN 10.1.2 Website shields.io GPLv3 license

This repository contains the official PyTorch implementation of the following paper to appear at IEEE Security and Privacy 2022:

SoK: How Robust is Deep Neural Network Image Classification Watermarking?
Nils Lukas, Edward Jiang, Xinda Li, Florian Kerschbaum
https://arxiv.org/abs/2108.04974

Abstract: Deep Neural Network (DNN) watermarking is a method for provenance verification of DNN models. Watermarking should be robust against watermark removal attacks that derive a surrogate model that evades provenance verification. Many watermarking schemes that claim robustness have been proposed, but their robustness is only validated in isolation against a relatively small set of attacks. There is no systematic, empirical evaluation of these claims against a common, comprehensive set of removal attacks. This uncertainty about a watermarking scheme's robustness causes difficulty to trust their deployment in practice. In this paper, we evaluate whether recently proposed watermarking schemes that claim robustness are robust against a large set of removal attacks. We survey methods from the literature that (i) are known removal attacks, (ii) derive surrogate models but have not been evaluated as removal attacks, and (iii) novel removal attacks. Weight shifting, transfer learning and smooth retraining are novel removal attacks adapted to the DNN watermarking schemes surveyed in this paper. We propose taxonomies for watermarking schemes and removal attacks. Our empirical evaluation includes an ablation study over sets of parameters for each attack and watermarking scheme on the image classification datasets CIFAR-10 and ImageNet. Surprisingly, our study shows that none of the surveyed watermarking schemes is robust in practice. We find that schemes fail to withstand adaptive attacks and known methods for deriving surrogate models that have not been evaluated as removal attacks. This points to intrinsic flaws in how robustness is currently evaluated. Our evaluation includes a discussion of the runtime of each attack to underpin their practical relevance. While none of the schemes is robust against all attacks, none of the attacks removes all watermarks. We show that attacks can be combined and find combined attacks that remove all watermarks. We show that watermarking schemes need to be evaluated against a more extensive set of removal attacks with a more realistic adversary model. Our source code and a complete dataset of evaluation results will be made publicly available, which allows to independently verify our conclusions.

Features

All watermarking schemes and removal attacks are configured for the image classification datasets CIFAR-10 (32x32 pixels, 10 classes) and ImageNet (224x224 pixels, 1k classes). We implemented the following watermarking schemes, sorted by their categories:

.. and the following removal attacks, sorted by their categories:

Get Started

At this point, the Watermark-Robustness-Toolbox project is not available as a standalone pip package, but we are working on allowing an installation via pip. We describe a manual installation and usage. First, install all dependencies via pip.

$ pip install -r requirements.txt

The following four main scripts provide the entire toolbox's functionality:

  • train.py: Pre-trains an unmarked neural network.
  • embed.py: Embeds a watermark into a pre-trained neural network.
  • steal.py: Performs a removal attack against a watermarked neural network.
  • decision_threshold.py: Computes the decision threshold for a watermarking scheme.

We use the mlconfig library to pass configuration hyperparameters to each script. Configuration files used in our paper for CIFAR-10 and ImageNet can be found in the configs/ directory. Configuration files store all hyperparameters needed to reproduce an experiment.

Step 1: Pre-train a Model on CIFAR-10

$ python train.py --config configs/cifar10/train_configs/resnet.yaml

This creates an outputs directory and saves a model file at outputs/cifar10/null_models/resnet/.

Step 2: Embed an Adi Watermark

$ python embed.py --wm_config configs/cifar10/wm_configs/adi.yaml \
                  --filename outputs/cifar10/null_models/resnet/best.pth

This embeds an Adi watermark into the pre-trained model from 'Example 1' and saves (i) the watermarked model and (ii) all data to read the watermark under outputs/cifar10/wm/adi/00000_adi/.

Step 3: Attempt to Remove a Watermark

$ python steal.py --attack_config configs/cifar10/attack_configs/ftal.yaml \
                  --wm_dir outputs/cifar10/wm/adi/00000_adi/

This runs the Fine-Tuning (FTAL) removal attack against the watermarked model and creates a surrogate model stored under outputs/cifar10/attacks/ftal/. The directory also contains human-readable debug files, such as the surrogate model's watermark and test accuracies.

Datasets

Our toolbox currently implements custom data loaders (class WRTDataLoader) for the following datasets.

  • CIFAR-10
  • ImageNet (needs manual download)
  • Omniglot (needs manual download)
  • Open Images (needs manual download)

Documentation

We are actively working on documenting the parameters of each watermarking scheme and removal attack. At this point, we can only refer to the method's source code (at wrt/defenses/ and wrt/attacks/). Soon we will host a complete documentation of all parameters, so stay tuned!

Contribute

We encourage authors of watermarking schemes or removal attacks to implement their methods in the Watermark-Robustness-Toolbox to make them publicly accessible in a unified framework. Our aim is to improve reproducibility which makes it easier to evaluate a scheme's robustness. Any contributions or suggestions for improvements are welcome and greatly appreciated. This toolbox is maintained as part of a university project by graduate students.

Reference

The codebase has been based off an early version of the Adversarial-Robustness-Tooblox.

Cite our paper

@InProceedings{lukas2022watermarkingsok,
  title={SoK: How Robust is Deep Neural Network Image Classification Watermarking?}, 
  author={Lukas, Nils and Jiang, Edward and Li, Xinda and Kerschbaum, Florian},
  year={2022},
  booktitle={IEEE Symposium on Security and Privacy}
}
Machine learning and Deep learning models, deploy on telegram (the best social media)

Semi Intelligent BOT The project involves : Classifying fake news Classifying objects such as aeroplane, automobile, bird, cat, deer, dog, frog, horse

MohammadReza Norouzi 5 Mar 06, 2022
Mail classification with tensorflow and MS Exchange Server (ham or spam).

Mail classification with tensorflow and MS Exchange Server (ham or spam).

Metin Karatas 1 Sep 11, 2021
BASH - Biomechanical Animated Skinned Human

We developed a method animating a statistical 3D human model for biomechanical analysis to increase accessibility for non-experts, like patients, athletes, or designers.

Machine Learning and Data Analytics Lab FAU 66 Nov 19, 2022
On the Complementarity between Pre-Training and Back-Translation for Neural Machine Translation (Findings of EMNLP 2021))

PTvsBT On the Complementarity between Pre-Training and Back-Translation for Neural Machine Translation (Findings of EMNLP 2021) Citation Please cite a

Sunbow Liu 10 Nov 25, 2022
Code for "Learning Graph Cellular Automata"

Learning Graph Cellular Automata This code implements the experiments from the NeurIPS 2021 paper: "Learning Graph Cellular Automata" Daniele Grattaro

Daniele Grattarola 37 Oct 26, 2022
Implementation of CSRL from the AAAI2022 paper: Constraint Sampling Reinforcement Learning: Incorporating Expertise For Faster Learning

CSRL Implementation of CSRL from the AAAI2022 paper: Constraint Sampling Reinforcement Learning: Incorporating Expertise For Faster Learning Python: 3

4 Apr 14, 2022
This is an official implementation for "Swin Transformer: Hierarchical Vision Transformer using Shifted Windows" on Object Detection and Instance Segmentation.

Swin Transformer for Object Detection This repo contains the supported code and configuration files to reproduce object detection results of Swin Tran

Swin Transformer 1.4k Dec 30, 2022
Code of paper "CDFI: Compression-Driven Network Design for Frame Interpolation", CVPR 2021

CDFI (Compression-Driven-Frame-Interpolation) [Paper] (Coming soon...) | [arXiv] Tianyu Ding*, Luming Liang*, Zhihui Zhu, Ilya Zharkov IEEE Conference

Tianyu Ding 95 Dec 04, 2022
Free Book about Deep-Learning approaches for Chess (like AlphaZero, Leela Chess Zero and Stockfish NNUE)

Free Book about Deep-Learning approaches for Chess (like AlphaZero, Leela Chess Zero and Stockfish NNUE)

Dominik Klein 189 Dec 21, 2022
Implementation of TabTransformer, attention network for tabular data, in Pytorch

Tab Transformer Implementation of Tab Transformer, attention network for tabular data, in Pytorch. This simple architecture came within a hair's bread

Phil Wang 420 Jan 05, 2023
PyTorch implementation of Interpretable Explanations of Black Boxes by Meaningful Perturbation

PyTorch implementation of Interpretable Explanations of Black Boxes by Meaningful Perturbation The paper: https://arxiv.org/abs/1704.03296 What makes

Jacob Gildenblat 322 Dec 17, 2022
An example project demonstrating how the Autonomous Learning Library can be used to build new reinforcement learning agents.

About This repository shows how Autonomous Learning Library can be used to build new reinforcement learning agents. In particular, it contains a model

Chris Nota 5 Aug 30, 2022
Evaluating Cross-lingual Sentence Representations

XNLI: The Cross-Lingual NLI Corpus XNLI is an evaluation corpus for language transfer and cross-lingual sentence classification in 15 languages. New:

Meta Research 395 Dec 19, 2022
TumorInsight is a Brain Tumor Detection and Classification model built using RESNET50 architecture.

A Brain Tumor Detection and Classification Model built using RESNET50 architecture. The model is also deployed as a web application using Flask framework.

Pranav Khurana 0 Aug 17, 2021
End-to-End Object Detection with Fully Convolutional Network

This project provides an implementation for "End-to-End Object Detection with Fully Convolutional Network" on PyTorch.

472 Dec 22, 2022
An end-to-end machine learning library to directly optimize AUC loss

LibAUC An end-to-end machine learning library for AUC optimization. Why LibAUC? Deep AUC Maximization (DAM) is a paradigm for learning a deep neural n

Andrew 75 Dec 12, 2022
Panoptic SegFormer: Delving Deeper into Panoptic Segmentation with Transformers

Panoptic SegFormer: Delving Deeper into Panoptic Segmentation with Transformers Results results on COCO val Backbone Method Lr Schd PQ Config Download

155 Dec 20, 2022
Using VapourSynth with super resolution models and speeding them up with TensorRT.

VSGAN-tensorrt-docker Using image super resolution models with vapoursynth and speeding them up with TensorRT. Using NVIDIA/Torch-TensorRT combined wi

111 Jan 05, 2023
Python Actor concurrency library

Thespian Actor Library This library provides the framework of an Actor model for use by applications implementing Actors. Thespian Site with Documenta

Kevin Quick 177 Dec 11, 2022
A Context-aware Visual Attention-based training pipeline for Object Detection from a Webpage screenshot!

CoVA: Context-aware Visual Attention for Webpage Information Extraction Abstract Webpage information extraction (WIE) is an important step to create k

Keval Morabia 41 Jan 01, 2023