eXPeditious Data Transfer

Last update: Jan 06, 2022

Overview

xpdt: eXPeditious Data Transfer

About

xpdt is (yet another) language for defining data-types and generating code for serializing and deserializing them. It aims to produce code with little or no overhead and is based on fixed-length representations which allows for zero-copy deserialization and (at-most-)one-copy writes (source to buffer).

The generated C code, in particular, is highly optimized and often permits the elimination of data-copying for writes and enables optimizations such as loop-unrolling for fixed-length objects. This can lead to read speeds in excess of 500 million objects per second (~1.8 nsec per object).

Examples

The xpdt source language looks similar to C struct definitions:

struct timestamp {
	u32	tv_sec;
	u32	tv_nsec;
};

struct point {
	i32	x;
	i32	y;
	i32	z;
};

struct line {
	timestamp	time;
	point		line_start;
	point		line_end;
	bytes		comment;
};

Fixed width integer types from 8 to 128 bit are supported, along with the bytes type, which is a variable-length sequence of bytes.

Target Languages

The following target languages are currently supported:

C
Python

The C code is very highly optimized.

The Python code is about as well optimized for CPython as I can make it. It uses typed NamedTuple for objects, which has some small overhead over regular tuples, and it uses struct.Struct to do the packing/unpacking. I have also code-golfed the generated bytecodes down to what I think is minimal given the design constraints. As a result, performance of the pure Python code is comparable to a JSON library implemented in C or Rust.

For better performance in Python, it may be desirable to develop a Cython target. In some instances CFFI structs may be more performant since they can avoid the creation/destruction of an object for each record.

Target languages are implemented purely as jinja2 templates.

Serialization format

The serialization format for fixed-length objects is simply a packed C struct.

For any object which contains bytes type fields:

a 32bit unsigned record length is prepended to the struct
all bytes type fields are converted to u32 and contain the length of the bytes
all bytes contents are appended after the struct in the order in which they appear

For example, following the example above, the serialization would be:

u32 tot_len # = 41
u32 time.tv_sec
u32 time.tv_usec
i32 line_start.x
i32 line_start.y
i32 line_start.z
i32 line_end.x
i32 line_end.y
i32 line_end.z
u32 comment # = 5
u8 'H'
u8 'e'
u8 'l'
u8 'l'
u8 'o'

Features

The feature-set is, as of now, pretty slim.

There are no array / sequence / map types, and no keyed unions.

Support for such things may be added in future provided that suitable implementations exist. An implementation is suitable if:

It admits a zero (or close to zero) overhead implementation
it causes no overhead when the feature isn't being used

License

The compiler is released under the GPLv3.

The C support code/headers are released under the MIT license.

The generated code is yours.

Official code for the CVPR 2021 paper "How Well Do Self-Supervised Models Transfer?"

How Well Do Self-Supervised Models Transfer? This repository hosts the code for the experiments in the CVPR 2021 paper How Well Do Self-Supervised Mod

157 Dec 16, 2022

Implementation of Cross Transformer for spatially-aware few-shot transfer, in Pytorch

Cross Transformers - Pytorch (wip) Implementation of Cross Transformer for spatially-aware few-shot transfer, in Pytorch Install $ pip install cross-t

40 Dec 22, 2022

Neural style transfer as a class in PyTorch

pt-styletransfer Neural style transfer as a class in PyTorch Based on: https://github.com/alexis-jacq/Pytorch-Tutorials Adds: StyleTransferNet as a cl

31 Jun 27, 2022

Offcial repository for the IEEE ICRA 2021 paper Auto-Tuned Sim-to-Real Transfer.

47 Jun 30, 2022

transfer attack; adversarial examples; black-box attack; unrestricted Adversarial Attacks on ImageNet; CVPR2021 天池黑盒竞赛

transfer_adv CVPR-2021 AIC-VI: unrestricted Adversarial Attacks on ImageNet CVPR2021 安全AI挑战者计划第六期赛道2：ImageNet无限制对抗攻击介绍：深度神经网络已经在各种视觉识别问题上取得了最先进的性能。

25 Dec 8, 2022

PyKale is a PyTorch library for multimodal learning and transfer learning as well as deep learning and dimensionality reduction on graphs, images, texts, and videos

PyKale is a PyTorch library for multimodal learning and transfer learning as well as deep learning and dimensionality reduction on graphs, images, texts, and videos. By adopting a unified pipeline-based API design, PyKale enforces standardization and minimalism, via reusing existing resources, reducing repetitions and redundancy, and recycling learning models across areas.

370 Dec 27, 2022

Two-Stage Peer-Regularized Feature Recombination for Arbitrary Image Style Transfer

Releases(v0.0.5)

v0.0.5(Jan 6, 2022)

Woops. Fixed mypy error in previous release.
Source code(tar.gz)
Source code(zip)
v0.0.4(Jan 6, 2022)

Recover from malformed inputs in enum stream by yielding an exception.
Source code(tar.gz)
Source code(zip)
v0.0.3(Dec 21, 2021)

First cut of multiplexed files support, where you can read/write structs of different types to and from the same file. A discriminator field and record length is prepended to each record.

Fields whose names begin with underscore are now considered hidden/reserved fields. They can be use to add padding and force specific alignments.

Improve the error messages in the tokenization stage.

Numerous improvements to the C and python code. Added support for new types: bytearray, stringlist, intstack.
Source code(tar.gz)
Source code(zip)
v0.0.2(Jun 27, 2021)

A new string type was added, as well as the ability to add reserved/padding fields which are set to all zeroes.

Some language-breaking changes were made: the "type" keyword changed to "struct" and the signed integer types were renamed to the more conventional "i8" ... "i64".
Source code(tar.gz)
Source code(zip)
v0.0.1(May 23, 2021)

Source code(tar.gz)
Source code(zip)

eXPeditious Data Transfer

Related tags

Overview

xpdt: eXPeditious Data Transfer

About

Examples

Target Languages

Serialization format

Features

License

You might also like...

Official code for the CVPR 2021 paper "How Well Do Self-Supervised Models Transfer?"

Implementation of Cross Transformer for spatially-aware few-shot transfer, in Pytorch

Neural style transfer as a class in PyTorch

Offcial repository for the IEEE ICRA 2021 paper Auto-Tuned Sim-to-Real Transfer.

transfer attack; adversarial examples; black-box attack; unrestricted Adversarial Attacks on ImageNet; CVPR2021 天池黑盒竞赛

PyKale is a PyTorch library for multimodal learning and transfer learning as well as deep learning and dimensionality reduction on graphs, images, texts, and videos

Two-Stage Peer-Regularized Feature Recombination for Arbitrary Image Style Transfer

Instant Real-Time Example-Based Style Transfer to Facial Videos

An implementation of "Optimal Textures: Fast and Robust Texture Synthesis and Style Transfer through Optimal Transport"

Releases(v0.0.5)

v0.0.5(Jan 6, 2022)

v0.0.4(Jan 6, 2022)

v0.0.3(Dec 21, 2021)

v0.0.2(Jun 27, 2021)

v0.0.1(May 23, 2021)

Owner

Gianni Tedesco

Rethinking the Importance of Implementation Tricks in Multi-Agent Reinforcement Learning

Ultra-lightweight human body posture key point CNN model. ModelSize:2.3MB HUAWEI P40 NCNN benchmark: 6ms/img,

Official pytorch implementation of "Scaling-up Disentanglement for Image Translation", ICCV 2021.

Implementation of "Efficient Regional Memory Network for Video Object Segmentation" (Xie et al., CVPR 2021).

Pytorch implementation of Value Iteration Networks (NIPS 2016 best paper)

Detecting Blurred Ground-based Sky/Cloud Images

Car Parking Tracker Using OpenCv

GoodNews Everyone! Context driven entity aware captioning for news images

PyTorch implementation DRO: Deep Recurrent Optimizer for Structure-from-Motion

MinkLoc3D-SI: 3D LiDAR place recognition with sparse convolutions,spherical coordinates, and intensity

Degree-Quant: Quantization-Aware Training for Graph Neural Networks.

Code for Paper Predicting Osteoarthritis Progression via Unsupervised Adversarial Representation Learning

A python toolbox for predictive uncertainty quantification, calibration, metrics, and visualization

Official implementation of "Dynamic Anchor Learning for Arbitrary-Oriented Object Detection" (AAAI2021).

MixText: Linguistically-Informed Interpolation of Hidden Space for Semi-Supervised Text Classification

Local Multi-Head Channel Self-Attention for FER2013

Deep Anomaly Detection with Outlier Exposure (ICLR 2019)

Escaping the Gradient Vanishing: Periodic Alternatives of Softmax in Attention Mechanism

A real-time speech emotion recognition application using Scikit-learn and gradio

BlockUnexpectedPackets - Preventing BungeeCord CPU overload due to Layer 7 DDoS attacks by scanning BungeeCord's logs