BREP : Binary Search in plaintext and gzip files

Last update: Dec 24, 2021

Related tags

Overview

BREP : Binary Search in plaintext and gzip files

Search large files in O(log n) time using binary search.
We support plaintext and Gzipped files.

Benchmark : 8x faster than `grep` on a 2GB dataset !

brep is usually faster than grep for >1GB datasets.

Check tests/benchmark.py to reproduce the results.

grep ^777 test.txt : 1.594 s (15 runs)
brep 777 test.txt : 206.8 ms (15 runs)

Installation

pip install brep or pip install . from this repo

Index your file

In order to conduct binary search, your file needs to be sorted.
We recommend GNU sort, as it's multithreaded and supports large files.
LC_ALL=C sort -u -o output_file input_file

BREP supports compressed file in the GZIP format.
We recommend pigz for quick multicore compression : pigz file

Usage

Provide 1 prefix search term and 1 filepath
brep 77777 test/large.gz

You can also search from our Python class

from brep import Search

for result in Search("77777", "test/large.gz"):
    print(result)

Contribute

PRs are welcome!

Install dev dependencies: pip install -e .[dev]
Test and lint before submitting: pytest && flake8

Todo

Reimplement in Rust
Faster gz size estimation
Search multiple strings at once

dotsend is a web application which helps you to upload your large files and share file via link

0 Dec 3, 2022

A wrapper for DVD file structure and ISO files.

vs-parsedvd DVDs were an error. A wrapper for DVD file structure and ISO files. You can find me in the IEW Discord server

7 Nov 17, 2022

Python Fstab Generator is a small Python script to write and generate /etc/fstab files based on yaml file on Unix-like systems.

PyFstab Generator PyFstab Generator is a small Python script to write and generate /etc/fstab files based on yaml file on Unix-like systems. NOTE : Th

2 Nov 9, 2021

A tool for batch processing large fasta files and accompanying metadata table to upload to repositories via API

Fasta Uploader A tool for batch processing large fasta files and accompanying metadata table to repositories via API The python fasta_uploader.py scri

Centre for Infectious Disease and One Health

1 Dec 9, 2021

Python interface for reading and appending tar files

Python interface for reading and appending tar files, while keeping a fast index for finding and reading files in the archive. This interface has been

1 Nov 12, 2021

A simple file module for creating, editing and saving files.

1 Nov 25, 2021

This program can help you to move and rename many files at once

This program can help you to rename and save many files in a folder in seconds, but don't give the same name to files, it can delete both files.

1 Oct 10, 2022

Fast Python reader and editor for ASAM MDF / MF4 (Measurement Data Format) files

asammdf is a fast parser and editor for ASAM (Association for Standardization of Automation and Measuring Systems) MDF (Measurement Data Format) files

440 Dec 31, 2022

Here is some Python code that allows you to read in SVG files and approximate their paths using a Fourier series.

Here is some Python code that allows you to read in SVG files and approximate their paths using a Fourier series. The Fourier series can be animated and visualized, the function can be output as a two dimensional vector for Desmos and there is a method to output the coefficients as LaTeX code.

12 Jan 1, 2023

BREP : Binary Search in plaintext and gzip files

Related tags

Overview

BREP : Binary Search in plaintext and gzip files

Benchmark : 8x faster than grep on a 2GB dataset !

Installation

Index your file

Usage

Contribute

Todo

You might also like...

dotsend is a web application which helps you to upload your large files and share file via link

A wrapper for DVD file structure and ISO files.

Python Fstab Generator is a small Python script to write and generate /etc/fstab files based on yaml file on Unix-like systems.

A tool for batch processing large fasta files and accompanying metadata table to upload to repositories via API

Python interface for reading and appending tar files

A simple file module for creating, editing and saving files.

This program can help you to move and rename many files at once

Fast Python reader and editor for ASAM MDF / MF4 (Measurement Data Format) files

Here is some Python code that allows you to read in SVG files and approximate their paths using a Fourier series.

Releases(v1.0.1)

v1.0.1(Nov 7, 2021)

Owner

Arnaud de Saint Meloir

A python wrapper for libmagic

Object-oriented file system path manipulation

A bot discord that can create directories, file, rename, move, navigate throw directories etc....

Pti-file-format - Reverse engineering the Polyend Tracker instrument file format

A Python library that provides basic functions to read / write Aseprite format files

Get Your TXT File Length !.

A file utility for accessing both local and remote files through a unified interface.

Measure file similarity in a many-to-many fashion

A simple bulk file renamer, written in python.

A python script generate password files in plain text

BREP : Binary Search in plaintext and gzip files

MetaMove is written in Python3 and aims at easing batch renaming operations based on file meta data.

Search for files under the specified directory. Extract the file name and file path and import them as data.

Convert CSV files into a SQLite database

An easy-to-use library for emulating code in minidump files.

FUSE filesystem Python scripts for Nintendo console files

Python code snippets for extracting PDB codes from .fasta files

A python script to pull the transactions of an Algorand wallet and put them into a CSV file.

Extract longest transcript or longest CDS transcript from GTF annotation file or gencode transcripts fasta file.

CSV To VCF (Multiples en un archivo)

Benchmark : 8x faster than `grep` on a 2GB dataset !