BREP : Binary Search in plaintext and gzip files

Last update: Dec 24, 2021

Related tags

Overview

BREP : Binary Search in plaintext and gzip files

Search large files in O(log n) time using binary search.
We support plaintext and Gzipped files.

Benchmark : 8x faster than `grep` on a 2GB dataset !

brep is usually faster than grep for >1GB datasets.

Check tests/benchmark.py to reproduce the results.

grep ^777 test.txt : 1.594 s (15 runs)
brep 777 test.txt : 206.8 ms (15 runs)

Installation

pip install brep or pip install . from this repo

Index your file

In order to conduct binary search, your file needs to be sorted.
We recommend GNU sort, as it's multithreaded and supports large files.
LC_ALL=C sort -u -o output_file input_file

BREP supports compressed file in the GZIP format.
We recommend pigz for quick multicore compression : pigz file

Usage

Provide 1 prefix search term and 1 filepath
brep 77777 test/large.gz

You can also search from our Python class

from brep import Search

for result in Search("77777", "test/large.gz"):
    print(result)

Contribute

PRs are welcome!

Install dev dependencies: pip install -e .[dev]
Test and lint before submitting: pytest && flake8

Todo

Reimplement in Rust
Faster gz size estimation
Search multiple strings at once

dotsend is a web application which helps you to upload your large files and share file via link

0 Dec 3, 2022

A wrapper for DVD file structure and ISO files.

vs-parsedvd DVDs were an error. A wrapper for DVD file structure and ISO files. You can find me in the IEW Discord server

7 Nov 17, 2022

Python Fstab Generator is a small Python script to write and generate /etc/fstab files based on yaml file on Unix-like systems.

PyFstab Generator PyFstab Generator is a small Python script to write and generate /etc/fstab files based on yaml file on Unix-like systems. NOTE : Th

2 Nov 9, 2021

A tool for batch processing large fasta files and accompanying metadata table to upload to repositories via API

Fasta Uploader A tool for batch processing large fasta files and accompanying metadata table to repositories via API The python fasta_uploader.py scri

Centre for Infectious Disease and One Health

1 Dec 9, 2021

Python interface for reading and appending tar files

Python interface for reading and appending tar files, while keeping a fast index for finding and reading files in the archive. This interface has been

1 Nov 12, 2021

A simple file module for creating, editing and saving files.

1 Nov 25, 2021

This program can help you to move and rename many files at once

This program can help you to rename and save many files in a folder in seconds, but don't give the same name to files, it can delete both files.

1 Oct 10, 2022

Fast Python reader and editor for ASAM MDF / MF4 (Measurement Data Format) files

asammdf is a fast parser and editor for ASAM (Association for Standardization of Automation and Measuring Systems) MDF (Measurement Data Format) files

440 Dec 31, 2022

Here is some Python code that allows you to read in SVG files and approximate their paths using a Fourier series.

Here is some Python code that allows you to read in SVG files and approximate their paths using a Fourier series. The Fourier series can be animated and visualized, the function can be output as a two dimensional vector for Desmos and there is a method to output the coefficients as LaTeX code.

12 Jan 1, 2023

BREP : Binary Search in plaintext and gzip files

Related tags

Overview

BREP : Binary Search in plaintext and gzip files

Benchmark : 8x faster than grep on a 2GB dataset !

Installation

Index your file

Usage

Contribute

Todo

You might also like...

dotsend is a web application which helps you to upload your large files and share file via link

A wrapper for DVD file structure and ISO files.

Python Fstab Generator is a small Python script to write and generate /etc/fstab files based on yaml file on Unix-like systems.

A tool for batch processing large fasta files and accompanying metadata table to upload to repositories via API

Python interface for reading and appending tar files

A simple file module for creating, editing and saving files.

This program can help you to move and rename many files at once

Fast Python reader and editor for ASAM MDF / MF4 (Measurement Data Format) files

Here is some Python code that allows you to read in SVG files and approximate their paths using a Fourier series.

Releases(v1.0.1)

v1.0.1(Nov 7, 2021)

Owner

Arnaud de Saint Meloir

Here is some Python code that allows you to read in SVG files and approximate their paths using a Fourier series.

Python codes for the server and client end that facilitates file transfers. (Using AWS EC2 instance as the server)

Test app for importing contact information in CSV files.

Organize the files into the relevant sub-folders

Python interface for reading and appending tar files

A bot discord that can create directories, file, rename, move, navigate throw directories etc....

LightCSV - This CSV reader is implemented in just pure Python.

A Python script to backup your favorite Discord gifs

Python virtual filesystem for SQLite to read from and write to S3

Python's Filesystem abstraction layer

Pti-file-format - Reverse engineering the Polyend Tracker instrument file format

Two scripts help you to convert csv file to md file by template

shred - A cross-platform library for securely deleting files beyond recovery.

Uncompress DEFLATE streams in pure Python

Simple Python File Manager

Pure Python tools for reading and writing all TIFF IFDs, sub-IFDs, and tags.

Small Python script to generate a calendar (.ics) file from SIMASTER courses schedule.

MetaMove is written in Python3 and aims at easing batch renaming operations based on file meta data.

File-manager - A basic file manager, written in Python

Quick and dirty FAT12 filesystem to ZIP file converter

Benchmark : 8x faster than `grep` on a 2GB dataset !