BREP : Binary Search in plaintext and gzip files

Last update: Dec 24, 2021

Related tags

Overview

BREP : Binary Search in plaintext and gzip files

Search large files in O(log n) time using binary search.
We support plaintext and Gzipped files.

Benchmark : 8x faster than `grep` on a 2GB dataset !

brep is usually faster than grep for >1GB datasets.

Check tests/benchmark.py to reproduce the results.

grep ^777 test.txt : 1.594 s (15 runs)
brep 777 test.txt : 206.8 ms (15 runs)

Installation

pip install brep or pip install . from this repo

Index your file

In order to conduct binary search, your file needs to be sorted.
We recommend GNU sort, as it's multithreaded and supports large files.
LC_ALL=C sort -u -o output_file input_file

BREP supports compressed file in the GZIP format.
We recommend pigz for quick multicore compression : pigz file

Usage

Provide 1 prefix search term and 1 filepath
brep 77777 test/large.gz

You can also search from our Python class

from brep import Search

for result in Search("77777", "test/large.gz"):
    print(result)

Contribute

PRs are welcome!

Install dev dependencies: pip install -e .[dev]
Test and lint before submitting: pytest && flake8

Todo

Reimplement in Rust
Faster gz size estimation
Search multiple strings at once

dotsend is a web application which helps you to upload your large files and share file via link

0 Dec 3, 2022

A wrapper for DVD file structure and ISO files.

vs-parsedvd DVDs were an error. A wrapper for DVD file structure and ISO files. You can find me in the IEW Discord server

7 Nov 17, 2022

Python Fstab Generator is a small Python script to write and generate /etc/fstab files based on yaml file on Unix-like systems.

PyFstab Generator PyFstab Generator is a small Python script to write and generate /etc/fstab files based on yaml file on Unix-like systems. NOTE : Th

2 Nov 9, 2021

A tool for batch processing large fasta files and accompanying metadata table to upload to repositories via API

Fasta Uploader A tool for batch processing large fasta files and accompanying metadata table to repositories via API The python fasta_uploader.py scri

Centre for Infectious Disease and One Health

1 Dec 9, 2021

Python interface for reading and appending tar files

Python interface for reading and appending tar files, while keeping a fast index for finding and reading files in the archive. This interface has been

1 Nov 12, 2021

A simple file module for creating, editing and saving files.

1 Nov 25, 2021

This program can help you to move and rename many files at once

This program can help you to rename and save many files in a folder in seconds, but don't give the same name to files, it can delete both files.

1 Oct 10, 2022

Fast Python reader and editor for ASAM MDF / MF4 (Measurement Data Format) files

asammdf is a fast parser and editor for ASAM (Association for Standardization of Automation and Measuring Systems) MDF (Measurement Data Format) files

440 Dec 31, 2022

Here is some Python code that allows you to read in SVG files and approximate their paths using a Fourier series.

Here is some Python code that allows you to read in SVG files and approximate their paths using a Fourier series. The Fourier series can be animated and visualized, the function can be output as a two dimensional vector for Desmos and there is a method to output the coefficients as LaTeX code.

12 Jan 1, 2023

BREP : Binary Search in plaintext and gzip files

Related tags

Overview

BREP : Binary Search in plaintext and gzip files

Benchmark : 8x faster than grep on a 2GB dataset !

Installation

Index your file

Usage

Contribute

Todo

You might also like...

dotsend is a web application which helps you to upload your large files and share file via link

A wrapper for DVD file structure and ISO files.

Python Fstab Generator is a small Python script to write and generate /etc/fstab files based on yaml file on Unix-like systems.

A tool for batch processing large fasta files and accompanying metadata table to upload to repositories via API

Python interface for reading and appending tar files

A simple file module for creating, editing and saving files.

This program can help you to move and rename many files at once

Fast Python reader and editor for ASAM MDF / MF4 (Measurement Data Format) files

Here is some Python code that allows you to read in SVG files and approximate their paths using a Fourier series.

Releases(v1.0.1)

v1.0.1(Nov 7, 2021)

Owner

Arnaud de Saint Meloir

Simple archive format designed for quickly reading some files without extracting the entire archive

Annotate your Python requirements.txt file with summaries of each package.

Provides a convenient way to append numpy arrays to a file.

Nmap XML output to CSV and HTTP/HTTPS URLS.

Two scripts help you to convert csv file to md file by template

QSynthesis is a Python3 API to perform I/O based program synthesis of bitvector expressions.

Remove [x]_ from StudIP zip Archives and archive_filelist.csv completely

CSV-Handler written in Python3

The best way to convert files on your computer, be it .pdf to .png, .pdf to .docx, .png to .ico, or anything you can imagine.

Small-File-Explorer - I coded a small file explorer with several options

Measure file similarity in a many-to-many fashion

Python function to construct a ZIP archive with on the fly - without having to store the entire ZIP in memory or disk

Pure Python tools for reading and writing all TIFF IFDs, sub-IFDs, and tags.

ValveVMF - A python library to parse Valve's VMF files

Import Python modules from any file system path

Convert All TXT Files To One File.

Here is some Python code that allows you to read in SVG files and approximate their paths using a Fourier series.

A small Python module for determining appropriate platform-specific dirs, e.g. a "user data dir".

Various technical documentation, in electronically parseable format

File-manager - A basic file manager, written in Python

Benchmark : 8x faster than `grep` on a 2GB dataset !