A Python client for the Softcite software mention recognizer server

Overview

Softcite software mention recognizer client

Python client for using the Softcite software mention recognition service. It can be applied to

  • individual PDF files

  • recursively to a local directory, processing all the encountered PDF

  • to a collection of documents harvested by biblio-glutton-harvester and article-dataset-builder, with the benefit of re-using the collection manifest for injectng metadata and keeping track of progress. The collection can be stored locally or on a S3 storage.

Requirements

The client has been tested with Python 3.5-3.7.

The client requires a working Softcite software mention recognition service. Service host and port can be changed in the config.json file of the client.

Install

cd software_mention_client/

It is advised to setup first a virtual environment to avoid falling into one of these gloomy python dependency marshlands:

virtualenv --system-site-packages -p python3 env

source env/bin/activate

Install the dependencies, use:

pip3 install -r requirements.txt

Usage and options

usage: software_mention_client.py [-h] [--repo-in REPO_IN] [--file-in FILE_IN]
                                  [--file-out FILE_OUT]
                                  [--data-path DATA_PATH] [--config CONFIG]
                                  [--reprocess] [--reset] [--load]
                                  [--diagnostic] [--scorched-earth]

Softcite software mention recognizer client

optional arguments:
  -h, --help            show this help message and exit
  --repo-in REPO_IN     path to a directory of PDF files to be processed by
                        the Softcite software mention recognizer
  --file-in FILE_IN     a single PDF input file to be processed by the
                        Softcite software mention recognizer
  --file-out FILE_OUT   path to a single output the software mentions in JSON
                        format, extracted from the PDF file-in
  --data-path DATA_PATH
                        path to the resource files created/harvested by
                        biblio-glutton-harvester
  --config CONFIG       path to the config file, default is ./config.json
  --reprocess           reprocessed failed PDF
  --reset               ignore previous processing states and re-init the
                        annotation process from the beginning
  --load                load json files into the MongoDB instance, the --repo-
                        in parameter must indicate the path to the directory
                        of resulting json files to be loaded
  --diagnostic          perform a full count of annotations and diagnostic
                        using MongoDB regarding the harvesting and
                        transformation process
  --scorched-earth      remove a PDF file after its successful processing in
                        order to save storage space, careful with this!

The logs are written by default in a file ./client.log, but the location of the logs can be changed in the configuration file (default ./config.json).

Processing local PDF files

For processing a single file., the resulting json being written as file at the indicated output path:

python3 software_mention_client.py --file-in toto.pdf --file-out toto.json

For processing recursively a directory of PDF files, the results will be:

  • written to a mongodb server and database indicated in the config file

  • and in the directory of PDF files, as json files, together with each processed PDF

python3 software_mention_client.py --repo-in /mnt/data/biblio/pmc_oa_dir/

The default config file is ./config.json, but could also be specified via the parameter --config:

python3 software_mention_client.py --repo-in /mnt/data/biblio/pmc_oa_dir/ --config ./my_config.json

Processing a collection of PDF harvested by biblio-glutton-harvester

biblio-glutton-harvester and article-dataset-builder creates a collection manifest as a LMDB database to keep track of the harvesting of large collection of files. Storage of the resource can be located on a local file system or on a AWS S3 storage. The software-mention client will use the collection manifest to process these harvested documents.

  • locally:

python3 software_mention_client.py --data-path /mnt/data/biblio-glutton-harvester/data/

--data-path indicates the path to the repository of data harvested by biblio-glutton-harvester.

The resulting JSON files will be enriched by the metadata records of the processed PDF and will be stored together with each processed PDF in the data repository.

If the harvested collection is located on a S3 storage, the access information must be indicated in the configuration file of the client config.json. The extracted software mention will be written in a file with extension .software.json, for example:

-rw-rw-r-- 1 lopez lopez 1.1M Aug  8 03:26 0100a44b-6f3f-4cf7-86f9-8ef5e8401567.pdf
-rw-rw-r-- 1 lopez lopez  485 Aug  8 03:41 0100a44b-6f3f-4cf7-86f9-8ef5e8401567.software.json

If a MongoDB server access information is indicated in the configuration file config.json, the extracted information will additionally be written in MongoDB.

License and contact

Distributed under Apache 2.0 license. The dependencies used in the project are either themselves also distributed under Apache 2.0 license or distributed under a compatible license.

Main author and contact: Patrice Lopez ([email protected])

The gPodder podcast client.

___ _ _ ____ __ _| _ \___ __| |__| |___ _ _ |__ / / _` | _/ _ \/ _` / _` / -_) '_| |_ \ \__, |_| \___/\__,_\__,_\___|_| |_

gPodder and related projects 1.1k Jan 04, 2023
Low-level, feature rich and easy to use discord python wrapper

PWRCord Low-level, feature rich and easy to use discord python wrapper Important Note: At this point, this library API is considered unstable and can

MIguel Lopes 1 Dec 26, 2021
Bendford analysis of Ethereum transaction

Bendford analysis of Ethereum transaction The python script script.py extract from already downloaded archive file the ethereum transaction. The value

sleepy ramen 2 Dec 18, 2021
Documentation and Samples for the Official HN API

Hacker News API Overview In partnership with Firebase, we're making the public Hacker News data available in near real time. Firebase enables easy acc

Y Combinator Hacker News 9.6k Jan 03, 2023
Simple Telegram bot to confess to your crush this Valentine's Day

Simple Telegram bot to confess to your crush this Valentine's Day! Steps pip install python-telegram-bot Register a Telegram bot & get the token by fo

3 Mar 18, 2022
Terminal Bot which will Execute your Commands From telegram bot!

Terminal-Bot see this bot alive: https://t.me/HerokuTerminal_Bot With this bot you can execute system commands on your server. how to config? clone or

Moshe 41 Dec 09, 2022
IOGen - An Open source discord token generator

_____ ____ _____ |_ _/ __ \ / ____| | || | | | |

0xVichy#1234 85 Nov 03, 2022
A Telegram Bot which notifies the user when a vaccine is available on CoWin Platform.

Cowin Vaccine Availability Notifier Telegram Bot A bot that notifies the available vaccines at given district in realtime. Introduction • Requirements

Arham Shah 7 Jul 31, 2021
Unfollows Users You're Following

Github-Unfollow-Bot Info It unfollows users you're following, it runs in the background so you can still do what you do without it bothering you. It's

ExT 4 Sep 03, 2022
Request based Python module(s) to help with the Newegg raffle.

Newegg Shuffle Python module(s) to help you with the Newegg raffle How to use $ git clone https://github.com/Matthew17-21/Newegg-Shuffle $ cd Newegg-S

Matthew 45 Dec 01, 2022
An example of using discordpy 2.0.0a to create a bot that supports slash commands

DpySlashBotExample An example of using discordpy 2.0.0a to create a bot that supports slash commands. This is not a fully complete bot, just an exampl

7 Oct 17, 2022
Discord Bot for bugbounty Web

BugbountyBot Discord Bot for Bug Bounty Web The purpose of this bot is to automa

Beek Labs 6 May 03, 2022
𝐀 𝐦𝐨𝐝𝐮𝐥𝐚𝐫 𝐓𝐞𝐥𝐞𝐠𝐫𝐚𝐦 𝐆𝐫𝐨𝐮𝐩 𝐦𝐚𝐧𝐚𝐠𝐞𝐦𝐞𝐧𝐭 𝐛𝐨𝐭 𝐰𝐢𝐭𝐡 𝐮𝐥𝐭𝐢𝐦𝐚𝐭𝐞 𝐟𝐞𝐚𝐭𝐮𝐫𝐞𝐬 !!

𝐇𝐨𝐰 𝐓𝐨 𝐃𝐞𝐩𝐥𝐨𝐲 For easiest way to deploy this Bot click on the below button 𝐌𝐚𝐝𝐞 𝐁𝐲 𝐒𝐮𝐩𝐩𝐨𝐫𝐭 𝐆𝐫𝐨𝐮𝐩 𝐒𝐨𝐮𝐫𝐜𝐞𝐬 𝐆𝐞𝐧𝐞?

Mukesh Solanki 1 Dec 10, 2021
Sukoshi is a proof-of-concept Python implant that leverages the MQTT protocol for C2 and uses AWS IoT Core as infrastructure.

Sukoshi | 少し Overview Sukoshi is a proof-of-concept Python implant that leverages the MQTT protocol for C2 and uses AWS IoT Core as infrastructure. It

Steven Patterson 37 Oct 29, 2022
Video Stream: an Advanced Telegram Bot that's allow you to play Video & Music on Telegram Group Video Chat

Video Stream is an Advanced Telegram Bot that's allow you to play Video & Music on Telegram Group Video Chat 🧪 Get SESSION_NAME from below: Pyrogram

Jonathan 6 Feb 08, 2022
修改自SharpNoPSExec的基于python的横移工具 A Lateral Movement Tool Learned From SharpNoPSExec -- Twitter: @juliourena

PyNoPSExec A Lateral Movement Tool Learned From SharpNoPSExec -- Twitter: @juliourena 根据@juliourena大神的SharpNOPsExec项目改写的横向移动工具 Platform(平台): Windows 1

<a href=[email protected]"> 23 Nov 09, 2022
Telegram Bot to Connect Strangers

Telegram Bot to Connect Strangers How to Run Set your telegram bot token as environment variable TELEGRAM_BOT_TOKEN: export TELEGRAM_BOT_TOKEN=your_t

PyTopia 12 Dec 24, 2022
Open Source Discord bot with many cool features like Weather, Balance, Avatar, User, Server, RP-commands, Gif search, YouTube search, VK post search etc.

Сокобот Дискорд бот с открытым исходным кодом. Содержит в себе экономику, полезные команды (!аватар, !юзер, !сервер и тд.), рп-команды (!обнять, !глад

serverok 2 Jan 16, 2022
Download apps and remove icloud

Download apps and remove icloud

🐍 VerificaC19 SDK implementation for Python

VerificaC19 Python SDK 🐍 VerificaC19 SDK implementation for Python. Requirements Python version = 3.7 Make sure zbar is installed in your system For

Lotrèk 10 Jan 14, 2022