Automatically download and crop key information from the arxiv daily paper. (cpu version)

Related tags

DownloaderFocusAX
Overview

FocusAX

按关键词筛选arxiv每日最新paper或从arxiv搜索。

  • 自动下载、获取摘要、自动截取文中表格和图片。

安装必要的环境

  • 安装 paddle
# GPU安装
python3 -m pip install paddlepaddle-gpu==2.1.1 -i https://mirror.baidu.com/pypi/simple

# CPU安装
 python3 -m pip install paddlepaddle==2.1.1 -i https://mirror.baidu.com/pypi/simple
  • 安装 Layout-Parser
=2.2"">
pip3 install -U https://paddleocr.bj.bcebos.com/whl/layoutparser-0.0.0-py3-none-any.whl
pip install "paddleocr>=2.2"
  • 按照其他必要的包
pip3 install -r requirements.txt
  • 下载模型权重
  • PubLayNet 下载解压后放置在paperparse目录下。目录结构如下
FocusAX
    - paperparse
        - ppyolov2_r50vd_dcn_365e_publaynet
            - inference.pdiparams
            - inference.pdiparams.info
            - inference.pdmodel
        - ...
    - downloader
        - ...
    - utils
        - ...
    - configs.py
    - focus_daily.py
    - focus_search.py
    - README.py
    - ...

使用教程

  • configs.py :程序参数配置文件
# =============== 网络代理 ================
# proxy = None # 不使用代理
proxy = {"http": "socks5://127.0.0.1:8080", "https": "socks5://127.0.0.1:8080"}
# =============== 保存文件根目录 ================
root_path = "./arxiv"
# =============== DNN模型推理配置信息 ================
threshold = 0.5
enable_mkldnn = True
enforce_cpu = True
thread_num = 4
  • focus_daily.py :按关键字过滤arxiv daily上的文章(仅当日)
if __name__ == '__main__':
    key_words = ['GAN'] # 要包含的关键词
    subject_words = ['ML', 'CV', 'AI']  # 要包含的类别
    start_parse(key_words, subject_words, needPDF=True, needZip=False)
  • focus_search.py :按关键字在arxiv检索
start_parse('Keyword')
  • root_path 目录中将创建新的文件夹保存结果

效果图

每个文件夹中的abs.md文件保留的是当前pdf的介绍,使用Typora等markdown编辑器打开。 image

image

ps:论文排版不规范会导致截图混乱。

其他

Owner
HeoLis
Interesting in generate methods.
HeoLis
Download and save Bing wallpapers and set as background for GNOME desktop

Save Bing wallpapers and set as background for GNOME desktop This script downloads the Bing wallpaper and sets it in the background of your gnome desk

manikamran 2 Nov 06, 2021
Pypixiv - A fully-typed, asynchronous api wrapper for pixiv

pypixiv this library is a fully-typed, asynchronous api wrapper for pixiv. featu

DeltaLaboratory 2 Nov 16, 2022
Shit-fetch - Shitpost fetcher (downloader)

shit-fetch Download shitpost (random) from https://random-shitpost.com/ Usage ./shitfetch.py --nsfw (true/false) --output ~/Downloads (default : ./)

Pinokaille 1 Jan 02, 2022
GTK4 + Python tutorial with code examples

Taiko's GTK4 Python tutorial Wanna make apps for Linux but not sure how to start with GTK? This guide will hopefully help! The intent is to show you h

190 Jan 08, 2023
Will load an SRC page, logged in with Firefox's cookies imported, and delete all comments from every run

SRCCommentsAutoDeleter Will load an SRC page, logged in with a support browser's cookies, and delete all comments from every run Config is all done in

3 Oct 29, 2021
This Program helps you download songs from the Spotify track's link you give in.

Spotify-Downloader-GUI This Program helps you download songs from the Spotify track's link you give in. It uses yt-dlp to download songs from Youtube.

Harish 12 Jun 14, 2022
Ebook downloader built using python

ebook-downloader Getting Started Open a terminal and run the following commands. git clone github.com/georgemunyoro/ebook-downloader cd ./ebook-downlo

George Munyoro 1 Oct 19, 2021
lo2: Simple youtube-dl web frontend

Simple youtube-dl web frontend

Denis Volk 22 Jun 03, 2022
Audio/Video downloader

youtubeDownloader Audio/Video downloader • The project downloads audio/video/both after link is entered • It also shows total size of the file, time l

Tulsi Thakur 1 Nov 16, 2021
A cli tool to download purchased products from the DLsite.

dlsite-downloader A cli tool to download purchased products from the DLsite. How can I use? This program runs with configurations defined at settings.

AcrylicShrimp 9 Dec 23, 2022
Python module to download all media from a GoFile gallery.

GoFile Downloader Setup First of all, clone this repository : ~$ git clone https://github.com/quatrecentquatre-404/gofile-downloader Second, oh wait..

Quatrecentquatre 61 Jan 01, 2023
Noto fonts go universal! Download Noto fonts combined to suit your region (South Asia, SE Asia, Africa-MiddleEast, Europe-Americas).

Go Noto Universal Noto fonts go universal! Download Noto fonts combined to suit your region (South Asia, SE Asia, East Asia, Africa-MiddleEast, Europe

Satish B 67 Jan 06, 2023
Download your bandcamp collection using this python script.

bandcamp-downloader Download your Bandcamp collection using this python script. It requires you to have a browser with a logged in session of bandcamp

72 Dec 20, 2022
A Python package for downloading / archiving all available episodes from a podcast RSS feed.

allcasts 📻 🗃 A Python package for downloading all available episodes from a podcast RSS feed. Useful for making private archives of your favourite p

Lewis Gentle 5 Nov 20, 2022
A tool to download program information from Bugcrowd, for use by researchers to compare programs they are eligible to participate in

Description bcstats is a tool which allows Bugcrowd researchers to download information about all accessible programs (public and private) into a sing

19 Oct 13, 2022
Userscript qutebrowser for downloading audio / video from youtube using aria2

Yt-Downloader Userscript qutebrowser for downloading video / audio from youtube using aria2 by hint links. Requirements Rofi youtube-dl aria2 dunst In

Ara 0 Dec 11, 2021
Tool To download Amazon 4k SDR HDR 1080, CDM IS Not Included

WV-AMZN-4K-RIPPER Tool To download Amazon 4k SDR HDR 1080, CDM IS Not Included For CDM You can Mail :- 11 Dec 23, 2021

Simple avogadr.io batch downloader python script

Simple avogadr.io batch downloader python script

2 Jan 19, 2022
Python script to automate youtube-dl downloads

Automated Download Tool !! Project status I am writing a new version of this program, which will solve several errors. The new version only supports G

Devil64-Dev 21 Sep 22, 2022
𝐴 𝑡𝑒𝑙𝑒𝑔𝑟𝑎𝑚 𝑏𝑜𝑡 𝑡ℎ𝑎𝑡 𝑐𝑎𝑛 𝑑𝑜𝑤𝑛𝑙𝑜𝑎𝑑 𝑣𝑖𝑑𝑒𝑜 𝑎𝑛𝑑 𝑎𝑢𝑑𝑖𝑜 𝑓𝑟𝑜𝑚 𝑦𝑜𝑢𝑡𝑢𝑏𝑒 𝑎𝑛𝑑 𝑣𝑖𝑑𝑒𝑜 𝑤𝑒𝑏𝑠𝑖𝑡𝑒𝑠 𝑞𝑢𝑖𝑐𝑘𝑙𝑦

𝐴 𝑡𝑒𝑙𝑒𝑔𝑟𝑎𝑚 𝑏𝑜𝑡 𝑡ℎ𝑎𝑡 𝑐𝑎𝑛 𝑑𝑜𝑤𝑛𝑙𝑜𝑎𝑑 𝑣𝑖𝑑𝑒𝑜 𝑎𝑛𝑑 𝑎𝑢𝑑𝑖𝑜 𝑓𝑟𝑜𝑚 𝑦𝑜𝑢𝑡𝑢𝑏𝑒 𝑎𝑛𝑑 𝑣𝑖𝑑𝑒𝑜 𝑤𝑒𝑏𝑠𝑖𝑡𝑒𝑠 𝑞𝑢𝑖𝑐𝑘𝑙𝑦

SOCIAL MECHANIC 2 Aug 04, 2022