SimpleChinese2 集成了许多基本的中文NLP功能,使基于 Python 的中文文字处理和信息提取变得简单方便。

Overview

SimpleChinese2

SimpleChinese2 集成了许多基本的中文NLP功能,使基于 Python 的中文文字处理和信息提取变得简单方便。

声明

本项目是为方便个人工作所创建的,仅有部分代码原创。包括分词、词云在内的诸多功能来自于其他项目,并非本人所写,如遇问题,请至原项目链接下提问,谢谢!

安装

pip install -U simplechinese==0.2.8

如从 git 上 clone,需要从以下地址下载词向量文件:

https://drive.google.com/file/d/1ltyiTHZk8kIBYQGbZS9GoO_DwDOEWnL9/view?usp=sharing

并拷贝至"./simplechinese/data/"文件夹下

使用方法

import simplechinese as sc

1. 文字预处理

>> print(sc.only_digits(x)) # 仅保留数字 01234 >>> print(sc.only_zh(x)) # 仅保留中文 测试测试测试测试 >>> print(sc.only_en(x)) # 仅保留英文 TestING >>> print(sc.remove_space(x)) # 去除空格 测试测试,TestING;¥%&01234测试测试 >>> print(sc.remove_digits(x)) # 去除数字 测试测试,TestING ;¥%& 测试测试 >>> print(sc.remove_zh(x)) # 去除中文 ,TestING ;¥%& 01234 >>> print(sc.remove_en(x)) # 去除英文 测试测试, ;¥%& 01234测试测试 >>> print(sc.remove_punctuations(x)) # 去除标点符号 测试测试TestING 01234测试测试 >>> print(sc.toLower(x)) # 修改为全小写字母 测试测试,testing ;¥%& 01234测试测试 >>> print(sc.toUpper(x)) # 修改为全大写字母 测试测试,TESTING ;¥%& 01234测试测试 >>> x = "测试,TestING:12345@#【】+=-()。." >>> print(sc.punc_norm(x)) # 将中文标点符号转换成英文标点符号 测试,TestING:12345@#[]+=-().. >>> # y = fillna(df) # 将pandas.DataFrame中的N/A单元格填充为长度为0的str ">
>>> x = "测试测试,TestING    ;¥%& 01234测试测试"

>>> print(sc.only_digits(x))         # 仅保留数字
01234

>>> print(sc.only_zh(x))             # 仅保留中文
测试测试测试测试

>>> print(sc.only_en(x))             # 仅保留英文
TestING

>>> print(sc.remove_space(x))        # 去除空格
测试测试,TestING;¥%&01234测试测试

>>> print(sc.remove_digits(x))       # 去除数字
测试测试,TestING    ;¥%& 测试测试

>>> print(sc.remove_zh(x))           # 去除中文
,TestING    ;¥%& 01234

>>> print(sc.remove_en(x))           # 去除英文
测试测试,    ;¥%& 01234测试测试

>>> print(sc.remove_punctuations(x)) # 去除标点符号
测试测试TestING     01234测试测试

>>> print(sc.toLower(x))             # 修改为全小写字母
测试测试,testing    ;¥%& 01234测试测试

>>> print(sc.toUpper(x))             # 修改为全大写字母
测试测试,TESTING    ;¥%& 01234测试测试

>>> x = "测试,TestING:12345@#【】+=-()。."
>>> print(sc.punc_norm(x))           # 将中文标点符号转换成英文标点符号
测试,TestING:12345@#[]+=-()..

>>> # y = fillna(df) # 将pandas.DataFrame中的N/A单元格填充为长度为0的str

2. 基础NLP信息提取功能

该部分中,分词功能使用 jieba 实现,源码请参考:https://github.com/fxsjy/jieba

同/近义词查找功能复用了 synonyms 中的词向量数据文件,源码请参考:https://github.com/chatopera/Synonyms 但有所改动,改动如下

  1. 由于 pip 上传文件限制,synonyms 需要用户在完成 pip 安装后再下载词向量文件,国内下载需要设置镜像地址或使用特殊手段,有所不便。因此此处将词向量用 float16 表示,并使用 pca 降维至 64 维。总体效果差别不大,如果在意,请直接安装 synonyms 处理同/近义词查找任务。

  2. 原项目通过构建 KDTree 实现快速查找,但比较相似度是使用 cosine similarity,而 KDTree (sklearn) 本身不支持通过 cosine similarity 构建。因此原项目使用欧式距离构建树,导致输出结果有部分顺序混乱。为修复该问题,本项目将词向量归一化后再构建 KDTree,使得向量间的 cosine similarity 与欧式距离(即割线距离)正相关。具体推导可参考下文:https://stackoverflow.com/questions/34144632/using-cosine-distance-with-scikit-learn-kneighborsclassifier

  3. 原项目中未设置缓存上限,本项目中仅保留最近10000次查找记录。

x = "今天是我参加工作的第1天,我花了23.33元买了写零食犒劳一下自己。"
print(sc.extract_nums(x))              # 提取数字信息
[1.0, 23.33]

# mode: 0: No single character words. The words may be overlapped.
#       1: Have single character words. The words may be overlapped.
#       2: No single character words. The words are not overlapped.
#       3: Have single character words. The words are not overlapped.
#       4: Only single characters.
print(sc.extract_words(x, mode=0))      # 分词
['今天', '参加', '工作', '我花', '23.33', '零食', '犒劳', '一下', '自己']

a = "做人真的好难"
b = "做人实在太难了"
print(sc.string_distance(a,b))  # 编辑距离
0.46153846153846156

x = "种族歧视"
print(sc.find_synonyms(x, n=3))  # 同/近义词
[('种族歧视', 1.0), ('种族主义', 0.84619140625), ('歧视', 0.76416015625)]

3. 繁体简体转换

该部分使用 chinese_converter 实现,源码请参考:https://github.com/zachary822/chinese-converter

>> print(sc.to_traditional(x)) # 转换为繁体 烏龜測試123 >>> x = "烏龜測試123" >>> print(sc.to_simplified(x)) # 转换为简体 乌龟测试123 ">
>>> x = "乌龟测试123"
>>> print(sc.to_traditional(x))  # 转换为繁体
烏龜測試123

>>> x = "烏龜測試123"
>>> print(sc.to_simplified(x))   # 转换为简体
乌龟测试123

4. 特征提取和向量化

5. 词云和可视化

TODO:

  1. 句子向量化及句子相似度
  2. 其他特征提取相关工具
Owner
Ming
惊了
Ming
Python functions for summarizing and improving voice dictation input.

Helpmespeak Help me speak uses Python functions for summarizing and improving voice dictation input. Get started with OpenAI gpt-3 OpenAI is a amazing

Margarita Humanitarian Foundation 6 Dec 17, 2022
Unofficial Parallel WaveGAN (+ MelGAN & Multi-band MelGAN & HiFi-GAN & StyleMelGAN) with Pytorch

Parallel WaveGAN implementation with Pytorch This repository provides UNOFFICIAL pytorch implementations of the following models: Parallel WaveGAN Mel

Tomoki Hayashi 1.2k Dec 23, 2022
Simple and efficient RevNet-Library with DeepSpeed support

RevLib Simple and efficient RevNet-Library with DeepSpeed support Features Half the constant memory usage and faster than RevNet libraries Less memory

Lucas Nestler 112 Dec 05, 2022
Python library for interactive topic model visualization. Port of the R LDAvis package.

pyLDAvis Python library for interactive topic model visualization. This is a port of the fabulous R package by Carson Sievert and Kenny Shirley. pyLDA

Ben Mabey 1.7k Dec 20, 2022
Chinese real time voice cloning (VC) and Chinese text to speech (TTS).

Chinese real time voice cloning (VC) and Chinese text to speech (TTS). 好用的中文语音克隆兼中文语音合成系统,包含语音编码器、语音合成器、声码器和可视化模块。

Kuang Dada 6 Nov 08, 2022
An implementation of WaveNet with fast generation

pytorch-wavenet This is an implementation of the WaveNet architecture, as described in the original paper. Features Automatic creation of a dataset (t

Vincent Herrmann 858 Dec 27, 2022
Shared, streaming Python dict

UltraDict Sychronized, streaming Python dictionary that uses shared memory as a backend Warning: This is an early hack. There are only few unit tests

Ronny Rentner 192 Dec 23, 2022
Repository for the paper "Optimal Subarchitecture Extraction for BERT"

Bort Companion code for the paper "Optimal Subarchitecture Extraction for BERT." Bort is an optimal subset of architectural parameters for the BERT ar

Alexa 461 Nov 21, 2022
CoSENT 比Sentence-BERT更有效的句向量方案

CoSENT 比Sentence-BERT更有效的句向量方案

苏剑林(Jianlin Su) 201 Dec 12, 2022
A Chinese to English Neural Model Translation Project

ZH-EN NMT Chinese to English Neural Machine Translation This project is inspired by Stanford's CS224N NMT Project Dataset used in this project: News C

Zhenbang Feng 29 Nov 26, 2022
Chinese version of GPT2 training code, using BERT tokenizer.

GPT2-Chinese Description Chinese version of GPT2 training code, using BERT tokenizer or BPE tokenizer. It is based on the extremely awesome repository

Zeyao Du 5.6k Jan 04, 2023
Extract city and country mentions from Text like GeoText without regex, but FlashText, a Aho-Corasick implementation.

flashgeotext ⚡ 🌍 Extract and count countries and cities (+their synonyms) from text, like GeoText on steroids using FlashText, a Aho-Corasick impleme

Ben 57 Dec 16, 2022
ConvBERT: Improving BERT with Span-based Dynamic Convolution

ConvBERT Introduction In this repo, we introduce a new architecture ConvBERT for pre-training based language model. The code is tested on a V100 GPU.

YITUTech 237 Dec 10, 2022
Dense Passage Retriever - is a set of tools and models for open domain Q&A task.

Dense Passage Retrieval Dense Passage Retrieval (DPR) - is a set of tools and models for state-of-the-art open-domain Q&A research. It is based on the

Meta Research 1.1k Jan 07, 2023
Need: Image Search With Python

Need: Image Search The problem is that a user needs to search for a specific ima

Surya Komandooru 1 Dec 30, 2021
Higher quality textures for the Metal Gear Solid series.

Metal Gear Solid: HD Textures Higher quality textures for the Metal Gear Solid series. The goal is to maximize the quality of assets that the engine w

Samantha 6 Dec 06, 2022
मराठी भाषा वाचविण्याचा एक प्रयास. इंग्रजी ते मराठीचा शब्दकोश. An attempt to preserve the Marathi language. A lightweight and ad free English to Marathi thesaurus.

For English, scroll down मराठी शब्द मराठी भाषा वाचवण्यासाठी मी हा ओपन सोर्स प्रोजेक्ट सुरू केला आहे. माझ्या मते, आपली भाषा हळूहळू आणि कोणाचाही लक्षात

मुक्त स्त्रोत 20 Oct 11, 2022
Mysticbbs-rjam - rJAM splitscreen message reader for MysticBBS A46+

rJAM splitscreen message reader for MysticBBS A46+

Robbert Langezaal 4 Nov 22, 2022
[EMNLP 2021] LM-Critic: Language Models for Unsupervised Grammatical Error Correction

LM-Critic: Language Models for Unsupervised Grammatical Error Correction This repo provides the source code & data of our paper: LM-Critic: Language M

Michihiro Yasunaga 98 Nov 24, 2022
AI-powered literature discovery and review engine for medical/scientific papers

AI-powered literature discovery and review engine for medical/scientific papers paperai is an AI-powered literature discovery and review engine for me

NeuML 819 Dec 30, 2022