Latent Semantic Analysis
Pipeline for training LSA models using Scikit-Learn.
Usage
Instead of writing custom code for latent semantic analysis, you just need:
- install pipeline:
pip install latent-semantic-analysis
- run pipeline:
- either in terminal:
lsa-train --path_to_config config.yaml
- or in python:
import latent_semantic_analysis
latent_semantic_analysis.train(path_to_config="config.yaml")
NOTE: more about config file here.
No data preparation is needed, only a csv file with raw text column (with arbitrary name).
Config
The user interface consists of only one files:
- config.yaml - general configuration with sklearn TF-IDF and SVD parameters
Change config.yaml to create the desired configuration and train LSA model with the following command:
- terminal:
lsa-train --path_to_config config.yaml
- python:
import latent_semantic_analysis
latent_semantic_analysis.train(path_to_config="config.yaml")
Default config.yaml:
seed: 42
path_to_save_folder: models
# data
data:
data_path: data/data.csv
sep: ','
text_column: text
# tf-idf
tf-idf:
lowercase: true
ngram_range: (1, 1)
max_df: 1.0
min_df: 1
# svd
svd:
n_components: 10
algorithm: arpack
NOTE: tf-idf and svd are sklearn TfidfVectorizer and TruncatedSVD parameters correspondingly, so you can parameterize instances of these classes however you want.
Output
After training the model, the pipeline will return the following files:
model.joblib- sklearn pipeline with LSA (TF-IDF and SVD steps)config.yaml- config that was used to train the modellogging.txt- logging filedoc2topic.json- document embeddingsterm2topic.json- term embeddings
Requirements
Python >= 3.6
Citation
If you use latent-semantic-analysis in a scientific publication, we would appreciate references to the following BibTex entry:
@misc{dayyass2021lsa,
author = {El-Ayyass, Dani},
title = {Pipeline for training LSA models},
howpublished = {\url{https://github.com/dayyass/latent-semantic-analysis}},
year = {2021}
}
9 Jul 17, 2021
42 Dec 20, 2022
190 Dec 21, 2022
7 Sep 20, 2022
57 Dec 7, 2022
74 Oct 7, 2022
6 Jun 27, 2022
654 Jan 2, 2023
14 Dec 9, 2022
3 Jan 17, 2022
13 Sep 20, 2022
1 Mar 14, 2022
61 Oct 26, 2022
31 Nov 30, 2022
12 Sep 28, 2022
2 Mar 17, 2022
401 Dec 11, 2022
6 Apr 29, 2022
11 Nov 11, 2022
2 Jun 10, 2022
128 Dec 29, 2022
133 Sep 20, 2022
171 Nov 21, 2022
2 Dec 27, 2022
1 Dec 31, 2021
7 Oct 30, 2022
5 Oct 29, 2022
3 May 29, 2022