metamaker
Simple command line tool to train and deploy your machine learning models with AWS SageMaker
Features
metamaker enables you to:
- Build a docker image for training and inference with poetry and FastAPI
- Train your own machine learning model with SageMaker
- Deploy inference endpoint with SageMaker
Usage
- Create poetry project and install metamaker
❯ poetry new your_module
❯ cd your_module
❯ poetry add metamaker
- Define scripts for traning and inference in
main.py
Output: # ... from your_module import Model, Input, Output app = MetaMaker[Model, Input, Output]() @app.trainer def train( dataset_path: Path, artifact_path: Path, hyperparameters: Dict[str, Any], ) -> None: model = Model(**hyperparameters) model.train(dataset_path / "train.csv") model.save(artifact_path / "model.tar.gz") @app.loader def load(artifact_path: Path) -> Model: return Model.load(artifact_path / "model.tar.gz") @app.predictor def predict(model: Model, data: Input) -> Output: return model.predict(data) ">
from pathlib import Path from typing import Any, Dict from metamaker import MetaMaker # Import your model, and input/output data classs: # # Model ... machine learning model class you want to use # Input ... input data class for inference # Output ... ouput data class for inference # # Note that the Input and Output are used as type hints to # create API endpoint with FastAPI like below: # # @fastapi_app.post("/invocations") # def predict(data: Input) -> Output: # ... from your_module import Model, Input, Output app = MetaMaker[Model, Input, Output]() @app.trainer def train( dataset_path: Path, artifact_path: Path, hyperparameters: Dict[str, Any], ) -> None: model = Model(**hyperparameters) model.train(dataset_path / "train.csv") model.save(artifact_path / "model.tar.gz") @app.loader def load(artifact_path: Path) -> Model: return Model.load(artifact_path / "model.tar.gz") @app.predictor def predict(model: Model, data: Input) -> Output: return model.predict(data)
- Write metamaker configs in
metamaker.yaml
# Specify metamaker handler like: `path.to.module:app_name`
handler: main:app
# dataset_path and artifact_path should be directories and end with '/'
dataset_path: s3://your-bucket/path/to/dataset/
artifact_path: s3://your-bucket/path/to/artifacts/
hyperparameter_path: ./hparams.yaml
image:
name: metamaker
includes:
- your_module/
- main.py
excludes:
- __pycache__/
- '*.py[cod]'
training:
execution_role: arn:aws:iam::xxxxxxxxxxxx:role/SageMakerExecutionRole
instance:
type: ml.m5.large
count: 1
inference:
endpoint_name: your_endpoint
instance:
type: ml.t2.meduim
count: 1
- Build docker image and push to ECR
metamaker build --deploy .
- Train your model with SageMaker and deploy endpoint
metamker sagemaker train --deploy
2 Feb 13, 2022
5 Dec 15, 2021
1.5k Jan 05, 2023
2 Dec 30, 2021
6 Jun 26, 2022
13 Sep 23, 2021
476 Jan 07, 2023
3 Jan 19, 2022
7 Mar 30, 2022
1.9k Jan 09, 2023
222 Jan 03, 2023
1 Oct 26, 2022
528 Jan 02, 2023
3 Mar 14, 2022
22 Nov 10, 2022
13 Nov 28, 2022
808 Jan 04, 2023
3 Nov 28, 2021
340 Dec 18, 2022
1 Oct 20, 2021