API Reference

​

MyxBoard

The MyxBoard is a tool designed to enhance reproducibility, tracking, and comparison of foundation models by managing evaluations and comparisons.

​

Constructor

MyxBoard(model_repo_ids: Optional[List[str]] = None, name: Optional[str] = None, hf_collection_name: Optional[str] = None)

• model_repo_ids: Optional list of model repository IDs to evaluate.
• name: Optional name for the MyxBoard instance.
• hf_collection_name: Optional Hugging Face collection name to initialize models from.

​

Class Attributes

• MYXMATCH_SUPPORTED_MODELS: List of supported model names for evaluation.

​

Methods

​

poll_and_store_results() -> bool

Polls for job completion and stores results. Returns True if all jobs are completed, otherwise False.

​

fetch_results() -> Dict[str, Union[str, dict]]

Fetches and updates results for completed jobs. Returns updated results or all results if no updates.

​

view_results() -> dict

Returns a simplified version of the evaluation results without polling the server.

​

get_results(verbose: bool = False) -> dict

Returns evaluation results. If verbose is True, returns detailed results; otherwise, returns simplified results.

​

push_to_hf() -> None

Pushes evaluation results to Hugging Face by creating a dataset, tagging it, and adding it to the original collection.

​

delete() -> None

Deletes the MyxBoard instance from the server.

​

Usage Example

# Initialize MyxBoard
from remyxai.client.myxboard import MyxBoard
from remyxai.client.remyx_client import RemyxAPI
from remyxai.api.evaluations import EvaluationTask

myxboard = MyxBoard(name="My Evaluation", hf_collection_name="my-model-collection")

remyx_api = RemyxAPI()
tasks = [EvaluationTask.MYXMATCH]
prompt = "You are a media analyst. Objective: Analyze the media coverage. Phase 1: Begin analysis."

# Run the evaluation
remyx_api.evaluate(myx_board, tasks, prompt=prompt)
# Once the evaluation is complete, fetch result
results = myx_board.get_results()

​

Evaluation

These functions provide additional capabilities for managing evaluations in the RemyxAI platform.

​

EvaluationTask

Possible evaluation tasks:

• MYXMATCH: Represents the “myxmatch” evaluation task.
• BENCHMARK: Represents the “benchmark” task powered by lighteval

​

Functions

​

list_evaluations() -> list

Lists all evaluations from the server.

Returns:

• A list of evaluations if successful.
• A dictionary with an "error" key if the request fails.

Example:

evaluations = list_evaluations()
print(evaluations)

​

download_evaluation(task_name: str, eval_name: str) -> dict

Downloads evaluation results for a specific task and evaluation name.

Parameters:

• task_name: The name of the task.
• eval_name: The name of the evaluation.

Returns:

• A dictionary containing the evaluation results if successful.
• A dictionary with an "error" key if the request fails.

Example:

results = download_evaluation("my_task", "my_evaluation")
print(results)

​

delete_evaluation(eval_type: str, eval_name: str) -> dict

Deletes an evaluation from the server.

Parameters:

• eval_type: The type of the evaluation.
• eval_name: The name of the evaluation.

Returns:

• A dictionary containing the response from the server if successful.
• A dictionary with an "error" key if the request fails.

Example:

response = delete_evaluation("myxmatch", "my_evaluation")
print(response)

​

Model Deployment

The Model Deployment API provides functions for downloading and deploying machine learning models using Docker and Triton Server.

​

Functions

​

download_deployment_package(model_name: str, output_path: str) -> Optional[requests.Response]

Downloads a deployment package for the specified model and saves it to the provided output path.

Parameters:

• model_name: The name of the model to download.
• output_path: The file path where the deployment package should be saved.

Returns:

• A requests.Response object if the download is successful.
• None if the download fails.

Example:

response = download_deployment_package("my_model", "/path/to/output.zip")
if response:
    print("Download successful!")
else:
    print("Download failed.")

​

deploy_model(model_name: str, action: str = "up") -> None

Deploys or stops a model container using Docker Compose. The deployment package is downloaded and extracted, and a Triton server is started or stopped.

Parameters:

• model_name: The name of the model to deploy.
• action: The action to perform, either "up" to start the container or "down" to stop it. Default is "up".

Behavior:

• For "up" action:
  • Downloads the deployment package.
  • Creates necessary directories and Docker Compose files.
  • Builds and runs the Docker container for the model using Triton Server.
• For "down" action:
  • Stops the Docker container using the docker compose down command.

Example:

# Deploy the model
deploy_model("my_model", action="up")

# Stop the model container
deploy_model("my_model", action="down")

​

Usage Example

​

Download and Deploy a Model

# Download the deployment package for a model
download_deployment_package("my_model", "/path/to/output.zip")

# Deploy the model using Docker Compose
deploy_model("my_model", action="up")

# To stop the model container
deploy_model("my_model", action="down")

​

Inference

This API enables running inference on a model deployed using Triton Inference Server. It sends input data to the server, waits for a response, and returns the results along with the time taken to execute the inference.

​

Functions

​

run_inference(model_name: str, prompt: str, server_url: str = "localhost:8000", model_version: str = "1") -> Tuple[str, float]

Runs inference on a model deployed on Triton Inference Server and returns the results along with the time taken for inference.

Parameters:

• model_name: The name of the model to run inference on.
• prompt: The input text or prompt to send to the model.
• server_url: The URL of the Triton server. Default is "localhost:8000".
• model_version: The version of the model to use for inference. Default is "1".

Returns:

• A tuple containing:
  • results: The inference output as a string.
  • elapsed_time: The time taken to run the inference, in seconds.

Example:

results, elapsed_time = run_inference("my_model", "Generate some text")
print("Results:", results)
print("Time taken:", elapsed_time, "seconds")

​

Usage Example

# Run inference on the model with a prompt
results, elapsed_time = run_inference("text_generation_model", "Once upon a time")

# Print the results and time taken
print("Inference Results:", results)
print("Inference Time:", elapsed_time, "seconds")

​

Model Management

This API provides functions to interact with a model repository, allowing you to list, get summaries, delete, and download models.

​

Functions

​

list_models() -> dict

Retrieves a list of available models from the server.

Returns:

• A dictionary containing the list of models.

Example:

models = list_models()
print("Available Models:", models)

​

get_model_summary(model_name: str) -> dict

Fetches a detailed summary of the specified model.

Parameters:

• model_name: The name of the model for which to retrieve the summary.

Returns:

• A dictionary containing the summary information of the model.

Example:

summary = get_model_summary("my_model")
print("Model Summary:", summary)

​

delete_model(model_name: str) -> dict

Deletes the specified model from the server.

Parameters:

• model_name: The name of the model to delete.

Returns:

• A dictionary containing the response from the server after the deletion request.

Example:

response = delete_model("my_model")
print("Delete Response:", response)

​

download_model(model_name: str, model_format: str) -> requests.Response

Downloads the specified model in the desired format from the server.

Parameters:

• model_name: The name of the model to download.
• model_format: The format in which the model should be downloaded (e.g., "onnx", "torchscript").

Returns:

• A requests.Response object containing the server’s response.
• If the request is successful, the model will be saved as a .zip file with the format {model_name}.zip.

Example:

response = download_model("my_model", "onnx")
if response.status_code == 200:
    print(f"{model_name}.zip downloaded successfully!")
else:
    print("Error downloading the model:", response.content)

​

Usage Example

​

Listing and Managing Models

from remyxai.api.models import list_models, get_model_summary, delete_model, download_model

# List all models
models = list_models()
print("Available Models:", models)

# Get summary of a model

summary = get_model_summary("text_generation_model")
print("Model Summary:", summary)

# Delete a model
response = delete_model("old_model")
print("Delete Response:", response)

# Download a model
response = download_model("text_generation_model", "onnx")
if response.status_code == 200:
    print("Model downloaded successfully!")
else:
    print("Failed to download model:", response.content)

​

Dataset Management

This API provides functions to list, download, and delete datasets from the server.

​

list_datasets() -> list

Lists all datasets available from the server.

Returns:

• A list of datasets if the request is successful, otherwise an error message.

Example:

datasets = list_datasets()
print("Available Datasets:", datasets)

​

download_dataset(dataset_type: str, dataset_name: str) -> dict

Downloads a dataset by generating a presigned URL.

Parameters:

• dataset_type: The type of the dataset to download.
• dataset_name: The name of the dataset to download.

Returns:

• A dictionary containing a success message if the dataset is downloaded successfully, otherwise an error message.

Example:

response = download_dataset("public", "example_dataset")
print("Download Response:", response)

​

delete_dataset(dataset_type: str, dataset_name: str) -> str

Deletes a dataset from the server.

Parameters:

• dataset_type: The type of the dataset to delete.
• dataset_name: The name of the dataset to delete.

Returns:

• A success message if the dataset is deleted successfully, otherwise an error message.

Example:

response = delete_dataset("public", "example_dataset")
print("Delete Response:", response)

​

Usage Example

from remyxai.api.datasets import list_datasets, download_dataset, delete_dataset

# List all datasets
datasets = list_datasets()
print("Available Datasets:", datasets)

# Download a dataset
response = download_dataset("public", "example_dataset")
print("Download Response:", response)

# Delete a dataset
response = delete_dataset("public", "example_dataset")
print("Delete Response:", response)

​

Run Tasks

Train models and Curate datasets using tasks APIs

​

MyxMatch

Submit a MyxMatch task to the server.

• Endpoint: POST /task/myxmatch/{name}/{prompt}/{models}
• Parameters:
  • name: The name of the task.
  • prompt: The prompt for the task.
  • models: A list of model names.
• Returns: A dictionary containing the task status.

Example:

from remyxai.api.tasks import run_myxmatch

response = run_myxmatch("my_task", "my_prompt", ["model1", "model2"])
print("Task Status:", response)

​

Benchmark

Submit a benchmark task to the server.

• Endpoint: POST /task/benchmark
• Parameters:
  • name: The name of the task.
  • models: A list of model names.
  • evals: A list of evaluation names.
• Returns: A dictionary containing the task status.

Example:

from remyxai.api.tasks import run_benchmark

response = run_benchmark("my_benchmark", ["model1", "model2"], ["eval1", "eval2"])
print("Benchmark Status:", response)

​

Get Job Status

Get the status of a specific job by job name.

• Endpoint: GET /task/job-status/{job_name}
• Parameters:
  • job_name: The name of the job to check.
• Returns: A dictionary containing the job status.

Example:

from remyxai.api.tasks import get_job_status

status = get_job_status("my_job")
print("Job Status:", status)

​

Train Classifier

Train a classifier model.

• Endpoint: POST /task/classify/{model_name}/{labels}/{model_selector}
• Parameters:
  • model_name: The name of the model to train.
  • labels: A list of labels for the model.
  • model_selector: The selector for the model.
  • hf_dataset: The Hugging Face dataset to use (optional).
• Returns: A dictionary containing the training status.

Example:

from remyxai.api.tasks import train_classifier

response = train_classifier("my_model", ["label1", "label2"], "my_selector")
print("Training Status:", response)

​

Train Detector

Train a detector model.

• Endpoint: POST /task/detect/{model_name}/{labels}/{model_selector}
• Parameters:
  • model_name: The name of the model to train.
  • labels: A list of labels for the model.
  • model_selector: The selector for the model.
  • hf_dataset: The Hugging Face dataset to use (optional).
• Returns: A dictionary containing the training status.

Example:

from remyxai.api.tasks import train_detector

response = train_detector("my_model", ["label1", "label2"], "my_selector")
print("Training Status:", response)

​

Train Generator

Train a generator model.

• Endpoint: POST /task/generate/{model_name}
• Parameters:
  • model_name: The name of the model to train.
  • hf_dataset: The Hugging Face dataset to use.
• Returns: A dictionary containing the training status.

Example:

from remyxai.api.tasks import train_generator

response = train_generator("my_model", "my_dataset")
print("Training Status:", response)

​

Data Composer

Submit a Data Composer task to the server.

• Endpoint: POST /task/datacomposer/{dataset_name}/{num_samples}
• Parameters:
  • dataset_name: The name of the dataset to compose or extend.
  • num_samples: The number of samples to create.
  • context: The context, which could be a Hugging Face dataset or a text prompt (optional).
  • dataset_file: The path to the dataset file to upload (optional).
• Returns: A dictionary containing the task status.

Example:

from remyxai.api.tasks import submit_data_composer_task

response = submit_data_composer_task("my_dataset", 100, context="my_context", dataset_file="path_to_file")
print("Task Status:", response)

​

User Management

This API provides functions to retrieve user profile information and check available credits from the server.

​

Functions

​

get_user_profile() -> dict

Fetches the user profile details from the server.

Returns:

• A dictionary containing the user profile information.

Example:

profile = get_user_profile()
print("User Profile:", profile)

​

get_user_credits() -> dict

Retrieves the available credits for the user from the server.

Returns:

• A dictionary containing the user’s credit information.

Example:

credits = get_user_credits()
print("Available Credits:", credit)

​

Usage Example

from remyxai.api.user import get_user_credits, get_user_credits

# Get the user profile
user_profile = get_user_profile()
print("User Profile:", user_profile)

# Get the user's available credits
user_credits = get_user_credits()
print("User Credits:", user_credits)