This document provides an overview of all supported and available constants for Remyx APIs and tasks. These constants define the models, benchmark, and datasets supported by Remyx.

​
Available Models

Remyx supports evaluation, training, and deployment for these model families:

​
Language Models

  • Bittensor: "BTLMLMHeadModel"
  • BioGPT: "BioGptForCausalLM"
  • Bloom: "BloomForCausalLM"
  • Bloom: "BloomModel"
  • ChatGLM: "ChatGLMModel"
  • CodeGen: "CodeGenForCausalLM"
  • Falcon: "FalconForCausalLM"
  • Falcon Mamba: "FalconMambaForCausalLM"
  • GPT-2 Base: "GPT2Model"
  • GPT BigCode: "GPTBigCodeForCausalLM"
  • GPT BigCode with LM Head: "GPTBigCodeLMHeadModel"
  • GPT-J: "GPTJForCausalLM"
  • GPT-Neo: "GPTNeoForCausalLM"
  • GPT-NeoX: "GPTNeoXForCausalLM"
  • Gemma v2: "Gemma2ForCausalLM"
  • Gemma: "GemmaForCausalLM"
  • LLaMA: "LlamaForCausalLM"
  • MPT: "MPTForCausalLM"
  • Mistral: "MistralForCausalLM"
  • MobileLLM: "MobileLLMForCausalLM"
  • MosaicGPT Base: "MosaicGPT"
  • OPT: "OPTForCausalLM"
  • Phi 3: "Phi3ForCausalLM"
  • Phi 3 Small: "Phi3SmallForCausalLM"
  • Phi: "PhiForCausalLM"
  • QWen with LM Head: "QWenLMHeadModel"
  • Qwen v2: "Qwen2ForCausalLM"
  • Qwen v2 MoE: "Qwen2MoeForCausalLM"
  • RW: "RWForCausalLM"
  • Recurrent Gemma: "RecurrentGemmaForCausalLM"
  • Reformer with LM Head: "ReformerModelWithLMHead"
  • Rwkv v5: "Rwkv5ForCausalLM"
  • Rwkv: "RwkvForCausalLM"
  • StableLM Alpha: "StableLMAlphaForCausalLM"
  • StableLM Epoch: "StableLMEpochForCausalLM"
  • StableLM: "StableLmForCausalLM"
  • Starcoder v2: "Starcoder2ForCausalLM"
  • XGLM: "XGLMForCausalLM"

​
Multi-modal Models

Remyx supports training and deployment for these model families:

  • LLaVA V1.5: "LlavaLlamaForCausalLM"

​
Evaluation Tasks

Remyx currently supports the following evaluation types:

  • MYXMATCH: "myxmatch"
  • BENCHMARK: "benchmark"

​
Benchmark Tasks

The following lighteval evaluation tasks are currently supported:

​
BIG-Bench (BIGBENCH)

  • BIGBENCH_ANALOGICAL_SIMILARITY: "bigbench|analogical_similarity|0|0"
  • BIGBENCH_AUTHORSHIP_VERIFICATION: "bigbench|authorship_verification|0|0"
  • BIGBENCH_CODE_LINE_DESCRIPTION: "bigbench|code_line_description|0|0"
  • BIGBENCH_CONCEPTUAL_COMBINATIONS: "bigbench|conceptual_combinations|0|0"
  • BIGBENCH_LOGICAL_DEDUCTION: "bigbench|logical_deduction|0|0"

​
Harness

  • HARNESS_CAUSAL_JUDGMENT: "harness|bbh:causal_judgment|0|0"
  • HARNESS_DATE_UNDERSTANDING: "harness|bbh:date_understanding|0|0"
  • HARNESS_DISAMBIGUATION_QA: "harness|bbh:disambiguation_qa|0|0"
  • HARNESS_GEOMETRIC_SHAPES: "harness|bbh:geometric_shapes|0|0"
  • HARNESS_LOGICAL_DEDUCTION_FIVE_OBJECTS: "harness|bbh:logical_deduction_five_objects|0|0"

​
HELM

  • HELM_BABI_QA: "helm|babi_qa|0|0"
  • HELM_BBQ: "helm|bbq|0|0"
  • HELM_BOOLQ: "helm|boolq|0|0"
  • HELM_COMMONSENSEQA: "helm|commonsenseqa|0|0"
  • HELM_MMLU_PHILOSOPHY: "helm|mmlu:philosophy|0|0"

​
Leaderboard

  • LEADERBOARD_ARC_CHALLENGE: "leaderboard|arc:challenge|0|0"
  • LEADERBOARD_GSM8K: "leaderboard|gsm8k|0|0"
  • LEADERBOARD_HELLASWAG: "leaderboard|hellaswag|0|0"
  • LEADERBOARD_TRUTHFULQA_MC: "leaderboard|truthfulqa:mc|0|0"
  • LEADERBOARD_MMLU_WORLD_RELIGIONS: "leaderboard|mmlu:world_religions|0|0"

​
LightEval

  • LIGHTEVAL_ARC_EASY: "lighteval|arc:easy|0|0"
  • LIGHTEVAL_ASDIV: "lighteval|asdiv|0|0"
  • LIGHTEVAL_BIGBENCH_MOVIE_RECOMMENDATION: "lighteval|bigbench:movie_recommendation|0|0"
  • LIGHTEVAL_GLUE_COLA: "lighteval|glue:cola|0|0"
  • LIGHTEVAL_TRUTHFULQA_GEN: "lighteval|truthfulqa:gen|0|0"