在大语言模型（LLM）迅速发展的今天，开发者们面临着海量的资源和工具选择。如何高效地筛选和利用这些资源，成为了每一个 LLM 开发者的关键任务。 今天，我们要介绍的 GitHub 仓库——LLM Engineer Toolkit，或许能成为你的得力助手！
https://github.com/KalyanKS-NLP/llm-engineer-toolkit
这个由 KalyanKS-NLP 创建的仓库，精心整理了超过 120 个 LLM 相关的库，并按照类别进行了分类。无论是训练、推理、应用开发，还是数据提取、安全评估，你都能在这里找到对应的工具。
大模型工具划分
LLM Training：专注于 LLM 训练和微调的工具，帮助你更快、更高效地优化模型。
LLM Application Development：从框架到多 API 接入，再到缓存和低代码开发，为应用开发提供全方位支持。
LLM RAG：Retrieval-Augmented Generation（检索增强生成）相关的库，提升模型的知识检索能力。
LLM Inference：推理加速和优化工具，让模型运行更流畅。
LLM Serving：模型部署和推理服务的解决方案。
LLM Data Extraction：数据提取工具，帮助你从各种来源获取高质量数据。
LLM Data Generation：生成合成数据，丰富你的训练集。
LLM Agents：构建智能代理，实现自动化任务和多代理协作。
LLM Evaluation：评估工具，确保模型性能达到预期。
LLM Monitoring：监控模型运行状态，及时发现并解决问题。
LLM Prompts：优化和管理提示词，提升模型输出质量。
LLM Structured Outputs：生成结构化输出，让模型结果更易用。
LLM Safety and Security：保障模型的安全性和可靠性。
LLM Embedding Models：提供先进的文本嵌入模型。
Others：其他实用工具，涵盖更多开发场景。
LLM Training and Fine-Tuning
| Library | Description |
|---|---|
| unsloth | Fine-tune LLMs faster with less memory. |
| PEFT | State-of-the-art Parameter-Efficient Fine-Tuning library. |
| TRL | Train transformer language models with reinforcement learning. |
| Transformers | Transformers provides thousands of pretrained models to perform tasks on different modalities such as text, vision, and audio. |
| Axolotl | Tool designed to streamline post-training for various AI models. |
| LLMBox | A comprehensive library for implementing LLMs, including a unified training pipeline and comprehensive model evaluation. |
| LitGPT | Train and fine-tune LLM lightning fast. |
| Mergoo | A library for easily merging multiple LLM experts, and efficiently train the merged LLM. |
| Llama-Factory | Easy and efficient LLM fine-tuning. |
| Ludwig | Low-code framework for building custom LLMs, neural networks, and other AI models. |
| Txtinstruct | A framework for training instruction-tuned models. |
| Lamini | An integrated LLM inference and tuning platform. |
| XTuring | xTuring provides fast, efficient and simple fine-tuning of open-source LLMs, such as Mistral, LLaMA, GPT-J, and more. |
| RL4LMs | A modular RL library to fine-tune language models to human preferences. |
| DeepSpeed | DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective. |
| torchtune | A PyTorch-native library specifically designed for fine-tuning LLMs. |
| PyTorch Lightning | A library that offers a high-level interface for pretraining and fine-tuning LLMs. |
LLM Application Development
Frameworks
| Library | Description |
|---|---|
| LangChain | LangChain is a framework for developing applications powered by large language models (LLMs). |
| Llama Index | LlamaIndex is a data framework for your LLM applications. |
| HayStack | Haystack is an end-to-end LLM framework that allows you to build applications powered by LLMs, Transformer models, vector search and more. |
| Prompt flow | A suite of development tools designed to streamline the end-to-end development cycle of LLM-based AI applications. |
| Griptape | A modular Python framework for building AI-powered applications. |
| Weave | Weave is a toolkit for developing Generative AI applications. |
| Llama Stack | Build Llama Apps. |
Data Preparation
| Library | Description |
|---|---|
| Data Prep Kit | Data Prep Kit accelerates unstructured data preparation for LLM app developers. Developers can use Data Prep Kit to cleanse, transform, and enrich use case-specific unstructured data to pre-train LLMs, fine-tune LLMs, instruct-tune LLMs, or build RAG applications. |
Multi API Access
| Library | Description |
|---|---|
| LiteLLM | Library to call 100+ LLM APIs in OpenAI format. |
| AI Gateway | A Blazing Fast AI Gateway with integrated Guardrails. Route to 200+ LLMs, 50+ AI Guardrails with 1 fast & friendly API. |
Routers
| Library | Description |
|---|---|
| RouteLLM | Framework for serving and evaluating LLM routers - save LLM costs without compromising quality. Drop-in replacement for OpenAI's client to route simpler queries to cheaper models. |
Memory
| Library | Description |
|---|---|
| mem0 | The Memory layer for your AI apps. |
| Memoripy | An AI memory layer with short- and long-term storage, semantic clustering, and optional memory decay for context-aware applications. |
| Letta (MemGPT) | An open-source framework for building stateful LLM applications with advanced reasoning capabilities and transparent long-term memory |
| Memobase | A user profile-based memory system designed to bring long-term user memory to your Generative AI applications. |
Interface
| Library | Description |
|---|---|
| Streamlit | A faster way to build and share data apps. Streamlit lets you transform Python scripts into interactive web apps in minutes |
| Gradio | Build and share delightful machine learning apps, all in Python. |
| AI SDK UI | Build chat and generative user interfaces. |
| AI-Gradio | Create AI apps powered by various AI providers. |
| Simpleaichat | Python package for easily interfacing with chat apps, with robust features and minimal code complexity. |
| Chainlit | Build production-ready Conversational AI applications in minutes. |
Low Code
| Library | Description |
|---|---|
| LangFlow | LangFlow is a low-code app builder for RAG and multi-agent AI applications. It’s Python-based and agnostic to any model, API, or database. |
Cache
| Library | Description |
|---|---|
| GPTCache | A Library for Creating Semantic Cache for LLM Queries. Slash Your LLM API Costs by 10x $, Boost Speed by 100x. Fully integrated with LangChain and LlamaIndex. |
LLM RAG
| Library | Description |
|---|---|
| FastGraph RAG | Streamlined and promptable Fast GraphRAG framework designed for interpretable, high-precision, agent-driven retrieval workflows. |
| Chonkie | RAG chunking library that is lightweight, lightning-fast, and easy to use. |
| RAGChecker | A Fine-grained Framework For Diagnosing RAG. |
| RAG to Riches | Build, scale, and deploy state-of-the-art Retrieval-Augmented Generation applications. |
| BeyondLLM | Beyond LLM offers an all-in-one toolkit for experimentation, evaluation, and deployment of Retrieval-Augmented Generation (RAG) systems. |
| SQLite-Vec | A vector search SQLite extension that runs anywhere! |
| fastRAG | fastRAG is a research framework for efficient and optimized retrieval-augmented generative pipelines, incorporating state-of-the-art LLMs and Information Retrieval. |
| FlashRAG | A Python Toolkit for Efficient RAG Research. |
| Llmware | Unified framework for building enterprise RAG pipelines with small, specialized models. |
| Rerankers | A lightweight unified API for various reranking models. |
| Vectara | Build Agentic RAG applications. |
LLM Inference
| Library | Description |
|---|---|
| LLM Compressor | Transformers-compatible library for applying various compression algorithms to LLMs for optimized deployment. |
| LightLLM | Python-based LLM inference and serving framework, notable for its lightweight design, easy scalability, and high-speed performance. |
| vLLM | High-throughput and memory-efficient inference and serving engine for LLMs. |
| torchchat | Run PyTorch LLMs locally on servers, desktop, and mobile. |
| TensorRT-LLM | TensorRT-LLM is a library for optimizing Large Language Model (LLM) inference. |
| WebLLM | High-performance In-browser LLM Inference Engine. |
LLM Serving
| Library | Description |
|---|---|
| Langcorn | Serving LangChain LLM apps and agents automagically with FastAPI. |
| LitServe | Lightning-fast serving engine for any AI model of any size. It augments FastAPI with features like batching, streaming, and GPU autoscaling. |
LLM Data Extraction
| Library | Description |
|---|---|
| Crawl4AI | Open-source LLM Friendly Web Crawler & Scraper. |
| ScrapeGraphAI | A web scraping Python library that uses LLM and direct graph logic to create scraping pipelines for websites and local documents (XML, HTML, JSON, Markdown, etc.). |
| Docling | Docling parses documents and exports them to the desired format with ease and speed. |
| Llama Parse | GenAI-native document parser that can parse complex document data for any downstream LLM use case (RAG, agents). |
| PyMuPDF4LLM | PyMuPDF4LLM library makes it easier to extract PDF content in the format you need for LLM & RAG environments. |
| Crawlee | A web scraping and browser automation library. |
| MegaParse | Parser for every type of document. |
| ExtractThinker | Document Intelligence library for LLMs. |
LLM Data Generation
| Library | Description |
|---|---|
| DataDreamer | DataDreamer is a powerful open-source Python library for prompting, synthetic data generation, and training workflows. |
| fabricator | A flexible open-source framework to generate datasets with large language models. |
| Promptwright | Synthetic Dataset Generation Library. |
| EasyInstruct | An Easy-to-use Instruction Processing Framework for Large Language Models. |
LLM Agents
| Library | Description |
|---|---|
| CrewAI | Framework for orchestrating role-playing, autonomous AI agents. |
| LangGraph | Build resilient language agents as graphs. |
| Agno | Build AI Agents with memory, knowledge, tools, and reasoning. Chat with them using a beautiful Agent UI. |
| Agents SDK | Build agentic apps using LLMs with context, tools, hand off to other specialized agents. |
| AutoGen | An open-source framework for building AI agent systems. |
| Smolagents | Library to build powerful agents in a few lines of code. |
| Pydantic AI | Python agent framework to build production grade applications with Generative AI. |
| BeeAI | Build production-ready multi-agent systems in Python. |
| gradio-tools | A Python library for converting Gradio apps into tools that can be leveraged by an LLM-based agent to complete its task. |
| Composio | Production Ready Toolset for AI Agents. |
| Atomic Agents | Building AI agents, atomically. |
| Memary | Open Source Memory Layer For Autonomous Agents. |
| Browser Use | Make websites accessible for AI agents. |
| OpenWebAgent | An Open Toolkit to Enable Web Agents on Large Language Models. |
| Lagent | A lightweight framework for building LLM-based agents. |
| LazyLLM | A Low-code Development Tool For Building Multi-agent LLMs Applications. |
| Swarms | The Enterprise-Grade Production-Ready Multi-Agent Orchestration Framework. |
| ChatArena | ChatArena is a library that provides multi-agent language game environments and facilitates research about autonomous LLM agents and their social interactions. |
| Swarm | Educational framework exploring ergonomic, lightweight multi-agent orchestration. |
| AgentStack | The fastest way to build robust AI agents. |
| Archgw | Intelligent gateway for Agents. |
| Flow | A lightweight task engine for building AI agents. |
| AgentOps | Python SDK for AI agent monitoring. |
| Langroid | Multi-Agent framework. |
| Agentarium | Framework for creating and managing simulations populated with AI-powered agents. |
| Upsonic | Reliable AI agent framework that supports MCP. |
LLM Evaluation
| Library | Description |
|---|---|
| Ragas | Ragas is your ultimate toolkit for evaluating and optimizing Large Language Model (LLM) applications. |
| Giskard | Open-Source Evaluation & Testing for ML & LLM systems. |
| DeepEval | LLM Evaluation Framework |
| Lighteval | All-in-one toolkit for evaluating LLMs. |
| Trulens | Evaluation and Tracking for LLM Experiments |
| PromptBench | A unified evaluation framework for large language models. |
| LangTest | Deliver Safe & Effective Language Models. 60+ Test Types for Comparing LLM & NLP Models on Accuracy, Bias, Fairness, Robustness & More. |
| EvalPlus | A rigorous evaluation framework for LLM4Code. |
| FastChat | An open platform for training, serving, and evaluating large language model-based chatbots. |
| judges | A small library of LLM judges. |
| Evals | Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks. |
| AgentEvals | Evaluators and utilities for evaluating the performance of your agents. |
| LLMBox | A comprehensive library for implementing LLMs, including a unified training pipeline and comprehensive model evaluation. |
| Opik | An open-source end-to-end LLM Development Platform which also includes LLM evaluation. |
LLM Monitoring
| Library | Description |
|---|---|
| MLflow | An open-source end-to-end MLOps/LLMOps Platform for tracking, evaluating, and monitoring LLM applications. |
| Opik | An open-source end-to-end LLM Development Platform which also includes LLM monitoring. |
| LangSmith | Provides tools for logging, monitoring, and improving your LLM applications. |
| Weights & Biases (W&B) | W&B provides features for tracking LLM performance. |
| Helicone | Open source LLM-Observability Platform for Developers. One-line integration for monitoring, metrics, evals, agent tracing, prompt management, playground, etc. |
| Evidently | An open-source ML and LLM observability framework. |
| Phoenix | An open-source AI observability platform designed for experimentation, evaluation, and troubleshooting. |
| Observers | A Lightweight Library for AI Observability. |
LLM Prompts
| Library | Description |
|---|---|
| PCToolkit | A Unified Plug-and-Play Prompt Compression Toolkit of Large Language Models. |
| Selective Context | Selective Context compresses your prompt and context to allow LLMs (such as ChatGPT) to process 2x more content. |
| LLMLingua | Library for compressing prompts to accelerate LLM inference. |
| betterprompt | Test suite for LLM prompts before pushing them to production. |
| Promptify | Solve NLP Problems with LLMs & easily generate different NLP Task prompts for popular generative models like GPT, PaLM, and more with Promptify. |
| PromptSource | PromptSource is a toolkit for creating, sharing, and using natural language prompts. |
| DSPy | DSPy is the open-source framework for programming—rather than prompting—language models. |
| Py-priompt | Prompt design library. |
| Promptimizer | Prompt optimization library. |
LLM Structured Outputs
| Library | Description |
|---|---|
| Instructor | Python library for working with structured outputs from large language models (LLMs). Built on top of Pydantic, it provides a simple, transparent, and user-friendly API. |
| XGrammar | An open-source library for efficient, flexible, and portable structured generation. |
| Outlines | Robust (structured) text generation |
| Guidance | Guidance is an efficient programming paradigm for steering language models. |
| LMQL | A language for constraint-guided and efficient LLM programming. |
| Jsonformer | A Bulletproof Way to Generate Structured JSON from Language Models. |
LLM Safety and Security
| Library | Description |
|---|---|
| JailbreakEval | A collection of automated evaluators for assessing jailbreak attempts. |
| EasyJailbreak | An easy-to-use Python framework to generate adversarial jailbreak prompts. |
| Guardrails | Adding guardrails to large language models. |
| LLM Guard | The Security Toolkit for LLM Interactions. |
| AuditNLG | AuditNLG is an open-source library that can help reduce the risks associated with using generative AI systems for language. |
| NeMo Guardrails | NeMo Guardrails is an open-source toolkit for easily adding programmable guardrails to LLM-based conversational systems. |
| Garak | LLM vulnerability scanner |
| DeepTeam | The LLM Red Teaming Framework |
LLM Embedding Models
| Library | Description |
|---|---|
| Sentence-Transformers | State-of-the-Art Text Embeddings |
| Model2Vec | Fast State-of-the-Art Static Embeddings |
| Text Embedding Inference | A blazing fast inference solution for text embeddings models. TEI enables high-performance extraction for the most popular models, including FlagEmbedding, Ember, GTE and E5. |
Others
| Library | Description |
|---|---|
| Text Machina | A modular and extensible Python framework, designed to aid in the creation of high-quality, unbiased datasets to build robust models for MGT-related tasks such as detection, attribution, and boundary detection. |
| LLM Reasoners | A library for advanced large language model reasoning. |
| EasyEdit | An Easy-to-use Knowledge Editing Framework for Large Language Models. |
| CodeTF | CodeTF: One-stop Transformer Library for State-of-the-art Code LLM. |
| spacy-llm | This package integrates Large Language Models (LLMs) into spaCy, featuring a modular system for fast prototyping and prompting, and turning unstructured responses into robust outputs for various NLP tasks. |
| pandas-ai | Chat with your database (SQL, CSV, pandas, polars, MongoDB, NoSQL, etc.). |
| LLM Transparency Tool | An open-source interactive toolkit for analyzing internal workings of Transformer-based language models. |
| Vanna | Chat with your SQL database. Accurate Text-to-SQL Generation via LLMs using RAG. |
| mergekit | Tools for merging pretrained large language models. |
| MarkLLM | An Open-Source Toolkit for LLM Watermarking. |
| LLMSanitize | An open-source library for contamination detection in NLP datasets and Large Language Models (LLMs). |
| Annotateai | Automatically annotate papers using LLMs. |
| LLM Reasoner | Make any LLM think like OpenAI o1 and DeepSeek R1. |
本文转自：Coggle数据科学，转载此文目的在于传递更多信息，版权归原作者所有。如不支持转载，请联系小编demi@eetrend.com删除。
