tys
/
codefuse-chatbot
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

add features code answer and multi-agents by markdown

This commit is contained in:
shanshi 2023-12-07 20:17:21 +08:00
parent aea1e2b134
commit fa24d1f081
129 changed files with 7102 additions and 2407 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
.gitignore vendored
View File

@ -5,6 +5,9 @@ embedding_models
jupyter_work
model_config.py
server_config.py
internal_start.py
code_base
.DS_Store
.idea
data
tests

2
Dockerfile
View File

@ -12,6 +12,8 @@ RUN apt-get install -y iputils-ping telnetd net-tools vim tcpdump
# RUN service inetutils-inetd start
# service inetutils-inetd status
RUN wget https://oss-cdn.nebula-graph.com.cn/package/3.6.0/nebula-graph-3.6.0.ubuntu1804.amd64.deb
RUN dpkg -i nebula-graph-3.6.0.ubuntu1804.amd64.deb
RUN pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple
RUN pip install -r /home/user/docker_requirements.txt

42
README.md
View File

@ -10,10 +10,11 @@
<br><br>
</p>
本项目是一个开源的 AI 智能助手，专为软件开发的全生命周期而设计，涵盖设计、编码、测试、部署和运维等阶段。通过知识检索、代码检索，工具使用和沙箱执行，Codefuse-ChatBot 能解答您开发过程中的各种专业问题、问答操作周边独立分散平台。
本项目是一个开源的 AI 智能助手，专为软件开发的全生命周期而设计，涵盖设计、编码、测试、部署和运维等阶段。通过知识检索、代码检索，工具使用和沙箱执行，Codefuse-ChatBot不仅能回答您在开发过程中遇到的专业问题，还能通过对话界面协调多个独立分散的平台。
## 🔔 更新
- [2023.12.01] Multi-Agent和代码库检索功能开放
- [2023.11.15] 增加基于本地代码库的问答增强模式
- [2023.09.15] 本地/隔离环境的沙盒功能开放，基于爬虫实现指定url知识检索
@ -29,13 +30,12 @@
💡 本项目旨在通过检索增强生成（Retrieval Augmented Generation，RAG）、工具学习（Tool Learning）和沙盒环境来构建软件开发全生命周期的AI智能助手，涵盖设计、编码、测试、部署和运维等阶段。 逐渐从各处资料查询、独立分散平台操作的传统开发运维模式转变到大模型问答的智能化开发运维模式，改变人们的开发运维习惯。
- 📚 知识库管理：DevOps专业高质量知识库 + 企业级知识库自助构建 + 对话实现快速检索开源/私有技术文档
- ⌨️ 代码知识库管理：支持本地代码库导入和代码结构解析 + 对话实现快速检索本地代码
- 🐳 隔离沙盒环境：实现代码的快速编译执行测试
- 🔄 React范式：支撑代码的自我迭代、自动执行
- 🛠️ Prompt管理：实现各种开发、运维任务的prompt管理
- 🔌 丰富的领域插件：执行各种定制开发任务
- 🚀 对话驱动：需求设计、系分设计、代码生成、开发测试、部署运维自动化
本项目核心差异技术、功能点：
- **🧠 智能调度核心：** 构建了体系链路完善的调度核心，支持多模式一键配置，简化操作流程。
- **💻 代码整库分析：** 实现了仓库级的代码深入理解，以及项目文件级的代码编写与生成，提升了开发效率。
- **📄 文档分析增强：** 融合了文档知识库与知识图谱，通过检索和推理增强，为文档分析提供了更深层次的支持。
- **🔧 垂类专属知识：** 为DevOps领域定制的专属知识库，支持垂类知识库的自助一键构建，便捷实用。
- **🤖 垂类模型兼容：** 针对DevOps领域的小型模型，保证了与DevOps相关平台的兼容性，促进了技术生态的整合。
🌍 依托于开源的 LLM 与 Embedding 模型，本项目可实现基于开源模型的离线私有部署。此外，本项目也支持 OpenAI API 的调用。
@ -57,26 +57,26 @@
## 🧭 技术路线
<div align=center>
<img src="sources/docs_imgs/devops-chatbot-module.png" alt="图片" width="600" height="503">
<img src="sources/docs_imgs/devops-chatbot-module-v2.png" alt="图片" width="600" height="503">
</div>
- 🕷️ **Web Crawl**：实现定期网络文档爬取，确保数据的及时性，并依赖于开源社区的持续补充。
- 🗂️ **DocLoader & TextSplitter**：对从多种来源爬取的数据进行数据清洗、去重和分类，并支持私有文档的导入。
- 🗄️ **Vector Database**：结合Text Embedding模型对文档进行Embedding并在Milvus中存储。
- 🔌 **Connector**：作为调度中心，负责LLM与Vector Database之间的交互调度，基于Langchain技术实现。
- 📝 **Prompt Control**：从开发和运维角度设计，为不同问题分类并为Prompt添加背景，确保答案的可控性和完整性。
- 💬 **LLM**：默认使用GPT-3.5-turbo，并为私有部署和其他涉及隐私的场景提供专有模型选择。
- 🔤 **Text Embedding**：默认采用OpenAI的Text Embedding模型，支持私有部署和其他隐私相关场景，并提供专有模型选择。
- 🚧 **SandBox**：对于生成的输出，如代码，为帮助用户判断其真实性，提供了一个交互验证环境（基于FaaS），并支持用户进行调整。
- 🧠 **Multi-Agent Schedule Core:** 多智能体调度核心，简易配置即可打造交互式智能体。
- 🕷️ **Multi Source Web Crawl:** 多源网络爬虫，提供对指定 URL 的爬取功能，以搜集所需信息。
- 🗂️ **Data Processor:** 数据处理器，轻松完成文档载入、数据清洗，及文本切分，整合不同来源的数据。
- 🔤 **Text Embedding & Index:**：文本嵌入索引，用户可以轻松上传文件进行文档检索，优化文档分析过程。
- 🗄️ **Vector Database & Graph Database:** 向量与图数据库，提供灵活强大的数据管理解决方案。
- 📝 **Prompt Control & Management:**：Prompt 控制与管理，精确定义智能体的上下文环境。
- 🚧 **SandBox:**：沙盒环境，安全地执行代码编译和动作。
- 💬 **LLM:**：智能体大脑，支持多种开源模型和 LLM 接口。
- 🛠️ **API Management:：** API 管理工具，实现对开源组件和运维平台的快速集成。
具体实现明细见：[技术路线明细](sources/readme_docs/roadmap.md)
## 模型接入
## 🌐 模型接入
有需要接入的model，可以提issue
如果您需要集成特定的模型，请通过提交issue来告知我们您的需求。
| model_name | model_size | gpu_memory | quantize | HFhub | ModelScope |
| ------------------ | ---------- | ---------- | -------- | ----- | ---------- |
@ -193,4 +193,4 @@ python start.py
## 🤗 致谢
本项目基于[langchain-chatchat](https://github.com/chatchat-space/Langchain-Chatchat)和[codebox-api](https://github.com/shroominic/codebox-api)，在此深深感谢他们的开源贡献！
本项目基于[langchain-chatchat](https://github.com/chatchat-space/Langchain-Chatchat)和[codebox-api](https://github.com/shroominic/codebox-api)，在此深深感谢他们的开源贡献！

41
README_en.md
View File

@ -9,16 +9,18 @@
</a>
<br><br>
</p>
This project is an open-source AI intelligent assistant, specifically designed for the entire lifecycle of software development, covering design, coding, testing, deployment, and operations. Through knowledge retrieval, tool utilization, and sandbox execution, Codefuse-ChatBot can answer various professional questions during your development process and perform question-answering operations on standalone, disparate platforms.
This project is an open-source AI intelligent assistant, specifically designed for the entire lifecycle of software development, covering design, coding, testing, deployment, and operations. Through knowledge retrieval, tool utilization, and sandbox execution, Codefuse-ChatBot can not only answer professional questions you encounter during the development process but also coordinate multiple independent, dispersed platforms through a conversational interface.
## 🔔 Updates
- [2023.09.15] Sandbox features for local/isolated environments are now available, implementing specified URL knowledge retrieval based on web crawling.
- [2023.12.01] Release of Multi-Agent and codebase retrieval functionalities.
- [2023.11.15] Addition of Q&A enhancement mode based on the local codebase.
- [2023.09.15] Launch of sandbox functionality for local/isolated environments, enabling knowledge retrieval from specified URLs using web crawlers.
## 📜 Contents
- [🤝 Introduction](#-introduction)
- [🧭 Technical Route](#-technical-route)
- [🌐 模型接入](#-模型接入)
- [🌐 Model Integration](#-model-integration)
- [🚀 Quick Start](#-quick-start)
- [🤗 Acknowledgements](#-acknowledgements)
@ -26,11 +28,11 @@ This project is an open-source AI intelligent assistant, specifically designed f
💡 The aim of this project is to construct an AI intelligent assistant for the entire lifecycle of software development, covering design, coding, testing, deployment, and operations, through Retrieval Augmented Generation (RAG), Tool Learning, and sandbox environments. It transitions gradually from the traditional development and operations mode of querying information from various sources and operating on standalone, disparate platforms to an intelligent development and operations mode based on large-model Q&A, changing people's development and operations habits.
- 📚 Knowledge Base Management: Professional high-quality Codefuse knowledge base + enterprise-level knowledge base self-construction + dialogue-based fast retrieval of open-source/private technical documents.
- 🐳 Isolated Sandbox Environment: Enables quick compilation, execution, and testing of code.
- 🔄 React Paradigm: Supports code self-iteration and automatic execution.
- 🛠️ Prompt Management: Manages prompts for various development and operations tasks.
- 🚀 Conversation Driven: Automates requirement design, system analysis design, code generation, development testing, deployment, and operations.
- **🧠 Intelligent Scheduling Core:** Constructed a well-integrated scheduling core system that supports multi-mode one-click configuration, simplifying the operational process.
- **💻 Comprehensive Code Repository Analysis:** Achieved in-depth understanding at the repository level and coding and generation at the project file level, enhancing development efficiency.
- **📄 Enhanced Document Analysis:** Integrated document knowledge bases with knowledge graphs, providing deeper support for document analysis through enhanced retrieval and reasoning.
- **🔧 Industry-Specific Knowledge:** Tailored a specialized knowledge base for the DevOps domain, supporting the self-service one-click construction of industry-specific knowledge bases for convenience and practicality.
- **🤖 Compatible Models for Specific Verticals:** Designed small models specifically for the DevOps field, ensuring compatibility with related DevOps platforms and promoting the integration of the technological ecosystem.
🌍 Relying on open-source LLM and Embedding models, this project can achieve offline private deployments based on open-source models. Additionally, this project also supports the use of the OpenAI API.
@ -47,24 +49,25 @@ This project is an open-source AI intelligent assistant, specifically designed f
## 🧭 Technical Route
<div align=center>
<img src="sources/docs_imgs/devops-chatbot-module.png" alt="Image" width="600" height="503">
<img src="sources/docs_imgs/devops-chatbot-module-v2.png" alt="Image" width="600" height="503">
</div>
- 🕷️ **Web Crawl**: Implements periodic web document crawling to ensure data timeliness and relies on continuous supplementation from the open-source community.
- 🗂️ **DocLoader & TextSplitter**: Cleans, deduplicates, and categorizes data crawled from various sources and supports the import of private documents.
- 🗄️ **Vector Database**: Integrates Text Embedding models to embed documents and store them in Milvus.
- 🔌 **Connector**: Acts as the scheduling center, responsible for coordinating interactions between LLM and Vector Database, implemented based on Langchain technology.
- 📝 **Prompt Control**: Designs from development and operations perspectives, categorizes different problems, and adds backgrounds to prompts to ensure the controllability and completeness of answers.
- 💬 **LLM**: Uses GPT-3.5-turbo by default and provides proprietary model options for private deployments and other privacy-related scenarios.
- 🔤 **Text Embedding**: Uses OpenAI's Text Embedding model by default, supports private deployments and other privacy-related scenarios, and provides proprietary model options.
- 🚧 **SandBox**: For generated outputs, like code, to help users judge their authenticity, an interactive verification environment is provided (based on FaaS), allowing user adjustments.
- 🧠 **Multi-Agent Schedule Core:** Easily configurable to create interactive intelligent agents.
- 🕷️ **Multi Source Web Crawl:** Offers the capability to crawl specified URLs for collecting the required information.
- 🗂️ **Data Processor:** Effortlessly handles document loading, data cleansing, and text segmentation, integrating data from different sources.
- 🔤 **Text Embedding & Index:**：Users can easily upload files for document retrieval, optimizing the document analysis process.
- 🗄️ **Vector Database & Graph Database:** Provides flexible and powerful data management solutions.
- 📝 **Prompt Control & Management:**：Precisely defines the contextual environment for intelligent agents.
- 🚧 **SandBox:**：Safely executes code compilation and actions.
- 💬 **LLM:**：Supports various open-source models and LLM interfaces.
- 🛠️ **API Management:：** Enables rapid integration of open-source components and operational platforms.
For implementation details, see: [Technical Route Details](sources/readme_docs/roadmap.md)
## 模型接入
## 🌐 Model Integration
有需要接入的model，可以提issue
If you need to integrate a specific model, please inform us of your requirements by submitting an issue.
| model_name | model_size | gpu_memory | quantize | HFhub | ModelScope |
| ------------------ | ---------- | ---------- | -------- | ----- | ---------- |

2
configs/__init__.py
View File

@ -1,4 +1,4 @@
from .model_config import *
from .server_config import *
VERSION = "v0.0.1"
VERSION = "v0.1.0"

32
configs/model_config.py.example
View File

@ -1,6 +1,10 @@
import os
import sys
import logging
import torch
import openai
import base64
from .utils import is_running_in_docker
# 日志格式
LOG_FORMAT = "%(asctime)s - %(filename)s[line:%(lineno)d] - %(levelname)s: %(message)s"
logger = logging.getLogger()
@ -8,8 +12,11 @@ logger.setLevel(logging.INFO)
logging.basicConfig(format=LOG_FORMAT)
# os.environ["OPENAI_PROXY"] = "socks5h://127.0.0.1:13659"
os.environ["API_BASE_URL"] = "http://openai.com/v1/chat/completions"
os.environ["OPENAI_API_KEY"] = ""
os.environ["DUCKDUCKGO_PROXY"] = "socks5://127.0.0.1:13659"
os.environ["BAIDU_OCR_API_KEY"] = ""
os.environ["BAIDU_OCR_SECRET_KEY"] = ""
import platform
system_name = platform.system()
@ -38,6 +45,7 @@ LOCAL_MODEL_DIR = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(_
embedding_model_dict = {k: f"/home/user/chatbot/embedding_models/{v}" if is_running_in_docker() else f"{LOCAL_MODEL_DIR}/{v}" for k, v in embedding_model_dict.items()}
# 选用的 Embedding 名称
EMBEDDING_ENGINE = 'openai'
EMBEDDING_MODEL = "text2vec-base"
# Embedding 模型运行设备
@ -99,6 +107,20 @@ llm_model_dict = {
}
LOCAL_LLM_MODEL_DIR = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), "llm_models")
llm_model_dict_c = {}
for k, v in llm_model_dict.items():
v_c = {}
for kk, vv in v.items():
if k=="local_model_path":
v_c[kk] = f"/home/user/chatbot/llm_models/{vv}" if is_running_in_docker() else f"{LOCAL_LLM_MODEL_DIR}/{vv}"
else:
v_c[kk] = vv
llm_model_dict_c[k] = v_c
llm_model_dict = llm_model_dict_c
# LLM 名称
LLM_MODEL = "gpt-3.5-turbo"
USE_FASTCHAT = "gpt" not in LLM_MODEL # 判断是否进行fastchat
@ -129,7 +151,10 @@ JUPYTER_WORK_PATH = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath
# WEB_CRAWL存储路径
WEB_CRAWL_PATH = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), "sources/docs")
for _path in [LOG_PATH, SOURCE_PATH, KB_ROOT_PATH, NLTK_DATA_PATH, JUPYTER_WORK_PATH, WEB_CRAWL_PATH]:
# NEBULA_DATA存储路径
NELUBA_PATH = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), "data/neluba_data")
for _path in [LOG_PATH, SOURCE_PATH, KB_ROOT_PATH, NLTK_DATA_PATH, JUPYTER_WORK_PATH, WEB_CRAWL_PATH, NELUBA_PATH]:
if not os.path.exists(_path):
os.mkdir(_path)
@ -194,6 +219,11 @@ CODE_PROMPT_TEMPLATE = """【指令】根据已知信息来回答问题。
【问题】{question}"""
# 代码解释模版
CODE_INTERPERT_TEMPLATE = '''{code}
解释一下这段代码'''
# API 是否开启跨域，默认为False，如果需要开启，请设置为True
# is open cross domain
OPEN_CROSS_DOMAIN = False

15
configs/server_config.py.example
View File

@ -45,6 +45,21 @@ FSCHAT_OPENAI_API = {
"docker_port": 8888, # model_config.llm_model_dict中模型配置的api_base_url需要与这里一致。
}
# nebula conf
NEBULA_HOST = DEFAULT_BIND_HOST
NEBULA_PORT = 9669
NEBULA_STORAGED_PORT = 9779
NEBULA_USER = 'root'
NEBULA_PASSWORD = ''
NEBULA_GRAPH_SERVER = {
"host": DEFAULT_BIND_HOST,
"port": NEBULA_PORT,
"docker_port": NEBULA_PORT
}
# chroma conf
CHROMA_PERSISTENT_PATH = '/home/user/chatbot/data/chroma_data'
# sandbox api server
SANDBOX_CONTRAINER_NAME = "devopsgpt_sandbox"
SANDBOX_IMAGE_NAME = "devopsgpt:py39"

4
dev_opsgpt/chat/__init__.py
View File

@ -2,12 +2,10 @@ from .base_chat import Chat
from .knowledge_chat import KnowledgeChat
from .llm_chat import LLMChat
from .search_chat import SearchChat
from .tool_chat import ToolChat
from .data_chat import DataChat
from .code_chat import CodeChat
from .agent_chat import AgentChat
__all__ = [
"Chat", "KnowledgeChat", "LLMChat", "SearchChat", "ToolChat", "DataChat", "CodeChat", "AgentChat"
"Chat", "KnowledgeChat", "LLMChat", "SearchChat", "CodeChat", "AgentChat"
]

171
dev_opsgpt/chat/agent_chat.py
View File

@ -1,10 +1,11 @@
from fastapi import Body, Request
from fastapi.responses import StreamingResponse
from typing import List
from typing import List, Union, Dict
from loguru import logger
import importlib
import copy
import json
from pathlib import Path
from configs.model_config import (
llm_model_dict, LLM_MODEL, PROMPT_TEMPLATE,
@ -18,12 +19,12 @@ from dev_opsgpt.tools import (
from dev_opsgpt.connector.phase import BasePhase
from dev_opsgpt.connector.agents import BaseAgent, ReactAgent
from dev_opsgpt.connector.chains import BaseChain
from dev_opsgpt.connector.connector_schema import (
from dev_opsgpt.connector.schema import (
Message,
load_phase_configs, load_chain_configs, load_role_configs
)
from dev_opsgpt.connector.shcema import Memory
from dev_opsgpt.connector.schema import Memory
from dev_opsgpt.utils.common_utils import file_normalize
from dev_opsgpt.chat.utils import History, wrap_done
from dev_opsgpt.connector.configs import PHASE_CONFIGS, AGETN_CONFIGS, CHAIN_CONFIGS
@ -41,6 +42,7 @@ class AgentChat:
) -> None:
self.top_k = top_k
self.stream = stream
self.chatPhase_dict: Dict[str, BasePhase] = {}
def chat(
self,
@ -67,7 +69,8 @@ class AgentChat:
custom_chain_configs: dict = Body({}, description="自定义chain配置"),
custom_role_configs: dict = Body({}, description="自定义role配置"),
history_node_list: List = Body([], description="代码历史相关节点"),
isDetaild: bool = Body([], description="是否输出完整的agent相关内容"),
isDetailed: bool = Body(False, description="是否输出完整的agent相关内容"),
upload_file: Union[str, Path, bytes] = "",
**kargs
) -> Message:
@ -83,11 +86,21 @@ class AgentChat:
# choose tools
tools = toLangchainTools([TOOL_DICT[i] for i in choose_tools if i in TOOL_DICT])
logger.debug(f"upload_file: {upload_file}")
if upload_file:
upload_file_name = upload_file if upload_file and isinstance(upload_file, str) else upload_file.name
for _filename_idx in range(len(upload_file_name), 0, -1):
if upload_file_name[:_filename_idx] in query:
query = query.replace(upload_file_name[:_filename_idx], upload_file_name)
break
input_message = Message(
role_content=query,
role_type="human",
role_name="user",
input_query=query,
origin_query=query,
phase_name=phase_name,
chain_name=chain_name,
do_search=do_search,
@ -101,7 +114,7 @@ class AgentChat:
tools=tools
)
# history memory mangemant
history = Memory([
history = Memory(messages=[
Message(role_name=i["role"], role_type=i["role"], role_content=i["content"])
for i in history
])
@ -128,14 +141,18 @@ class AgentChat:
# "figures": output_message.figures
# }
def chat_iterator(message: Message, local_memory: Memory, isDetaild=False):
def chat_iterator(message: Message, local_memory: Memory, isDetailed=False):
step_content = local_memory.to_str_messages(content_key='step_content', filter_roles=["user"])
final_content = message.role_content
result = {
"answer": "",
"db_docs": [str(doc) for doc in message.db_docs],
"search_docs": [str(doc) for doc in message.search_docs],
"code_docs": [str(doc) for doc in message.code_docs],
"related_nodes": [doc.get_related_node() for idx, doc in enumerate(message.code_docs) if idx==0],
"figures": message.figures
"figures": message.figures,
"step_content": step_content,
"final_content": final_content,
}
@ -146,8 +163,8 @@ class AgentChat:
related_nodes.append(node)
result["related_nodes"] = related_nodes
# logger.debug(f"{result['figures'].keys()}")
message_str = local_memory.to_str_messages(content_key='step_content') if isDetaild else message.role_content
# logger.debug(f"{result['figures'].keys()}, isDetailed: {isDetailed}")
message_str = step_content
if self.stream:
for token in message_str:
result["answer"] = token
@ -157,7 +174,139 @@ class AgentChat:
result["answer"] += token
yield json.dumps(result, ensure_ascii=False)
return StreamingResponse(chat_iterator(output_message, local_memory, isDetaild), media_type="text/event-stream")
return StreamingResponse(chat_iterator(output_message, local_memory, isDetailed), media_type="text/event-stream")
def achat(
self,
query: str = Body(..., description="用户输入", examples=["hello"]),
phase_name: str = Body(..., description="执行场景名称", examples=["chatPhase"]),
chain_name: str = Body(..., description="执行链的名称", examples=["chatChain"]),
history: List[History] = Body(
[], description="历史对话",
examples=[[{"role": "user", "content": "我们来玩成语接龙，我先来，生龙活虎"}]]
),
doc_engine_name: str = Body(..., description="知识库名称", examples=["samples"]),
search_engine_name: str = Body(..., description="搜索引擎名称", examples=["duckduckgo"]),
code_engine_name: str = Body(..., description="代码引擎名称", examples=["samples"]),
cb_search_type: str = Body(..., description="代码查询模式", examples=["tag"]),
top_k: int = Body(VECTOR_SEARCH_TOP_K, description="匹配向量数"),
score_threshold: float = Body(SCORE_THRESHOLD, description="知识库匹配相关度阈值，取值范围在0-1之间，SCORE越小，相关度越高，取到1相当于不筛选，建议设置在0.5左右", ge=0, le=1),
stream: bool = Body(False, description="流式输出"),
local_doc_url: bool = Body(False, description="知识文件返回本地路径(true)或URL(false)"),
choose_tools: List[str] = Body([], description="选择tool的集合"),
do_search: bool = Body(False, description="是否进行搜索"),
do_doc_retrieval: bool = Body(False, description="是否进行知识库检索"),
do_code_retrieval: bool = Body(False, description="是否执行代码检索"),
do_tool_retrieval: bool = Body(False, description="是否执行工具检索"),
custom_phase_configs: dict = Body({}, description="自定义phase配置"),
custom_chain_configs: dict = Body({}, description="自定义chain配置"),
custom_role_configs: dict = Body({}, description="自定义role配置"),
history_node_list: List = Body([], description="代码历史相关节点"),
isDetailed: bool = Body(False, description="是否输出完整的agent相关内容"),
upload_file: Union[str, Path, bytes] = "",
**kargs
) -> Message:
# update configs
phase_configs, chain_configs, agent_configs = self.update_configs(
custom_phase_configs, custom_chain_configs, custom_role_configs)
# choose tools
tools = toLangchainTools([TOOL_DICT[i] for i in choose_tools if i in TOOL_DICT])
logger.debug(f"upload_file: {upload_file}")
if upload_file:
upload_file_name = upload_file if upload_file and isinstance(upload_file, str) else upload_file.name
for _filename_idx in range(len(upload_file_name), 0, -1):
if upload_file_name[:_filename_idx] in query:
query = query.replace(upload_file_name[:_filename_idx], upload_file_name)
break
input_message = Message(
role_content=query,
role_type="human",
role_name="user",
input_query=query,
origin_query=query,
phase_name=phase_name,
chain_name=chain_name,
do_search=do_search,
do_doc_retrieval=do_doc_retrieval,
do_code_retrieval=do_code_retrieval,
do_tool_retrieval=do_tool_retrieval,
doc_engine_name=doc_engine_name,
search_engine_name=search_engine_name,
code_engine_name=code_engine_name,
cb_search_type=cb_search_type,
score_threshold=score_threshold, top_k=top_k,
history_node_list=history_node_list,
tools=tools
)
# history memory mangemant
history = Memory(messages=[
Message(role_name=i["role"], role_type=i["role"], role_content=i["content"])
for i in history
])
# start to execute
if phase_configs[input_message.phase_name]["phase_type"] not in self.chatPhase_dict:
phase_class = getattr(PHASE_MODULE, phase_configs[input_message.phase_name]["phase_type"])
phase = phase_class(input_message.phase_name,
task = input_message.task,
phase_config = phase_configs,
chain_config = chain_configs,
role_config = agent_configs,
do_summary=phase_configs[input_message.phase_name]["do_summary"],
do_code_retrieval=input_message.do_code_retrieval,
do_doc_retrieval=input_message.do_doc_retrieval,
do_search=input_message.do_search,
)
self.chatPhase_dict[phase_configs[input_message.phase_name]["phase_type"]] = phase
else:
phase = self.chatPhase_dict[phase_configs[input_message.phase_name]["phase_type"]]
def chat_iterator(message: Message, local_memory: Memory, isDetailed=False):
step_content = local_memory.to_str_messages(content_key='step_content', filter_roles=["user"])
step_content = "\n\n".join([f"{v}" for parsed_output in local_memory.get_parserd_output_list() for k, v in parsed_output.items() if k not in ["Action Status"]])
final_content = message.role_content
result = {
"answer": "",
"db_docs": [str(doc) for doc in message.db_docs],
"search_docs": [str(doc) for doc in message.search_docs],
"code_docs": [str(doc) for doc in message.code_docs],
"related_nodes": [doc.get_related_node() for idx, doc in enumerate(message.code_docs) if idx==0],
"figures": message.figures,
"step_content": step_content,
"final_content": final_content,
}
related_nodes, has_nodes = [], [ ]
for nodes in result["related_nodes"]:
for node in nodes:
if node not in has_nodes:
related_nodes.append(node)
result["related_nodes"] = related_nodes
# logger.debug(f"{result['figures'].keys()}, isDetailed: {isDetailed}")
message_str = step_content
if self.stream:
for token in message_str:
result["answer"] = token
yield json.dumps(result, ensure_ascii=False)
else:
for token in message_str:
result["answer"] += token
yield json.dumps(result, ensure_ascii=False)
for output_message, local_memory in phase.astep(input_message, history):
# logger.debug(f"output_message: {output_message.role_content}")
# output_message = Message(**output_message)
# local_memory = Memory(**local_memory)
for result in chat_iterator(output_message, local_memory, isDetailed):
yield result
def _chat(self, ):
pass

20
dev_opsgpt/chat/code_chat.py
View File

@ -53,23 +53,23 @@ class CodeChat(Chat):
def _process(self, query: str, history: List[History], model):
'''process'''
codes_res = search_code(query=query, cb_name=self.engine_name, code_limit=self.code_limit,
search_type=self.cb_search_type,
history_node_list=self.history_node_list)
codes = codes_res['related_code']
nodes = codes_res['related_node']
context = codes_res['context']
related_vertices = codes_res['related_vertices']
# update node names
node_names = [node[0] for node in nodes]
self.history_node_list.extend(node_names)
self.history_node_list = list(set(self.history_node_list))
# node_names = [node[0] for node in nodes]
# self.history_node_list.extend(node_names)
# self.history_node_list = list(set(self.history_node_list))
context = "\n".join(codes)
source_nodes = []
for inum, node_info in enumerate(nodes[0:5]):
node_name, node_type, node_score = node_info[0], node_info[1], node_info[2]
source_nodes.append(f'{inum + 1}. 节点名为 {node_name}, 节点类型为 `{node_type}`, 节点得分为 `{node_score}`')
for inum, node_name in enumerate(related_vertices[0:5]):
source_nodes.append(f'{inum + 1}. 节点名: `{node_name}`')
logger.info('history={}'.format(history))
logger.info('message={}'.format([i.to_msg_tuple() for i in history] + [("human", CODE_PROMPT_TEMPLATE)]))
@ -90,6 +90,7 @@ class CodeChat(Chat):
),
engine_name: str = Body(..., description="知识库名称", examples=["samples"]),
code_limit: int = Body(1, examples=['1']),
cb_search_type: str = Body('', examples=['1']),
stream: bool = Body(False, description="流式输出"),
local_doc_url: bool = Body(False, description="知识文件返回本地路径(true)或URL(false)"),
request: Request = None,
@ -100,6 +101,7 @@ class CodeChat(Chat):
self.stream = stream if isinstance(stream, bool) else stream.default
self.local_doc_url = local_doc_url if isinstance(local_doc_url, bool) else local_doc_url.default
self.request = request
self.cb_search_type = cb_search_type
return self._chat(query, history, **kargs)
def _chat(self, query: str, history: List[History], **kargs):

229
dev_opsgpt/chat/data_chat.py
View File

@ -1,229 +0,0 @@
import asyncio
from typing import List
from langchain import LLMChain
from langchain.callbacks import AsyncIteratorCallbackHandler
from langchain.prompts.chat import ChatPromptTemplate
from langchain.agents import AgentType, initialize_agent
from dev_opsgpt.tools import (
WeatherInfo, WorldTimeGetTimezoneByArea, Multiplier,
toLangchainTools, get_tool_schema
)
from .utils import History, wrap_done
from .base_chat import Chat
from loguru import logger
import json, re
from dev_opsgpt.sandbox import PyCodeBox, CodeBoxResponse
from configs.server_config import SANDBOX_SERVER
def get_tool_agent(tools, llm):
return initialize_agent(
tools,
llm,
agent=AgentType.STRUCTURED_CHAT_ZERO_SHOT_REACT_DESCRIPTION,
verbose=True,
)
PROMPT_TEMPLATE = """
`角色`
你是一个数据分析师，借鉴下述步骤，逐步完成数据分析任务的拆解和代码编写，尽可能帮助和准确地回答用户的问题。
数据文件的存放路径为 `./`
`数据分析流程`
- 判断文件是否存在，并读取文件数据
- 输出数据的基本信息，包括但不限于字段、文本、数据类型等
- 输出数据的详细统计信息
- 判断是否需要画图分析，选择合适的字段进行画图
- 判断数据是否需要进行清洗
- 判断数据或图片是否需要保存
...
- 结合数据统计分析结果和画图结果，进行总结和分析这份数据的价值
`要求`
- 每轮选择一个数据分析流程，需要综合考虑上轮和后续的可能影响
- 数据分析流程只提供参考，不要拘泥于它的具体流程，要有自己的思考
- 使用JSON blob来指定一个计划，通过提供task_status关键字（任务状态）、plan关键字（数据分析计划）和code关键字（可执行代码）。
合法的 "task_status" 值: "finished" 表明当前用户问题已被准确回答 或者 "continued" 表明用户问题仍需要进一步分析
`$JSON_BLOB如下所示`
```
{{
"task_status": $TASK_STATUS,
"plan": $PLAN,
"code": ```python\n$CODE```
}}
```
`跟随如下示例`
问题: 输入待回答的问题
行动：$JSON_BLOB
... (重复 行动 N 次，每次只生成一个行动)
行动:
```
{{
"task_status": "finished",
"plan": 我已经可以回答用户问题了，最后回答用户的内容
}}
```
`数据分析，开始`
问题：{query}
"""
PROMPT_TEMPLATE_2 = """
`角色`
你是一个数据分析师，借鉴下述步骤，逐步完成数据分析任务的拆解和代码编写，尽可能帮助和准确地回答用户的问题。
数据文件的存放路径为 `./`
`数据分析流程`
- 判断文件是否存在，并读取文件数据
- 输出数据的基本信息，包括但不限于字段、文本、数据类型等
- 输出数据的详细统计信息
- 判断数据是否需要进行清洗
- 判断是否需要画图分析，选择合适的字段进行画图
- 判断清洗后数据或图片是否需要保存
...
- 结合数据统计分析结果和画图结果，进行总结和分析这份数据的价值
`要求`
- 每轮选择一个数据分析流程，需要综合考虑上轮和后续的可能影响
- 数据分析流程只提供参考，不要拘泥于它的具体流程，要有自己的思考
- 使用JSON blob来指定一个计划，通过提供task_status关键字（任务状态）、plan关键字（数据分析计划）和code关键字（可执行代码）。
合法的 "task_status" 值: "finished" 表明当前用户问题已被准确回答 或者 "continued" 表明用户问题仍需要进一步分析
`$JSON_BLOB如下所示`
```
{{
"task_status": $TASK_STATUS,
"plan": $PLAN,
"code": ```python\n$CODE```
}}
```
`跟随如下示例`
问题: 输入待回答的问题
行动：$JSON_BLOB
... (重复 行动 N 次，每次只生成一个行动)
行动:
```
{{
"task_status": "finished",
"plan": 我已经可以回答用户问题了，最后回答用户的内容
}}
`数据分析，开始`
问题：上传了一份employee_data.csv文件，请对它进行数据分析
问题：{query}
{history}
"""
class DataChat(Chat):
def __init__(
self,
engine_name: str = "",
top_k: int = 1,
stream: bool = False,
) -> None:
super().__init__(engine_name, top_k, stream)
self.tool_prompt = """结合上下文信息，{tools} {input}"""
self.codebox = PyCodeBox(
remote_url=SANDBOX_SERVER["url"],
remote_ip=SANDBOX_SERVER["host"], # "http://localhost",
remote_port=SANDBOX_SERVER["port"],
token="mytoken",
do_code_exe=True,
do_remote=SANDBOX_SERVER["do_remote"]
)
def create_task(self, query: str, history: List[History], model):
'''构建 llm 生成任务'''
logger.debug("content:{}".format([i.to_msg_tuple() for i in history] + [("human", PROMPT_TEMPLATE)]))
chat_prompt = ChatPromptTemplate.from_messages(
[i.to_msg_tuple() for i in history] + [("human", PROMPT_TEMPLATE)]
)
pattern = re.compile(r"```(?:json)?\n(.*?)\n", re.DOTALL)
internal_history = []
retry_nums = 2
while retry_nums >= 0:
if len(internal_history) == 0:
chat_prompt = ChatPromptTemplate.from_messages(
[i.to_msg_tuple() for i in history] + [("human", PROMPT_TEMPLATE)]
)
else:
chat_prompt = ChatPromptTemplate.from_messages(
[i.to_msg_tuple() for i in history] + [("human", PROMPT_TEMPLATE_2)]
)
chain = LLMChain(prompt=chat_prompt, llm=model)
content = chain({"query": query, "history": "\n".join(internal_history)})["text"]
# content = pattern.search(content)
# logger.info(f"content: {content}")
# content = json.loads(content.group(1).strip(), strict=False)
internal_history.append(f"{content}")
refer_info = "\n".join(internal_history)
logger.info(f"refer_info: {refer_info}")
try:
content = content.split("行动:")[-1].split("行动：")[-1]
content = json.loads(content)
except:
content = content.split("行动:")[-1].split("行动：")[-1]
content = eval(content)
if "finished" == content["task_status"]:
break
elif "code" in content:
# elif "```code" in content or "```python" in content:
# code_text = self.codebox.decode_code_from_text(content)
code_text = content["code"]
codebox_res = self.codebox.chat("```"+code_text+"```", do_code_exe=True)
if codebox_res is not None and codebox_res.code_exe_status != 200:
logger.warning(f"{codebox_res.code_exe_response}")
internal_history.append(f"观察： 根据这个报错信息 {codebox_res.code_exe_response}，进行代码修复")
if codebox_res is not None and codebox_res.code_exe_status == 200:
if codebox_res.code_exe_type == "image/png":
base_text = f"```\n{code_text}\n```\n\n"
img_html = "<img src='data:image/png;base64,{}' class='img-fluid'>".format(
codebox_res.code_exe_response
)
internal_history.append(f"观察： {img_html}")
# logger.info('```\n'+code_text+'\n```'+"\n\n"+'```\n'+codebox_res.code_exe_response+'\n```')
else:
internal_history.append(f"观察： {codebox_res.code_exe_response}")
# logger.info('```\n'+code_text+'\n```'+"\n\n"+'```\n'+codebox_res.code_exe_response+'\n```')
else:
internal_history.append(f"观察：下一步应该怎么做？")
retry_nums -= 1
return {"answer": "", "docs": ""}, {"text": "\n".join(internal_history)}
def create_atask(self, query, history, model, callback: AsyncIteratorCallbackHandler):
chat_prompt = ChatPromptTemplate.from_messages(
[i.to_msg_tuple() for i in history] + [("human", PROMPT_TEMPLATE)]
)
chain = LLMChain(prompt=chat_prompt, llm=model)
task = asyncio.create_task(wrap_done(
chain.acall({"input": query}), callback.done
))
return task, {"answer": "", "docs": ""}

84
dev_opsgpt/chat/tool_chat.py
View File

@ -1,84 +0,0 @@
import asyncio
from typing import List
from langchain import LLMChain
from langchain.callbacks import AsyncIteratorCallbackHandler
from langchain.prompts.chat import ChatPromptTemplate
from langchain.agents import AgentType, initialize_agent
import langchain
from langchain.schema import (
AgentAction
)
# langchain.debug = True
from dev_opsgpt.tools import (
TOOL_SETS, TOOL_DICT,
toLangchainTools, get_tool_schema
)
from .utils import History, wrap_done
from .base_chat import Chat
from loguru import logger
def get_tool_agent(tools, llm):
return initialize_agent(
tools,
llm,
agent=AgentType.STRUCTURED_CHAT_ZERO_SHOT_REACT_DESCRIPTION,
verbose=True,
return_intermediate_steps=True
)
class ToolChat(Chat):
def __init__(
self,
engine_name: str = "",
top_k: int = 1,
stream: bool = False,
) -> None:
super().__init__(engine_name, top_k, stream)
self.tool_prompt = """结合上下文信息，{tools} {input}"""
self.tools = toLangchainTools([TOOL_DICT[i] for i in TOOL_SETS if i in TOOL_DICT])
def create_task(self, query: str, history: List[History], model, **kargs):
'''构建 llm 生成任务'''
logger.debug("content:{}".format([i.to_msg_tuple() for i in history] + [("human", "{query}")]))
# chat_prompt = ChatPromptTemplate.from_messages(
# [i.to_msg_tuple() for i in history] + [("human", "{query}")]
# )
tools = kargs.get("tool_sets", [])
tools = toLangchainTools([TOOL_DICT[i] for i in tools if i in TOOL_DICT])
agent = get_tool_agent(tools if tools else self.tools, model)
content = agent(query)
logger.debug(f"content: {content}")
s = ""
if isinstance(content, str):
s = content
else:
for i in content["intermediate_steps"]:
for j in i:
if isinstance(j, AgentAction):
s += j.log + "\n"
else:
s += "Observation: " + str(j) + "\n"
s += "final answer:" + content["output"]
# chain = LLMChain(prompt=chat_prompt, llm=model)
# content = chain({"tools": tools, "input": query})
return {"answer": "", "docs": ""}, {"text": s}
def create_atask(self, query, history, model, callback: AsyncIteratorCallbackHandler):
chat_prompt = ChatPromptTemplate.from_messages(
[i.to_msg_tuple() for i in history] + [("human", self.tool_prompt)]
)
chain = LLMChain(prompt=chat_prompt, llm=model)
task = asyncio.create_task(wrap_done(
chain.acall({"input": query}), callback.done
))
return task, {"answer": "", "docs": ""}

139
dev_opsgpt/codebase_handler/codebase_handler.py
View File

@ -1,139 +0,0 @@
# encoding: utf-8
'''
@author: 温进
@file: codebase_handler.py
@time: 2023/10/23 下午5:05
@desc:
'''
from loguru import logger
import time
import os
from dev_opsgpt.codebase_handler.parser.java_paraser.java_crawler import JavaCrawler
from dev_opsgpt.codebase_handler.parser.java_paraser.java_preprocess import JavaPreprocessor
from dev_opsgpt.codebase_handler.parser.java_paraser.java_dedup import JavaDedup
from dev_opsgpt.codebase_handler.parser.java_paraser.java_parser import JavaParser
from dev_opsgpt.codebase_handler.tagger.tagger import Tagger
from dev_opsgpt.codebase_handler.tagger.tuple_generation import node_edge_update
from dev_opsgpt.codebase_handler.networkx_handler.networkx_handler import NetworkxHandler
from dev_opsgpt.codebase_handler.codedb_handler.local_codedb_handler import LocalCodeDBHandler
class CodeBaseHandler():
def __init__(self, code_name: str, code_path: str = '', cb_root_path: str = '', history_node_list: list = []):
self.nh = None
self.lcdh = None
self.code_name = code_name
self.code_path = code_path
self.codebase_path = cb_root_path + os.sep + code_name
self.graph_path = self.codebase_path + os.sep + 'graph.pk'
self.codedb_path = self.codebase_path + os.sep + 'codedb.pk'
self.tagger = Tagger()
self.history_node_list = history_node_list
def import_code(self, do_save: bool=False, do_load: bool=False) -> bool:
'''
import code to codeBase
@param code_path:
@param do_save:
@param do_load:
@return: True as success; False as failure
'''
if do_load:
logger.info('start load from codebase_path')
load_graph_path = self.graph_path
load_codedb_path = self.codedb_path
st = time.time()
self.nh = NetworkxHandler(graph_path=load_graph_path)
logger.info('generate graph success, rt={}'.format(time.time() - st))
st = time.time()
self.lcdh = LocalCodeDBHandler(db_path=load_codedb_path)
logger.info('generate codedb success, rt={}'.format(time.time() - st))
else:
logger.info('start load from code_path')
st = time.time()
java_code_dict = JavaCrawler.local_java_file_crawler(self.code_path)
logger.info('crawl success, rt={}'.format(time.time() - st))
jp = JavaPreprocessor()
java_code_dict = jp.preprocess(java_code_dict)
jd = JavaDedup()
java_code_dict = jd.dedup(java_code_dict)
st = time.time()
j_parser = JavaParser()
parse_res = j_parser.parse(java_code_dict)
logger.info('parse success, rt={}'.format(time.time() - st))
st = time.time()
tagged_code = self.tagger.generate_tag(parse_res)
node_list, edge_list = node_edge_update(parse_res.values())
logger.info('get node and edge success, rt={}'.format(time.time() - st))
st = time.time()
self.nh = NetworkxHandler(node_list=node_list, edge_list=edge_list)
logger.info('generate graph success, rt={}'.format(time.time() - st))
st = time.time()
self.lcdh = LocalCodeDBHandler(tagged_code)
logger.info('CodeDB load success, rt={}'.format(time.time() - st))
if do_save:
save_graph_path = self.graph_path
save_codedb_path = self.codedb_path
self.nh.save_graph(save_graph_path)
self.lcdh.save_db(save_codedb_path)
def search_code(self, query: str, code_limit: int, history_node_list: list = []):
'''
search code related to query
@param self:
@param query:
@return:
'''
# get query tag
query_tag_list = self.tagger.generate_tag_query(query)
related_node_score_list = self.nh.search_node_with_score(query_tag_list=query_tag_list,
history_node_list=history_node_list)
score_dict = {
i[0]: i[1]
for i in related_node_score_list
}
related_node = [i[0] for i in related_node_score_list]
related_score = [i[1] for i in related_node_score_list]
related_code, code_related_node = self.lcdh.search_by_multi_tag(related_node, lim=code_limit)
related_node = [
(node, self.nh.get_node_type(node), score_dict[node])
for node in code_related_node
]
related_node.sort(key=lambda x: x[2], reverse=True)
logger.info('related_node={}'.format(related_node))
logger.info('related_code={}'.format(related_code))
logger.info('num of code={}'.format(len(related_code)))
return related_code, related_node
def refresh_history(self):
self.history_node_list = []

55
dev_opsgpt/codebase_handler/codedb_handler/local_codedb_handler.py
View File

@ -1,55 +0,0 @@
# encoding: utf-8
'''
@author: 温进
@file: local_codedb_handler.py
@time: 2023/10/23 下午5:05
@desc:
'''
import pickle
class LocalCodeDBHandler:
def __init__(self, tagged_code: dict = {}, db_path: str = ''):
if db_path:
with open(db_path, 'rb') as f:
self.data = pickle.load(f)
else:
self.data = {}
for code, tag in tagged_code.items():
self.data[code] = str(tag)
def search_by_single_tag(self, tag, lim):
res = list()
for k, v in self.data.items():
if tag in v and k not in res:
res.append(k)
if len(res) > lim:
break
return res
def search_by_multi_tag(self, tag_list, lim=3):
res = list()
res_related_node = []
for tag in tag_list:
single_tag_res = self.search_by_single_tag(tag, lim)
for code in single_tag_res:
if code not in res:
res.append(code)
res_related_node.append(tag)
if len(res) >= lim:
break
# reverse order so that most relevant one is close to the query
res = res[0:lim]
res.reverse()
return res, res_related_node
def save_db(self, save_path):
with open(save_path, 'wb') as f:
pickle.dump(self.data, f)
def __len__(self):
return len(self.data)

129
dev_opsgpt/codebase_handler/networkx_handler/networkx_handler.py
View File

@ -1,129 +0,0 @@
# encoding: utf-8
'''
@author: 温进
@file: networkx_handler.py
@time: 2023/10/23 下午5:02
@desc:
'''
import networkx as nx
from loguru import logger
import matplotlib.pyplot as plt
import pickle
from collections import defaultdict
import json
QUERY_SCORE = 10
HISTORY_SCORE = 5
RATIO = 0.5
class NetworkxHandler:
def __init__(self, graph_path: str = '', node_list: list = [], edge_list: list = []):
if graph_path:
self.graph_path = graph_path
with open(graph_path, 'r') as f:
self.G = nx.node_link_graph(json.load(f))
else:
self.G = nx.DiGraph()
self.populate_graph(node_list, edge_list)
logger.debug(
'number of nodes={}, number of edges={}'.format(self.G.number_of_nodes(), self.G.number_of_edges()))
self.query_score = QUERY_SCORE
self.history_score = HISTORY_SCORE
self.ratio = RATIO
def populate_graph(self, node_list, edge_list):
'''
populate graph with node_list and edge_list
'''
self.G.add_nodes_from(node_list)
for edge in edge_list:
self.G.add_edge(edge[0], edge[-1], relation=edge[1])
def draw_graph(self, save_path: str):
'''
draw and save to save_path
'''
sub = plt.subplot(111)
nx.draw(self.G, with_labels=True)
plt.savefig(save_path)
def search_node(self, query_tag_list: list, history_node_list: list = []):
'''
search node by tag_list, search from history_tag neighbors first
> query_tag_list: tag from query
> history_node_list
'''
node_list = set()
# search from history_tag_list first, then all nodes
for tag in query_tag_list:
add = False
for history_node in history_node_list:
connect_node_list: list = self.G.adj[history_node]
connect_node_list.insert(0, history_node)
for connect_node in connect_node_list:
node_name_lim = len(connect_node) if '_' not in connect_node else connect_node.index('_')
node_name = connect_node[0:node_name_lim]
if tag.lower() in node_name.lower():
node_list.add(connect_node)
add = True
if not add:
for node in self.G.nodes():
if tag.lower() in node.lower():
node_list.add(node)
return node_list
def search_node_with_score(self, query_tag_list: list, history_node_list: list = []):
'''
search node by tag_list, search from history_tag neighbors first
> query_tag_list: tag from query
> history_node_list
'''
logger.info('query_tag_list={}, history_node_list={}'.format(query_tag_list, history_node_list))
node_dict = defaultdict(lambda: 0)
# loop over query_tag_list and add node:
for tag in query_tag_list:
for node in self.G.nodes:
if tag.lower() in node.lower():
node_dict[node] += self.query_score
# loop over history_node and add node score
for node in history_node_list:
node_dict[node] += self.history_score
logger.info('temp_res={}'.format(node_dict))
# adj score broadcast
for node in node_dict:
adj_node_list = self.G.adj[node]
for adj_node in adj_node_list:
node_dict[node] += node_dict.get(adj_node, 0) * self.ratio
# sort
node_list = [(node, node_score) for node, node_score in node_dict.items()]
node_list.sort(key=lambda x: x[1], reverse=True)
return node_list
def save_graph(self, save_path: str):
to_save = nx.node_link_data(self.G)
with open(save_path, 'w') as f:
json.dump(to_save, f)
def __len__(self):
return self.G.number_of_nodes()
def get_node_type(self, node_name):
node_type = self.G.nodes[node_name]['type']
return node_type
def refresh_graph(self, ):
with open(self.graph_path, 'r') as f:
self.G = nx.node_link_graph(json.load(f))

7
dev_opsgpt/codebase_handler/parser/java_paraser/__init__.py
View File

@ -1,7 +0,0 @@
# encoding: utf-8
'''
@author: 温进
@file: __init__.py.py
@time: 2023/10/23 下午5:01
@desc:
'''

32
dev_opsgpt/codebase_handler/parser/java_paraser/java_crawler.py
View File

@ -1,32 +0,0 @@
# encoding: utf-8
'''
@author: 温进
@file: java_crawler.py
@time: 2023/10/23 下午5:02
@desc:
'''
import os
import glob
from loguru import logger
class JavaCrawler:
@staticmethod
def local_java_file_crawler(path: str):
'''
read local java file in path
> path: path to crawl, must be absolute path like A/B/C
< dict of java code string
'''
java_file_list = glob.glob('{path}{sep}**{sep}*.java'.format(path=path, sep=os.path.sep), recursive=True)
java_code_dict = {}
logger.debug('number of file={}'.format(len(java_file_list)))
# logger.debug(java_file_list)
for java_file in java_file_list:
with open(java_file) as f:
java_code = ''.join(f.readlines())
java_code_dict[java_file] = java_code
return java_code_dict

15
dev_opsgpt/codebase_handler/parser/java_paraser/java_dedup.py
View File

@ -1,15 +0,0 @@
# encoding: utf-8
'''
@author: 温进
@file: java_dedup.py
@time: 2023/10/23 下午5:02
@desc:
'''
class JavaDedup:
def __init__(self):
pass
def dedup(self, java_code_dict):
return java_code_dict

14
dev_opsgpt/codebase_handler/parser/java_paraser/java_preprocess.py
View File

@ -1,14 +0,0 @@
# encoding: utf-8
'''
@author: 温进
@file: java_preprocess.py
@time: 2023/10/23 下午5:04
@desc:
'''
class JavaPreprocessor:
def __init__(self):
pass
def preprocess(self, java_code_dict):
return java_code_dict

7
dev_opsgpt/codebase_handler/tagger/__init__.py
View File

@ -1,7 +0,0 @@
# encoding: utf-8
'''
@author: 温进
@file: __init__.py.py
@time: 2023/10/23 下午5:00
@desc:
'''

48
dev_opsgpt/codebase_handler/tagger/tagger.py
View File

@ -1,48 +0,0 @@
# encoding: utf-8
'''
@author: 温进
@file: tagger.py
@time: 2023/10/23 下午5:01
@desc:
'''
import re
from loguru import logger
class Tagger:
def __init__(self):
pass
def generate_tag(self, parse_res_dict: dict):
'''
generate tag from parse_res
'''
res = {}
for java_code, parse_res in parse_res_dict.items():
tag = {}
tag['pac_name'] = parse_res.get('pac_name')
tag['class_name'] = set(parse_res.get('class_name_list'))
tag['func_name'] = set()
for _, func_name_list in parse_res.get('func_name_dict', {}).items():
tag['func_name'].update(func_name_list)
res[java_code] = tag
return res
def generate_tag_query(self, query):
'''
generate tag from query
'''
# simple extract english
tag_list = re.findall(r'[a-zA-Z\_\.]+', query)
tag_list = list(set(tag_list))
return tag_list
if __name__ == '__main__':
tagger = Tagger()
logger.debug(tagger.generate_tag_query('com.CheckHolder 有哪些函数'))

51
dev_opsgpt/codebase_handler/tagger/tuple_generation.py
View File

@ -1,51 +0,0 @@
# encoding: utf-8
'''
@author: 温进
@file: tuple_generation.py
@time: 2023/10/23 下午5:01
@desc:
'''
def node_edge_update(parse_res_list: list, node_list: list = list(), edge_list: list = list()):
'''
generate node and edge by parse_res
< node: list of string node
< edge: (node_st, relation, node_ed)
'''
node_dict = {i: j for i, j in node_list}
for single_parse_res in parse_res_list:
pac_name = single_parse_res['pac_name']
node_dict[pac_name] = {'type': 'package'}
# class_name
for class_name in single_parse_res['class_name_list']:
node_dict[class_name] = {'type': 'class'}
edge_list.append((pac_name, 'contain', class_name))
edge_list.append((class_name, 'inside', pac_name))
# func_name
for class_name, func_name_list in single_parse_res['func_name_dict'].items():
node_list.append(class_name)
for func_name in func_name_list:
node_dict[func_name] = {'type': 'func'}
edge_list.append((class_name, 'contain', func_name))
edge_list.append((func_name, 'inside', class_name))
# depend
for depend_pac_name in single_parse_res['import_pac_name_list']:
if depend_pac_name.endswith('*'):
depend_pac_name = depend_pac_name[0:-2]
if depend_pac_name in node_dict:
continue
else:
node_dict[depend_pac_name] = {'type': 'unknown'}
edge_list.append((pac_name, 'depend', depend_pac_name))
edge_list.append((depend_pac_name, 'beDepended', pac_name))
node_list = [(i, j) for i, j in node_dict.items()]
return node_list, edge_list

2
dev_opsgpt/codebase_handler/codedb_handler/__init__.py → dev_opsgpt/codechat/__init__.py
View File

@ -2,6 +2,6 @@
'''
@author: 温进
@file: __init__.py.py
@time: 2023/10/23 下午5:04
@time: 2023/11/21 下午2:01
@desc:
'''

2
dev_opsgpt/codebase_handler/networkx_handler/__init__.py → dev_opsgpt/codechat/code_analyzer/__init__.py
View File

@ -2,6 +2,6 @@
'''
@author: 温进
@file: __init__.py.py
@time: 2023/10/23 下午5:00
@time: 2023/11/21 下午2:27
@desc:
'''

219
dev_opsgpt/codechat/code_analyzer/code_analyzer.py Normal file
View File

@ -0,0 +1,219 @@
# encoding: utf-8
'''
@author: 温进
@file: code_analyzer.py
@time: 2023/11/21 下午2:27
@desc:
'''
import time
from loguru import logger
from dev_opsgpt.codechat.code_analyzer.code_static_analysis import CodeStaticAnalysis
from dev_opsgpt.codechat.code_analyzer.code_intepreter import CodeIntepreter
from dev_opsgpt.codechat.code_analyzer.code_preprocess import CodePreprocessor
from dev_opsgpt.codechat.code_analyzer.code_dedup import CodeDedup
class CodeAnalyzer:
def __init__(self, language: str):
self.code_preprocessor = CodePreprocessor()
self.code_debup = CodeDedup()
self.code_interperter = CodeIntepreter()
self.code_static_analyzer = CodeStaticAnalysis(language=language)
def analyze(self, code_dict: dict, do_interpret: bool = True):
'''
analyze code
@param code_dict: {fp: code_text}
@param do_interpret: Whether to get analysis result
@return:
'''
# preprocess and dedup
st = time.time()
code_dict = self.code_preprocessor.preprocess(code_dict)
code_dict = self.code_debup.dedup(code_dict)
logger.debug('preprocess and dedup rt={}'.format(time.time() - st))
# static analysis
st = time.time()
static_analysis_res = self.code_static_analyzer.analyze(code_dict)
logger.debug('static analysis rt={}'.format(time.time() - st))
# interpretation
if do_interpret:
logger.info('start interpret code')
st = time.time()
code_list = list(code_dict.values())
interpretation = self.code_interperter.get_intepretation_batch(code_list)
logger.debug('interpret rt={}'.format(time.time() - st))
else:
interpretation = {i: '' for i in code_dict.values()}
return static_analysis_res, interpretation
if __name__ == '__main__':
engine = 'openai'
language = 'java'
code_dict = {'1': '''package com.theokanning.openai.client;
import com.theokanning.openai.DeleteResult;
import com.theokanning.openai.OpenAiResponse;
import com.theokanning.openai.audio.TranscriptionResult;
import com.theokanning.openai.audio.TranslationResult;
import com.theokanning.openai.billing.BillingUsage;
import com.theokanning.openai.billing.Subscription;
import com.theokanning.openai.completion.CompletionRequest;
import com.theokanning.openai.completion.CompletionResult;
import com.theokanning.openai.completion.chat.ChatCompletionRequest;
import com.theokanning.openai.completion.chat.ChatCompletionResult;
import com.theokanning.openai.edit.EditRequest;
import com.theokanning.openai.edit.EditResult;
import com.theokanning.openai.embedding.EmbeddingRequest;
import com.theokanning.openai.embedding.EmbeddingResult;
import com.theokanning.openai.engine.Engine;
import com.theokanning.openai.file.File;
import com.theokanning.openai.fine_tuning.FineTuningEvent;
import com.theokanning.openai.fine_tuning.FineTuningJob;
import com.theokanning.openai.fine_tuning.FineTuningJobRequest;
import com.theokanning.openai.finetune.FineTuneEvent;
import com.theokanning.openai.finetune.FineTuneRequest;
import com.theokanning.openai.finetune.FineTuneResult;
import com.theokanning.openai.image.CreateImageRequest;
import com.theokanning.openai.image.ImageResult;
import com.theokanning.openai.model.Model;
import com.theokanning.openai.moderation.ModerationRequest;
import com.theokanning.openai.moderation.ModerationResult;
import io.reactivex.Single;
import okhttp3.MultipartBody;
import okhttp3.RequestBody;
import okhttp3.ResponseBody;
import retrofit2.Call;
import retrofit2.http.*;
import java.time.LocalDate;
public interface OpenAiApi {
@GET("v1/models")
Single<OpenAiResponse<Model>> listModels();
@GET("/v1/models/{model_id}")
Single<Model> getModel(@Path("model_id") String modelId);
@POST("/v1/completions")
Single<CompletionResult> createCompletion(@Body CompletionRequest request);
@Streaming
@POST("/v1/completions")
Call<ResponseBody> createCompletionStream(@Body CompletionRequest request);
@POST("/v1/chat/completions")
Single<ChatCompletionResult> createChatCompletion(@Body ChatCompletionRequest request);
@Streaming
@POST("/v1/chat/completions")
Call<ResponseBody> createChatCompletionStream(@Body ChatCompletionRequest request);
@Deprecated
@POST("/v1/engines/{engine_id}/completions")
Single<CompletionResult> createCompletion(@Path("engine_id") String engineId, @Body CompletionRequest request);
@POST("/v1/edits")
Single<EditResult> createEdit(@Body EditRequest request);
@Deprecated
@POST("/v1/engines/{engine_id}/edits")
Single<EditResult> createEdit(@Path("engine_id") String engineId, @Body EditRequest request);
@POST("/v1/embeddings")
Single<EmbeddingResult> createEmbeddings(@Body EmbeddingRequest request);
@Deprecated
@POST("/v1/engines/{engine_id}/embeddings")
Single<EmbeddingResult> createEmbeddings(@Path("engine_id") String engineId, @Body EmbeddingRequest request);
@GET("/v1/files")
Single<OpenAiResponse<File>> listFiles();
@Multipart
@POST("/v1/files")
Single<File> uploadFile(@Part("purpose") RequestBody purpose, @Part MultipartBody.Part file);
@DELETE("/v1/files/{file_id}")
Single<DeleteResult> deleteFile(@Path("file_id") String fileId);
@GET("/v1/files/{file_id}")
Single<File> retrieveFile(@Path("file_id") String fileId);
@Streaming
@GET("/v1/files/{file_id}/content")
Single<ResponseBody> retrieveFileContent(@Path("file_id") String fileId);
@POST("/v1/fine_tuning/jobs")
Single<FineTuningJob> createFineTuningJob(@Body FineTuningJobRequest request);
@GET("/v1/fine_tuning/jobs")
Single<OpenAiResponse<FineTuningJob>> listFineTuningJobs();
@GET("/v1/fine_tuning/jobs/{fine_tuning_job_id}")
Single<FineTuningJob> retrieveFineTuningJob(@Path("fine_tuning_job_id") String fineTuningJobId);
@POST("/v1/fine_tuning/jobs/{fine_tuning_job_id}/cancel")
Single<FineTuningJob> cancelFineTuningJob(@Path("fine_tuning_job_id") String fineTuningJobId);
@GET("/v1/fine_tuning/jobs/{fine_tuning_job_id}/events")
Single<OpenAiResponse<FineTuningEvent>> listFineTuningJobEvents(@Path("fine_tuning_job_id") String fineTuningJobId);
@Deprecated
@POST("/v1/fine-tunes")
Single<FineTuneResult> createFineTune(@Body FineTuneRequest request);
@POST("/v1/completions")
Single<CompletionResult> createFineTuneCompletion(@Body CompletionRequest request);
@Deprecated
@GET("/v1/fine-tunes")
Single<OpenAiResponse<FineTuneResult>> listFineTunes();
@Deprecated
@GET("/v1/fine-tunes/{fine_tune_id}")
Single<FineTuneResult> retrieveFineTune(@Path("fine_tune_id") String fineTuneId);
@Deprecated
@POST("/v1/fine-tunes/{fine_tune_id}/cancel")
Single<FineTuneResult> cancelFineTune(@Path("fine_tune_id") String fineTuneId);
@Deprecated
@GET("/v1/fine-tunes/{fine_tune_id}/events")
Single<OpenAiResponse<FineTuneEvent>> listFineTuneEvents(@Path("fine_tune_id") String fineTuneId);
@DELETE("/v1/models/{fine_tune_id}")
Single<DeleteResult> deleteFineTune(@Path("fine_tune_id") String fineTuneId);
@POST("/v1/images/generations")
Single<ImageResult> createImage(@Body CreateImageRequest request);
@POST("/v1/images/edits")
Single<ImageResult> createImageEdit(@Body RequestBody requestBody);
@POST("/v1/images/variations")
Single<ImageResult> createImageVariation(@Body RequestBody requestBody);
@POST("/v1/audio/transcriptions")
Single<TranscriptionResult> createTranscription(@Body RequestBody requestBody);
@POST("/v1/audio/translations")
Single<TranslationResult> createTranslation(@Body RequestBody requestBody);
@POST("/v1/moderations")
Single<ModerationResult> createModeration(@Body ModerationRequest request);
@Deprecated
@GET("v1/engines")
Single<OpenAiResponse<Engine>> getEngines();
@Deprecated
@GET("/v1/engines/{engine_id}")
Single<Engine> getEngine(@Path("engine_id") String engineId);
/**
* Account information inquiry: It contains total amount (in US dollars) and other information.
*
* @return
*/
@Deprecated
@GET("v1/dashboard/billing/subscription")
Single<Subscription> subscription();
/**
* Account call interface consumption amount inquiry.
* totalUsage = Total amount used by the account (in US cents).
*
* @param starDate
* @param endDate
* @return Consumption amount information.
*/
@Deprecated
@GET("v1/dashboard/billing/usage")
Single<BillingUsage> billingUsage(@Query("start_date") LocalDate starDate, @Query("end_date") LocalDate endDate);
}''', '2': '''
package com.theokanning.openai;
/**
* OkHttp Interceptor that adds an authorization token header
*
* @deprecated Use {@link com.theokanning.openai.client.AuthenticationInterceptor}
*/
@Deprecated
public class AuthenticationInterceptor extends com.theokanning.openai.client.AuthenticationInterceptor {
AuthenticationInterceptor(String token) {
super(token);
}
}
'''}
ca = CodeAnalyzer(engine, language)
res = ca.analyze(code_dict)
logger.debug(res)

31
dev_opsgpt/codechat/code_analyzer/code_dedup.py Normal file
View File

@ -0,0 +1,31 @@
# encoding: utf-8
'''
@author: 温进
@file: code_dedup.py
@time: 2023/11/21 下午2:27
@desc:
'''
# encoding: utf-8
'''
@author: 温进
@file: java_dedup.py
@time: 2023/10/23 下午5:02
@desc:
'''
class CodeDedup:
def __init__(self):
pass
def dedup(self, code_dict):
code_dict = self.exact_dedup(code_dict)
return code_dict
def exact_dedup(self, code_dict):
res = {}
for fp, code_text in code_dict.items():
if code_text not in res.values():
res[fp] = code_text
return res

229
dev_opsgpt/codechat/code_analyzer/code_intepreter.py Normal file
View File

@ -0,0 +1,229 @@
# encoding: utf-8
'''
@author: 温进
@file: code_intepreter.py
@time: 2023/11/22 上午11:57
@desc:
'''
from loguru import logger
from langchain.schema import (
HumanMessage,
)
from configs.model_config import CODE_INTERPERT_TEMPLATE
from dev_opsgpt.llm_models.openai_model import getChatModel
class CodeIntepreter:
def __init__(self):
pass
def get_intepretation(self, code_list):
'''
get intepretion of code
@param code_list:
@return:
'''
chat_model = getChatModel()
res = {}
for code in code_list:
message = CODE_INTERPERT_TEMPLATE.format(code=code)
message = [HumanMessage(content=message)]
chat_res = chat_model.predict_messages(message)
content = chat_res.content
res[code] = content
return res
def get_intepretation_batch(self, code_list):
'''
get intepretion of code
@param code_list:
@return:
'''
chat_model = getChatModel()
res = {}
messages = []
for code in code_list:
message = CODE_INTERPERT_TEMPLATE.format(code=code)
messages.append(message)
chat_ress = chat_model.batch(messages)
for chat_res, code in zip(chat_ress, code_list):
res[code] = chat_res.content
return res
if __name__ == '__main__':
engine = 'openai'
code_list = ['''package com.theokanning.openai.client;
import com.theokanning.openai.DeleteResult;
import com.theokanning.openai.OpenAiResponse;
import com.theokanning.openai.audio.TranscriptionResult;
import com.theokanning.openai.audio.TranslationResult;
import com.theokanning.openai.billing.BillingUsage;
import com.theokanning.openai.billing.Subscription;
import com.theokanning.openai.completion.CompletionRequest;
import com.theokanning.openai.completion.CompletionResult;
import com.theokanning.openai.completion.chat.ChatCompletionRequest;
import com.theokanning.openai.completion.chat.ChatCompletionResult;
import com.theokanning.openai.edit.EditRequest;
import com.theokanning.openai.edit.EditResult;
import com.theokanning.openai.embedding.EmbeddingRequest;
import com.theokanning.openai.embedding.EmbeddingResult;
import com.theokanning.openai.engine.Engine;
import com.theokanning.openai.file.File;
import com.theokanning.openai.fine_tuning.FineTuningEvent;
import com.theokanning.openai.fine_tuning.FineTuningJob;
import com.theokanning.openai.fine_tuning.FineTuningJobRequest;
import com.theokanning.openai.finetune.FineTuneEvent;
import com.theokanning.openai.finetune.FineTuneRequest;
import com.theokanning.openai.finetune.FineTuneResult;
import com.theokanning.openai.image.CreateImageRequest;
import com.theokanning.openai.image.ImageResult;
import com.theokanning.openai.model.Model;
import com.theokanning.openai.moderation.ModerationRequest;
import com.theokanning.openai.moderation.ModerationResult;
import io.reactivex.Single;
import okhttp3.MultipartBody;
import okhttp3.RequestBody;
import okhttp3.ResponseBody;
import retrofit2.Call;
import retrofit2.http.*;
import java.time.LocalDate;
public interface OpenAiApi {
@GET("v1/models")
Single<OpenAiResponse<Model>> listModels();
@GET("/v1/models/{model_id}")
Single<Model> getModel(@Path("model_id") String modelId);
@POST("/v1/completions")
Single<CompletionResult> createCompletion(@Body CompletionRequest request);
@Streaming
@POST("/v1/completions")
Call<ResponseBody> createCompletionStream(@Body CompletionRequest request);
@POST("/v1/chat/completions")
Single<ChatCompletionResult> createChatCompletion(@Body ChatCompletionRequest request);
@Streaming
@POST("/v1/chat/completions")
Call<ResponseBody> createChatCompletionStream(@Body ChatCompletionRequest request);
@Deprecated
@POST("/v1/engines/{engine_id}/completions")
Single<CompletionResult> createCompletion(@Path("engine_id") String engineId, @Body CompletionRequest request);
@POST("/v1/edits")
Single<EditResult> createEdit(@Body EditRequest request);
@Deprecated
@POST("/v1/engines/{engine_id}/edits")
Single<EditResult> createEdit(@Path("engine_id") String engineId, @Body EditRequest request);
@POST("/v1/embeddings")
Single<EmbeddingResult> createEmbeddings(@Body EmbeddingRequest request);
@Deprecated
@POST("/v1/engines/{engine_id}/embeddings")
Single<EmbeddingResult> createEmbeddings(@Path("engine_id") String engineId, @Body EmbeddingRequest request);
@GET("/v1/files")
Single<OpenAiResponse<File>> listFiles();
@Multipart
@POST("/v1/files")
Single<File> uploadFile(@Part("purpose") RequestBody purpose, @Part MultipartBody.Part file);
@DELETE("/v1/files/{file_id}")
Single<DeleteResult> deleteFile(@Path("file_id") String fileId);
@GET("/v1/files/{file_id}")
Single<File> retrieveFile(@Path("file_id") String fileId);
@Streaming
@GET("/v1/files/{file_id}/content")
Single<ResponseBody> retrieveFileContent(@Path("file_id") String fileId);
@POST("/v1/fine_tuning/jobs")
Single<FineTuningJob> createFineTuningJob(@Body FineTuningJobRequest request);
@GET("/v1/fine_tuning/jobs")
Single<OpenAiResponse<FineTuningJob>> listFineTuningJobs();
@GET("/v1/fine_tuning/jobs/{fine_tuning_job_id}")
Single<FineTuningJob> retrieveFineTuningJob(@Path("fine_tuning_job_id") String fineTuningJobId);
@POST("/v1/fine_tuning/jobs/{fine_tuning_job_id}/cancel")
Single<FineTuningJob> cancelFineTuningJob(@Path("fine_tuning_job_id") String fineTuningJobId);
@GET("/v1/fine_tuning/jobs/{fine_tuning_job_id}/events")
Single<OpenAiResponse<FineTuningEvent>> listFineTuningJobEvents(@Path("fine_tuning_job_id") String fineTuningJobId);
@Deprecated
@POST("/v1/fine-tunes")
Single<FineTuneResult> createFineTune(@Body FineTuneRequest request);
@POST("/v1/completions")
Single<CompletionResult> createFineTuneCompletion(@Body CompletionRequest request);
@Deprecated
@GET("/v1/fine-tunes")
Single<OpenAiResponse<FineTuneResult>> listFineTunes();
@Deprecated
@GET("/v1/fine-tunes/{fine_tune_id}")
Single<FineTuneResult> retrieveFineTune(@Path("fine_tune_id") String fineTuneId);
@Deprecated
@POST("/v1/fine-tunes/{fine_tune_id}/cancel")
Single<FineTuneResult> cancelFineTune(@Path("fine_tune_id") String fineTuneId);
@Deprecated
@GET("/v1/fine-tunes/{fine_tune_id}/events")
Single<OpenAiResponse<FineTuneEvent>> listFineTuneEvents(@Path("fine_tune_id") String fineTuneId);
@DELETE("/v1/models/{fine_tune_id}")
Single<DeleteResult> deleteFineTune(@Path("fine_tune_id") String fineTuneId);
@POST("/v1/images/generations")
Single<ImageResult> createImage(@Body CreateImageRequest request);
@POST("/v1/images/edits")
Single<ImageResult> createImageEdit(@Body RequestBody requestBody);
@POST("/v1/images/variations")
Single<ImageResult> createImageVariation(@Body RequestBody requestBody);
@POST("/v1/audio/transcriptions")
Single<TranscriptionResult> createTranscription(@Body RequestBody requestBody);
@POST("/v1/audio/translations")
Single<TranslationResult> createTranslation(@Body RequestBody requestBody);
@POST("/v1/moderations")
Single<ModerationResult> createModeration(@Body ModerationRequest request);
@Deprecated
@GET("v1/engines")
Single<OpenAiResponse<Engine>> getEngines();
@Deprecated
@GET("/v1/engines/{engine_id}")
Single<Engine> getEngine(@Path("engine_id") String engineId);
/**
* Account information inquiry: It contains total amount (in US dollars) and other information.
*
* @return
*/
@Deprecated
@GET("v1/dashboard/billing/subscription")
Single<Subscription> subscription();
/**
* Account call interface consumption amount inquiry.
* totalUsage = Total amount used by the account (in US cents).
*
* @param starDate
* @param endDate
* @return Consumption amount information.
*/
@Deprecated
@GET("v1/dashboard/billing/usage")
Single<BillingUsage> billingUsage(@Query("start_date") LocalDate starDate, @Query("end_date") LocalDate endDate);
}''', '''
package com.theokanning.openai;
/**
* OkHttp Interceptor that adds an authorization token header
*
* @deprecated Use {@link com.theokanning.openai.client.AuthenticationInterceptor}
*/
@Deprecated
public class AuthenticationInterceptor extends com.theokanning.openai.client.AuthenticationInterceptor {
AuthenticationInterceptor(String token) {
super(token);
}
}
''']
ci = CodeIntepreter(engine)
res = ci.get_intepretation_batch(code_list)
logger.debug(res)

14
dev_opsgpt/codechat/code_analyzer/code_preprocess.py Normal file
View File

@ -0,0 +1,14 @@
# encoding: utf-8
'''
@author: 温进
@file: code_preprocess.py
@time: 2023/11/21 下午2:28
@desc:
'''
class CodePreprocessor:
def __init__(self):
pass
def preprocess(self, code_dict):
return code_dict

26
dev_opsgpt/codechat/code_analyzer/code_static_analysis.py Normal file
View File

@ -0,0 +1,26 @@
# encoding: utf-8
'''
@author: 温进
@file: code_static_analysis.py
@time: 2023/11/21 下午2:28
@desc:
'''
from dev_opsgpt.codechat.code_analyzer.language_static_analysis import *
class CodeStaticAnalysis:
def __init__(self, language):
self.language = language
def analyze(self, code_dict):
'''
analyze code
@param code_list:
@return:
'''
if self.language == 'java':
analyzer = JavaStaticAnalysis()
else:
raise ValueError('language should be one of [java]')
analyze_res = analyzer.analyze(code_dict)
return analyze_res

14
dev_opsgpt/codechat/code_analyzer/language_static_analysis/__init__.py Normal file
View File

@ -0,0 +1,14 @@
# encoding: utf-8
'''
@author: 温进
@file: __init__.py.py
@time: 2023/11/21 下午4:24
@desc:
'''
from .java_static_analysis import JavaStaticAnalysis
__all__ = [
'JavaStaticAnalysis'
]

45
dev_opsgpt/codebase_handler/parser/java_paraser/java_parser.py → dev_opsgpt/codechat/code_analyzer/language_static_analysis/java_static_analysis.py
View File

@ -1,26 +1,24 @@
# encoding: utf-8
'''
@author: 温进
@file: java_parser.py
@time: 2023/10/23 下午5:03
@file: java_static_analysis.py
@time: 2023/11/21 下午4:25
@desc:
'''
import json
import javalang
import glob
import os
from loguru import logger
import javalang
class JavaParser:
class JavaStaticAnalysis:
def __init__(self):
pass
def parse(self, java_code_list):
def analyze(self, java_code_dict):
'''
parse java code and extract entity
'''
tree_dict = self.preparse(java_code_list)
tree_dict = self.preparse(java_code_dict)
res = self.multi_java_code_parse(tree_dict)
return res
@ -38,15 +36,15 @@ class JavaParser:
continue
if tree.package is not None:
tree_dict[java_code] = tree
tree_dict[fp] = {'code': java_code, 'tree': tree}
logger.info('success parse {} files'.format(len(tree_dict)))
return tree_dict
def single_java_code_parse(self, tree):
def single_java_code_parse(self, tree, fp):
'''
parse single code file
> tree: javalang parse result
< {pac_name: '', class_name_list: [], func_name_list: [], import_pac_name_list: []]}
< {pac_name: '', class_name_list: [], func_name_dict: {}, import_pac_name_list: []]}
'''
import_pac_name_list = []
@ -57,25 +55,26 @@ class JavaParser:
import_pac_name = import_pac.path
import_pac_name_list.append(import_pac_name)
pac_name = tree.package.name
fp_last = fp.split(os.path.sep)[-1]
pac_name = tree.package.name + '#' + fp_last
class_name_list = []
func_name_dict = {}
for node in tree.types:
if type(node) in (javalang.tree.ClassDeclaration, javalang.tree.InterfaceDeclaration):
class_name = pac_name + '.' + node.name
class_name = pac_name + '#' + node.name
class_name_list.append(class_name)
for node_inner in node.body:
if type(node_inner) is javalang.tree.MethodDeclaration:
func_name = class_name + '.' + node_inner.name
func_name = class_name + '#' + node_inner.name
# add params name to func_name
params_list = node_inner.parameters
for params in params_list:
params_name = params.type.name
func_name = func_name + '_' + params_name
func_name = func_name + '-' + params_name
if class_name not in func_name_dict:
func_name_dict[class_name] = []
@ -97,11 +96,21 @@ class JavaParser:
< parse_result_dict
'''
res_dict = {}
for java_code, tree in tree_dict.items():
for fp, value in tree_dict.items():
java_code = value['code']
tree = value['tree']
try:
res_dict[java_code] = self.single_java_code_parse(tree)
res_dict[java_code] = self.single_java_code_parse(tree, fp)
except Exception as e:
logger.debug(java_code)
raise ImportError
return res_dict
return res_dict

15
dev_opsgpt/codechat/code_crawler/__init__.py Normal file
View File

@ -0,0 +1,15 @@
# encoding: utf-8
'''
@author: 温进
@file: __init__.py.py
@time: 2023/11/21 下午2:02
@desc:
'''
from .zip_crawler import ZipCrawler
from .dir_crawler import DirCrawler
__all__ = [
'ZipCrawler',
'DirCrawler'
]

39
dev_opsgpt/codechat/code_crawler/dir_crawler.py Normal file
View File

@ -0,0 +1,39 @@
# encoding: utf-8
'''
@author: 温进
@file: dir_crawler.py
@time: 2023/11/22 下午2:54
@desc:
'''
from loguru import logger
import os
import glob
class DirCrawler:
@staticmethod
def crawl(path: str, suffix: str):
'''
read local java file in path
> path: path to crawl, must be absolute path like A/B/C
< dict of java code string
'''
java_file_list = glob.glob('{path}{sep}**{sep}*.{suffix}'.format(path=path, sep=os.path.sep, suffix=suffix),
recursive=True)
java_code_dict = {}
logger.info(path)
logger.info('number of file={}'.format(len(java_file_list)))
logger.info(java_file_list)
for java_file in java_file_list:
with open(java_file) as f:
java_code = ''.join(f.readlines())
java_code_dict[java_file] = java_code
return java_code_dict
if __name__ == '__main__':
path = '/Users/bingxu/Desktop/工作/大模型/chatbot/test_code_repo/middleware-alipay-starters-parent'
suffix = 'java'
DirCrawler.crawl(path, suffix)

31
dev_opsgpt/codechat/code_crawler/zip_crawler.py Normal file
View File

@ -0,0 +1,31 @@
# encoding: utf-8
'''
@author: 温进
@file: zip_crawler.py
@time: 2023/11/21 下午2:02
@desc:
'''
from loguru import logger
import zipfile
from dev_opsgpt.codechat.code_crawler.dir_crawler import DirCrawler
class ZipCrawler:
@staticmethod
def crawl(zip_file, output_path, suffix):
'''
unzip to output_path
@param zip_file:
@param output_path:
@return:
'''
logger.info(f'output_path={output_path}')
print(f'output_path={output_path}')
with zipfile.ZipFile(zip_file, 'r') as z:
z.extractall(output_path)
code_dict = DirCrawler.crawl(output_path, suffix)
return code_dict

2
dev_opsgpt/codebase_handler/__init__.py → dev_opsgpt/codechat/code_search/__init__.py
View File

@ -2,6 +2,6 @@
'''
@author: 温进
@file: __init__.py.py
@time: 2023/10/23 下午4:57
@time: 2023/11/21 下午2:35
@desc:
'''

179
dev_opsgpt/codechat/code_search/code_search.py Normal file
View File

@ -0,0 +1,179 @@
# encoding: utf-8
'''
@author: 温进
@file: code_search.py
@time: 2023/11/21 下午2:35
@desc:
'''
import time
from loguru import logger
from collections import defaultdict
from dev_opsgpt.db_handler.graph_db_handler.nebula_handler import NebulaHandler
from dev_opsgpt.db_handler.vector_db_handler.chroma_handler import ChromaHandler
from dev_opsgpt.codechat.code_search.cypher_generator import CypherGenerator
from dev_opsgpt.codechat.code_search.tagger import Tagger
from dev_opsgpt.embeddings.get_embedding import get_embedding
# search_by_tag
VERTEX_SCORE = 10
HISTORY_VERTEX_SCORE = 5
VERTEX_MERGE_RATIO = 0.5
# search_by_description
MAX_DISTANCE = 0.5
class CodeSearch:
def __init__(self, nh: NebulaHandler, ch: ChromaHandler, limit: int = 3):
'''
init
@param nh: NebulaHandler
@param ch: ChromaHandler
@param limit: limit of result
'''
self.nh = nh
self.ch = ch
self.limit = limit
def search_by_tag(self, query: str):
'''
search_code_res by tag
@param query: str
@return:
'''
tagger = Tagger()
tag_list = tagger.generate_tag_query(query)
logger.info(f'query tag={tag_list}')
# get all verticex
vertex_list = self.nh.get_vertices().get('v', [])
vertex_vid_list = [i.as_node().get_id().as_string() for i in vertex_list]
logger.debug(vertex_vid_list)
# update score
vertex_score_dict = defaultdict(lambda: 0)
for vid in vertex_vid_list:
for tag in tag_list:
if tag in vid:
vertex_score_dict[vid] += VERTEX_SCORE
# merge depend adj score
vertex_score_dict_final = {}
for vertex in vertex_score_dict:
cypher = f'''MATCH (v1)-[e]-(v2) where id(v1) == "{vertex}" RETURN v2'''
cypher_res = self.nh.execute_cypher(cypher, self.nh.space_name)
cypher_res_dict = self.nh.result_to_dict(cypher_res)
adj_vertex_list = [i.as_node().get_id().as_string() for i in cypher_res_dict.get('v2', [])]
score = vertex_score_dict.get(vertex, 0)
for adj_vertex in adj_vertex_list:
score += vertex_score_dict.get(adj_vertex, 0) * VERTEX_MERGE_RATIO
if score > 0:
vertex_score_dict_final[vertex] = score
# get most prominent package tag
package_score_dict = defaultdict(lambda: 0)
for vertex, score in vertex_score_dict.items():
package = '#'.join(vertex.split('#')[0:2])
package_score_dict[package] += score
# get respective code
res = []
package_score_tuple = list(package_score_dict.items())
package_score_tuple.sort(key=lambda x: x[1], reverse=True)
ids = [i[0] for i in package_score_tuple]
chroma_res = self.ch.get(ids=ids, include=['metadatas'])
for vertex, score in package_score_tuple:
index = chroma_res['result']['ids'].index(vertex)
code_text = chroma_res['result']['metadatas'][index]['code_text']
res.append({
"vertex": vertex,
"code_text": code_text}
)
if len(res) >= self.limit:
break
return res
def search_by_desciption(self, query: str, engine: str):
'''
search by perform sim search
@param query:
@return:
'''
query = query.replace(',', '，')
query_emb = get_embedding(engine=engine, text_list=[query])
query_emb = query_emb[query]
query_embeddings = [query_emb]
query_result = self.ch.query(query_embeddings=query_embeddings, n_results=self.limit,
include=['metadatas', 'distances'])
logger.debug(query_result)
res = []
for idx, distance in enumerate(query_result['result']['distances'][0]):
if distance < MAX_DISTANCE:
vertex = query_result['result']['ids'][0][idx]
code_text = query_result['result']['metadatas'][0][idx]['code_text']
res.append({
"vertex": vertex,
"code_text": code_text
})
return res
def search_by_cypher(self, query: str):
'''
search by generating cypher
@param query:
@param engine:
@return:
'''
cg = CypherGenerator()
cypher = cg.get_cypher(query)
if not cypher:
return None
cypher_res = self.nh.execute_cypher(cypher, self.nh.space_name)
logger.info(f'cypher execution result={cypher_res}')
if not cypher_res.is_succeeded():
return {
'cypher': '',
'cypher_res': ''
}
res = {
'cypher': cypher,
'cypher_res': cypher_res
}
return res
if __name__ == '__main__':
from configs.server_config import NEBULA_HOST, NEBULA_PORT, NEBULA_USER, NEBULA_PASSWORD, NEBULA_STORAGED_PORT
from configs.server_config import CHROMA_PERSISTENT_PATH
codebase_name = 'testing'
nh = NebulaHandler(host=NEBULA_HOST, port=NEBULA_PORT, username=NEBULA_USER,
password=NEBULA_PASSWORD, space_name=codebase_name)
nh.add_host(NEBULA_HOST, NEBULA_STORAGED_PORT)
time.sleep(0.5)
ch = ChromaHandler(path=CHROMA_PERSISTENT_PATH, collection_name=codebase_name)
cs = CodeSearch(nh, ch)
# res = cs.search_by_tag(tag_list=['createFineTuneCompletion', 'OpenAiApi'])
# logger.debug(res)
# res = cs.search_by_cypher('代码中一共有多少个类', 'openai')
# logger.debug(res)
res = cs.search_by_desciption('使用不同的HTTP请求类型（GET、POST、DELETE等）来执行不同的操作', 'openai')
logger.debug(res)

63
dev_opsgpt/codechat/code_search/cypher_generator.py Normal file
View File

@ -0,0 +1,63 @@
# encoding: utf-8
'''
@author: 温进
@file: cypher_generator.py
@time: 2023/11/24 上午10:17
@desc:
'''
from loguru import logger
from dev_opsgpt.llm_models.openai_model import getChatModel
from dev_opsgpt.utils.postprocess import replace_lt_gt
from langchain.schema import (
HumanMessage,
)
from langchain.chains.graph_qa.prompts import NGQL_GENERATION_PROMPT
schema = '''
Node properties: [{'tag': 'package', 'properties': []}, {'tag': 'class', 'properties': []}, {'tag': 'method', 'properties': []}]
Edge properties: [{'edge': 'contain', 'properties': []}, {'edge': 'depend', 'properties': []}]
Relationships: ['(:package)-[:contain]->(:class)', '(:class)-[:contain]->(:method)', '(:package)-[:contain]->(:package)']
'''
class CypherGenerator:
def __init__(self):
self.model = getChatModel()
def get_cypher(self, query: str):
'''
get cypher from query
@param query:
@return:
'''
content = NGQL_GENERATION_PROMPT.format(schema=schema, question=query)
ans = ''
message = [HumanMessage(content=content)]
chat_res = self.model.predict_messages(message)
ans = chat_res.content
ans = replace_lt_gt(ans)
ans = self.post_process(ans)
return ans
def post_process(self, cypher_res: str):
'''
判断是否为正确的 cypher
@param cypher_res:
@return:
'''
if '(' not in cypher_res or ')' not in cypher_res:
return ''
return cypher_res
if __name__ == '__main__':
query = '代码中一共有多少个类'
cg = CypherGenerator(engine='openai')
cg.get_cypher(query)

23
dev_opsgpt/codechat/code_search/tagger.py Normal file
View File

@ -0,0 +1,23 @@
# encoding: utf-8
'''
@author: 温进
@file: tagger.py
@time: 2023/11/24 下午1:32
@desc:
'''
import re
from loguru import logger
class Tagger:
def __init__(self):
pass
def generate_tag_query(self, query):
'''
generate tag from query
'''
# simple extract english
tag_list = re.findall(r'[a-zA-Z\_\.]+', query)
tag_list = list(set(tag_list))
return tag_list

2
dev_opsgpt/codebase_handler/parser/__init__.py → dev_opsgpt/codechat/codebase_handler/__init__.py
View File

@ -2,6 +2,6 @@
'''
@author: 温进
@file: __init__.py.py
@time: 2023/10/23 下午5:00
@time: 2023/11/21 下午2:07
@desc:
'''

175
dev_opsgpt/codechat/codebase_handler/code_importer.py Normal file
View File

File diff suppressed because one or more lines are too long

169
dev_opsgpt/codechat/codebase_handler/codebase_handler.py Normal file
View File

@ -0,0 +1,169 @@
# encoding: utf-8
'''
@author: 温进
@file: codebase_handler.py
@time: 2023/11/21 下午2:25
@desc:
'''
import time
from loguru import logger
from configs.server_config import NEBULA_HOST, NEBULA_PORT, NEBULA_USER, NEBULA_PASSWORD, NEBULA_STORAGED_PORT
from configs.server_config import CHROMA_PERSISTENT_PATH
from configs.model_config import EMBEDDING_ENGINE
from dev_opsgpt.db_handler.graph_db_handler.nebula_handler import NebulaHandler
from dev_opsgpt.db_handler.vector_db_handler.chroma_handler import ChromaHandler
from dev_opsgpt.codechat.code_crawler.zip_crawler import *
from dev_opsgpt.codechat.code_analyzer.code_analyzer import CodeAnalyzer
from dev_opsgpt.codechat.codebase_handler.code_importer import CodeImporter
from dev_opsgpt.codechat.code_search.code_search import CodeSearch
class CodeBaseHandler:
def __init__(self, codebase_name: str, code_path: str = '',
language: str = 'java', crawl_type: str = 'ZIP'):
self.codebase_name = codebase_name
self.code_path = code_path
self.language = language
self.crawl_type = crawl_type
self.nh = NebulaHandler(host=NEBULA_HOST, port=NEBULA_PORT, username=NEBULA_USER,
password=NEBULA_PASSWORD, space_name=codebase_name)
self.nh.add_host(NEBULA_HOST, NEBULA_STORAGED_PORT)
time.sleep(1)
self.ch = ChromaHandler(path=CHROMA_PERSISTENT_PATH, collection_name=codebase_name)
def import_code(self, zip_file='', do_interpret=True):
'''
analyze code and save it to codekg and codedb
@return:
'''
# init graph to init tag and edge
code_importer = CodeImporter(engine=EMBEDDING_ENGINE, codebase_name=self.codebase_name,
nh=self.nh, ch=self.ch)
code_importer.init_graph()
time.sleep(5)
# crawl code
st0 = time.time()
logger.info('start crawl')
code_dict = self.crawl_code(zip_file)
logger.debug('crawl done, rt={}'.format(time.time() - st0))
# analyze code
logger.info('start analyze')
st1 = time.time()
code_analyzer = CodeAnalyzer(language=self.language)
static_analysis_res, interpretation = code_analyzer.analyze(code_dict, do_interpret=do_interpret)
logger.debug('analyze done, rt={}'.format(time.time() - st1))
# add info to nebula and chroma
st2 = time.time()
code_importer.import_code(static_analysis_res, interpretation, do_interpret=do_interpret)
logger.debug('update codebase done, rt={}'.format(time.time() - st2))
# get KG info
stat = self.nh.get_stat()
vertices_num, edges_num = stat['vertices'], stat['edges']
# get chroma info
file_num = self.ch.count()['result']
return vertices_num, edges_num, file_num
def delete_codebase(self, codebase_name: str):
'''
delete codebase
@param codebase_name: name of codebase
@return:
'''
self.nh.drop_space(space_name=codebase_name)
self.ch.delete_collection(collection_name=codebase_name)
def crawl_code(self, zip_file=''):
'''
@return:
'''
if self.language == 'java':
suffix = 'java'
logger.info(f'crawl_type={self.crawl_type}')
code_dict = {}
if self.crawl_type.lower() == 'zip':
code_dict = ZipCrawler.crawl(zip_file, output_path=self.code_path, suffix=suffix)
elif self.crawl_type.lower() == 'dir':
code_dict = DirCrawler.crawl(self.code_path, suffix)
return code_dict
def search_code(self, query: str, search_type: str, limit: int = 3):
'''
search code from codebase
@param limit:
@param engine:
@param query: query from user
@param search_type: ['cypher', 'graph', 'vector']
@return:
'''
assert search_type in ['cypher', 'tag', 'description']
code_search = CodeSearch(nh=self.nh, ch=self.ch, limit=limit)
if search_type == 'cypher':
search_res = code_search.search_by_cypher(query=query)
elif search_type == 'tag':
search_res = code_search.search_by_tag(query=query)
elif search_type == 'description':
search_res = code_search.search_by_desciption(query=query, engine=EMBEDDING_ENGINE)
context, related_vertice = self.format_search_res(search_res, search_type)
return context, related_vertice
def format_search_res(self, search_res: str, search_type: str):
'''
format search_res
@param search_res:
@param search_type:
@return:
'''
CYPHER_QA_PROMPT = '''
执行的 Cypher 是: {cypher}
Cypher 的结果是: {result}
'''
if search_type == 'cypher':
context = CYPHER_QA_PROMPT.format(cypher=search_res['cypher'], result=search_res['cypher_res'])
related_vertice = []
elif search_type == 'tag':
context = ''
related_vertice = []
for code in search_res:
context = context + code['code_text'] + '\n'
related_vertice.append(code['vertex'])
elif search_type == 'description':
context = ''
related_vertice = []
for code in search_res:
context = context + code['code_text'] + '\n'
related_vertice.append(code['vertex'])
return context, related_vertice
if __name__ == '__main__':
codebase_name = 'testing'
code_path = '/Users/bingxu/Desktop/工作/大模型/chatbot/test_code_repo/client'
cbh = CodeBaseHandler(codebase_name, code_path, crawl_type='dir')
# query = '使用不同的HTTP请求类型（GET、POST、DELETE等）来执行不同的操作'
# query = '代码中一共有多少个类'
query = 'intercept 函数作用是什么'
search_type = 'graph'
limit = 2
res = cbh.search_code(query, search_type, limit)
logger.debug(res)

5
dev_opsgpt/connector/agents/__init__.py
View File

@ -1,6 +1,9 @@
from .base_agent import BaseAgent
from .react_agent import ReactAgent
from .check_agent import CheckAgent
from .executor_agent import ExecutorAgent
from .selector_agent import SelectorAgent
__all__ = [
"BaseAgent", "ReactAgent"
"BaseAgent", "ReactAgent", "CheckAgent", "ExecutorAgent", "SelectorAgent"
]

732
dev_opsgpt/connector/agents/base_agent.py
View File

@ -7,16 +7,19 @@ import traceback
import uuid
from loguru import logger
from dev_opsgpt.connector.shcema.memory import Memory
from dev_opsgpt.connector.connector_schema import (
Task, Role, Message, ActionStatus, Doc, CodeDoc
)
from dev_opsgpt.connector.schema import (
Memory, Task, Env, Role, Message, ActionStatus, CodeDoc, Doc
)
from configs.server_config import SANDBOX_SERVER
from dev_opsgpt.sandbox import PyCodeBox, CodeBoxResponse
from dev_opsgpt.tools import DDGSTool, DocRetrieval, CodeRetrieval
from dev_opsgpt.connector.configs.agent_config import REACT_PROMPT_INPUT
from dev_opsgpt.connector.configs.prompts import BASE_PROMPT_INPUT, QUERY_CONTEXT_DOC_PROMPT_INPUT, BEGIN_PROMPT_INPUT
from dev_opsgpt.connector.message_process import MessageUtils
from dev_opsgpt.connector.configs.agent_config import REACT_PROMPT_INPUT, QUERY_CONTEXT_PROMPT_INPUT, PLAN_PROMPT_INPUT
from dev_opsgpt.llm_models import getChatModel
from dev_opsgpt.connector.utils import parse_section
class BaseAgent:
@ -34,67 +37,74 @@ class BaseAgent:
stop: Union[List[str], str] = None,
do_filter: bool = True,
do_use_self_memory: bool = True,
# docs_prompt: str,
focus_agents: List[str] = [],
focus_message_keys: List[str] = [],
# prompt_mamnger: PromptManager
):
self.task = task
self.role = role
self.message_utils = MessageUtils()
self.llm = self.create_llm_engine(temperature, stop)
self.memory = self.init_history(memory)
self.chat_turn = chat_turn
self.do_search = do_search
self.do_doc_retrieval = do_doc_retrieval
self.do_tool_retrieval = do_tool_retrieval
self.codebox = PyCodeBox(
remote_url=SANDBOX_SERVER["url"],
remote_ip=SANDBOX_SERVER["host"],
remote_port=SANDBOX_SERVER["port"],
token="mytoken",
do_code_exe=True,
do_remote=SANDBOX_SERVER["do_remote"],
do_check_net=False
)
self.focus_agents = focus_agents
self.focus_message_keys = focus_message_keys
self.do_filter = do_filter
self.do_use_self_memory = do_use_self_memory
# self.docs_prompt = docs_prompt
# self.prompt_manager = None
def run(self, query: Message, history: Memory = None, background: Memory = None) -> Message:
'''llm inference'''
def run(self, query: Message, history: Memory = None, background: Memory = None, memory_pool: Memory=None) -> Message:
'''agent reponse from multi-message'''
message = None
for message in self.arun(query, history, background, memory_pool):
pass
return message
def arun(self, query: Message, history: Memory = None, background: Memory = None, memory_pool: Memory=None) -> Message:
'''agent reponse from multi-message'''
# insert query into memory
query_c = copy.deepcopy(query)
self_memory = self.memory if self.do_use_self_memory else None
prompt = self.create_prompt(query_c, self_memory, history, background)
# create your llm prompt
prompt = self.create_prompt(query_c, self_memory, history, background, memory_pool=memory_pool)
content = self.llm.predict(prompt)
logger.debug(f"{self.role.role_name} prompt: {prompt}")
# logger.debug(f"{self.role.role_name} content: {content}")
logger.debug(f"{self.role.role_name} content: {content}")
output_message = Message(
role_name=self.role.role_name,
role_type="ai", #self.role.role_type,
role_content=content,
role_contents=[content],
step_content=content,
input_query=query_c.input_query,
tools=query_c.tools
tools=query_c.tools,
parsed_output_list=[query.parsed_output]
)
output_message = self.parser(output_message)
# common parse llm' content to message
output_message = self.message_utils.parser(output_message)
if self.do_filter:
output_message = self.filter(output_message)
output_message = self.message_utils.filter(output_message)
# 更新自身的回答
# update self_memory
self.append_history(query_c)
self.append_history(output_message)
logger.info(f"{self.role.role_name} step_run: {output_message.role_content}")
return output_message
# logger.info(f"{self.role.role_name} currenct question: {output_message.input_query}\nllm_step_run: {output_message.role_content}")
output_message.input_query = output_message.role_content
output_message.parsed_output_list.append(output_message.parsed_output)
# update memory pool
memory_pool.append(output_message)
yield output_message
def create_prompt(
self, query: Message, memory: Memory =None, history: Memory = None, background: Memory = None, prompt_mamnger=None) -> str:
self, query: Message, memory: Memory =None, history: Memory = None, background: Memory = None, memory_pool: Memory=None, prompt_mamnger=None) -> str:
'''
role\task\tools\docs\memory
prompt engineer, contains role\task\tools\docs\memory
'''
#
doc_infos = self.create_doc_prompt(query)
@ -105,50 +115,105 @@ class BaseAgent:
background_prompt = self.create_background_prompt(background, control_key="step_content")
history_prompt = self.create_history_prompt(history)
selfmemory_prompt = self.create_selfmemory_prompt(memory, control_key="step_content")
#
# extra_system_prompt = self.role.role_prompt
prompt = self.role.role_prompt.format(**{"formatted_tools": formatted_tools, "tool_names": tool_names})
#
memory_pool_select_by_agent_key = self.select_memory_by_agent_key(memory_pool)
memory_pool_select_by_agent_key_context = '\n\n'.join([f"*{k}*\n{v}" for parsed_output in memory_pool_select_by_agent_key.get_parserd_output_list() for k, v in parsed_output.items() if k not in ['Action Status']])
# input_query = query.input_query
# # logger.debug(f"{self.role.role_name} extra_system_prompt: {self.role.role_prompt}")
# # logger.debug(f"{self.role.role_name} input_query: {input_query}")
# # logger.debug(f"{self.role.role_name} doc_infos: {doc_infos}")
# # logger.debug(f"{self.role.role_name} tool_names: {tool_names}")
# if "**Context:**" in self.role.role_prompt:
# # logger.debug(f"parsed_output_list: {query.parsed_output_list}")
# # input_query = "'''" + "\n".join([f"###{k}###\n{v}" for i in query.parsed_output_list for k,v in i.items() if "Action Status" !=k]) + "'''"
# context = "\n".join([f"*{k}*\n{v}" for i in query.parsed_output_list for k,v in i.items() if "Action Status" !=k])
# # context = history_prompt or '""'
# # logger.debug(f"parsed_output_list: {t}")
# prompt += "\n" + QUERY_CONTEXT_PROMPT_INPUT.format(**{"context": context, "query": query.origin_query})
# else:
# prompt += "\n" + PLAN_PROMPT_INPUT.format(**{"query": input_query})
task = query.task or self.task
if task_prompt is not None:
prompt += "\n" + task.task_prompt
DocInfos = ""
if doc_infos is not None and doc_infos!="" and doc_infos!="不存在知识库辅助信息":
prompt += f"\n知识库信息: {doc_infos}"
DocInfos += f"\nDocument Information: {doc_infos}"
if code_infos is not None and code_infos!="" and code_infos!="不存在代码库辅助信息":
prompt += f"\n代码库信息: {code_infos}"
DocInfos += f"\nCodeBase Infomation: {code_infos}"
# if selfmemory_prompt:
# prompt += "\n" + selfmemory_prompt
if background_prompt:
prompt += "\n" + background_prompt
# if background_prompt:
# prompt += "\n" + background_prompt
if history_prompt:
prompt += "\n" + history_prompt
# if history_prompt:
# prompt += "\n" + history_prompt
if selfmemory_prompt:
prompt += "\n" + selfmemory_prompt
# input_query = memory.to_tuple_messages(content_key="step_content")
# input_query = "\n".join([f"{k}: {v}" for k, v in input_query if v])
input_query = query.role_content
input_query = query.input_query
# logger.debug(f"{self.role.role_name} extra_system_prompt: {self.role.role_prompt}")
logger.debug(f"{self.role.role_name} input_query: {input_query}")
# logger.debug(f"{self.role.role_name} input_query: {input_query}")
# logger.debug(f"{self.role.role_name} doc_infos: {doc_infos}")
logger.debug(f"{self.role.role_name} tool_names: {tool_names}")
prompt += "\n" + REACT_PROMPT_INPUT.format(**{"query": input_query})
# logger.debug(f"{self.role.role_name} tool_names: {tool_names}")
# extra_system_prompt = self.role.role_prompt
input_keys = parse_section(self.role.role_prompt, 'Input Format')
prompt = self.role.role_prompt.format(**{"formatted_tools": formatted_tools, "tool_names": tool_names})
prompt += "\n" + BEGIN_PROMPT_INPUT
for input_key in input_keys:
if input_key == "Origin Query":
prompt += "\n**Origin Query:**\n" + query.origin_query
elif input_key == "Context":
context = "\n".join([f"*{k}*\n{v}" for i in query.parsed_output_list for k,v in i.items() if "Action Status" !=k])
if history:
context = history_prompt + "\n" + context
if not context:
context = "there is no context"
if self.focus_agents and memory_pool_select_by_agent_key_context:
context = memory_pool_select_by_agent_key_context
prompt += "\n**Context:**\n" + context + "\n" + input_query
elif input_key == "DocInfos":
prompt += "\n**DocInfos:**\n" + DocInfos
elif input_key == "Question":
prompt += "\n**Question:**\n" + input_query
# if "**Context:**" in self.role.role_prompt:
# # logger.debug(f"parsed_output_list: {query.parsed_output_list}")
# # input_query = "'''" + "\n".join([f"###{k}###\n{v}" for i in query.parsed_output_list for k,v in i.items() if "Action Status" !=k]) + "'''"
# context = "\n".join([f"*{k}*\n{v}" for i in query.parsed_output_list for k,v in i.items() if "Action Status" !=k])
# if history:
# context = history_prompt + "\n" + context
# if not context:
# context = "there is no context"
# # logger.debug(f"parsed_output_list: {t}")
# if "DocInfos" in prompt:
# prompt += "\n" + QUERY_CONTEXT_DOC_PROMPT_INPUT.format(**{"context": context, "query": query.origin_query, "DocInfos": DocInfos})
# else:
# prompt += "\n" + QUERY_CONTEXT_PROMPT_INPUT.format(**{"context": context, "query": query.origin_query, "DocInfos": DocInfos})
# else:
# prompt += "\n" + BASE_PROMPT_INPUT.format(**{"query": input_query})
# prompt = extra_system_prompt.format(**{"query": input_query, "doc_infos": doc_infos, "formatted_tools": formatted_tools, "tool_names": tool_names})
while "{{" in prompt or "}}" in prompt:
prompt = prompt.replace("{{", "{")
prompt = prompt.replace("}}", "}")
# logger.debug(f"{self.role.role_name} prompt: {prompt}")
return prompt
# prompt_comp = [("system", extra_system_prompt)] + memory.to_tuple_messages()
# prompt = ChatPromptTemplate.from_messages(prompt_comp)
# prompt = prompt.format(**{"query": query.role_content, "doc_infos": doc_infos, "formatted_tools": formatted_tools, "tool_names": tool_names})
# return prompt
def create_doc_prompt(self, message: Message) -> str:
''''''
@ -197,7 +262,7 @@ class BaseAgent:
return "\n补充自身对话信息: " + selfmemory_message if selfmemory_message else None
def init_history(self, memory: Memory = None) -> Memory:
return Memory([])
return Memory(messages=[])
def update_history(self, message: Message):
self.memory.append(message)
@ -212,216 +277,389 @@ class BaseAgent:
def create_llm_engine(self, temperature=0.2, stop=None):
return getChatModel(temperature=temperature, stop=stop)
def filter(self, message: Message, stop=None) -> Message:
# def filter(self, message: Message, stop=None) -> Message:
tool_params = self.parser_spec_key(message.role_content, "tool_params")
code_content = self.parser_spec_key(message.role_content, "code_content")
plan = self.parser_spec_key(message.role_content, "plan")
plans = self.parser_spec_key(message.role_content, "plans", do_search=False)
content = self.parser_spec_key(message.role_content, "content", do_search=False)
# tool_params = self.parser_spec_key(message.role_content, "tool_params")
# code_content = self.parser_spec_key(message.role_content, "code_content")
# plan = self.parser_spec_key(message.role_content, "plan")
# plans = self.parser_spec_key(message.role_content, "plans", do_search=False)
# content = self.parser_spec_key(message.role_content, "content", do_search=False)
# logger.debug(f"tool_params: {tool_params}, code_content: {code_content}, plan: {plan}, plans: {plans}, content: {content}")
role_content = tool_params or code_content or plan or plans or content
message.role_content = role_content or message.role_content
return message
# # logger.debug(f"tool_params: {tool_params}, code_content: {code_content}, plan: {plan}, plans: {plans}, content: {content}")
# role_content = tool_params or code_content or plan or plans or content
# message.role_content = role_content or message.role_content
# return message
def token_usage(self, ):
'''calculate the usage of token'''
pass
def get_extra_infos(self, message: Message) -> Message:
''''''
if self.do_search:
message = self.get_search_retrieval(message)
def select_memory_by_key(self, memory: Memory) -> Memory:
return Memory(
messages=[self.select_message_by_key(message) for message in memory.messages
if self.select_message_by_key(message) is not None]
)
def select_memory_by_agent_key(self, memory: Memory) -> Memory:
return Memory(
messages=[self.select_message_by_agent_key(message) for message in memory.messages
if self.select_message_by_agent_key(message) is not None]
)
def select_message_by_agent_key(self, message: Message) -> Message:
# assume we focus all agents
if self.focus_agents == []:
return message
return None if message is None or message.role_name not in self.focus_agents else self.select_message_by_key(message)
def select_message_by_key(self, message: Message) -> Message:
# assume we focus all key contents
if message is None:
return message
if self.do_doc_retrieval:
message = self.get_doc_retrieval(message)
if self.do_tool_retrieval:
message = self.get_tool_retrieval(message)
if self.focus_message_keys == []:
return message
return message
def get_search_retrieval(self, message: Message,) -> Message:
SEARCH_ENGINES = {"duckduckgo": DDGSTool}
search_docs = []
for idx, doc in enumerate(SEARCH_ENGINES["duckduckgo"].run(message.role_content, 3)):
doc.update({"index": idx})
search_docs.append(Doc(**doc))
message.search_docs = search_docs
return message
def get_doc_retrieval(self, message: Message) -> Message:
query = message.role_content
knowledge_basename = message.doc_engine_name
top_k = message.top_k
score_threshold = message.score_threshold
if knowledge_basename:
docs = DocRetrieval.run(query, knowledge_basename, top_k, score_threshold)
message.db_docs = [Doc(**doc) for doc in docs]
return message
def get_code_retrieval(self, message: Message) -> Message:
# DocRetrieval.run("langchain是什么", "DSADSAD")
query = message.input_query
code_engine_name = message.code_engine_name
history_node_list = message.history_node_list
code_docs = CodeRetrieval.run(code_engine_name, query, code_limit=message.top_k, history_node_list=history_node_list)
message.code_docs = [CodeDoc(**doc) for doc in code_docs]
return message
def get_tool_retrieval(self, message: Message) -> Message:
return message
def step_router(self, message: Message) -> Message:
''''''
# message = self.parser(message)
# logger.debug(f"message.action_status: {message.action_status}")
if message.action_status == ActionStatus.CODING:
message = self.code_step(message)
elif message.action_status == ActionStatus.TOOL_USING:
message = self.tool_step(message)
message_c = copy.deepcopy(message)
message_c.parsed_output = {k: v for k,v in message_c.parsed_output.items() if k in self.focus_message_keys}
message_c.parsed_output_list = [{k: v for k,v in parsed_output.items() if k in self.focus_message_keys} for parsed_output in message_c.parsed_output_list]
return message_c
return message
def code_step(self, message: Message) -> Message:
'''execute code'''
# logger.debug(f"message.role_content: {message.role_content}, message.code_content: {message.code_content}")
code_answer = self.codebox.chat('```python\n{}```'.format(message.code_content))
code_prompt = f"执行上述代码后存在报错信息为 {code_answer.code_exe_response}，需要进行修复" \
if code_answer.code_exe_type == "error" else f"执行上述代码后返回信息为 {code_answer.code_exe_response}"
uid = str(uuid.uuid1())
if code_answer.code_exe_type == "image/png":
message.figures[uid] = code_answer.code_exe_response
message.code_answer = f"\n观察: 执行上述代码后生成一张图片, 图片名为{uid}\n"
message.observation = f"\n观察: 执行上述代码后生成一张图片, 图片名为{uid}\n"
message.step_content += f"\n观察: 执行上述代码后生成一张图片, 图片名为{uid}\n"
message.step_contents += [f"\n观察: 执行上述代码后生成一张图片, 图片名为{uid}\n"]
message.role_content += f"\n观察:执行上述代码后生成一张图片, 图片名为{uid}\n"
else:
message.code_answer = code_answer.code_exe_response
message.observation = code_answer.code_exe_response
message.step_content += f"\n观察: {code_prompt}\n"
message.step_contents += [f"\n观察: {code_prompt}\n"]
message.role_content += f"\n观察: {code_prompt}\n"
# logger.info(f"观察: {message.action_status}, {message.observation}")
return message
def tool_step(self, message: Message) -> Message:
'''execute tool'''
# logger.debug(f"message: {message.action_status}, {message.tool_name}, {message.tool_params}")
tool_names = [tool.name for tool in message.tools]
if message.tool_name not in tool_names:
message.tool_answer = "不存在可以执行的tool"
message.observation = "不存在可以执行的tool"
message.role_content += f"\n观察: 不存在可以执行的tool\n"
message.step_content += f"\n观察: 不存在可以执行的tool\n"
message.step_contents += [f"\n观察: 不存在可以执行的tool\n"]
for tool in message.tools:
if tool.name == message.tool_name:
tool_res = tool.func(**message.tool_params)
message.tool_answer = tool_res
message.observation = tool_res
message.role_content += f"\n观察: {tool_res}\n"
message.step_content += f"\n观察: {tool_res}\n"
message.step_contents += [f"\n观察: {tool_res}\n"]
# logger.info(f"观察: {message.action_status}, {message.observation}")
return message
def parser(self, message: Message) -> Message:
''''''
content = message.role_content
parser_keys = ["action", "code_content", "code_filename", "tool_params", "plans"]
try:
s_json = self._parse_json(content)
message.action_status = s_json.get("action")
message.code_content = s_json.get("code_content")
message.tool_params = s_json.get("tool_params")
message.tool_name = s_json.get("tool_name")
message.code_filename = s_json.get("code_filename")
message.plans = s_json.get("plans")
# for parser_key in parser_keys:
# message.action_status = content.get(parser_key)
except Exception as e:
# logger.warning(f"{traceback.format_exc()}")
action_value = self._match(r"'action':\s*'([^']*)'", content) if "'action'" in content else self._match(r'"action":\s*"([^"]*)"', content)
code_content_value = self._match(r"'code_content':\s*'([^']*)'", content) if "'code_content'" in content else self._match(r'"code_content":\s*"([^"]*)"', content)
filename_value = self._match(r"'code_filename':\s*'([^']*)'", content) if "'code_filename'" in content else self._match(r'"code_filename":\s*"([^"]*)"', content)
tool_params_value = self._match(r"'tool_params':\s*(\{[^{}]*\})", content, do_json=True) if "'tool_params'" in content \
else self._match(r'"tool_params":\s*(\{[^{}]*\})', content, do_json=True)
tool_name_value = self._match(r"'tool_name':\s*'([^']*)'", content) if "'tool_name'" in content else self._match(r'"tool_name":\s*"([^"]*)"', content)
plans_value = self._match(r"'plans':\s*(\[.*?\])", content, do_search=False) if "'plans'" in content else self._match(r'"plans":\s*(\[.*?\])', content, do_search=False, )
# re解析
message.action_status = action_value or "default"
message.code_content = code_content_value
message.code_filename = filename_value
message.tool_params = tool_params_value
message.tool_name = tool_name_value
message.plans = plans_value
# logger.debug(f"确认当前的action: {message.action_status}")
return message
def parser_spec_key(self, content, key, do_search=True, do_json=False) -> str:
''''''
key2pattern = {
"'action'": r"'action':\s*'([^']*)'", '"action"': r'"action":\s*"([^"]*)"',
"'code_content'": r"'code_content':\s*'([^']*)'", '"code_content"': r'"code_content":\s*"([^"]*)"',
"'code_filename'": r"'code_filename':\s*'([^']*)'", '"code_filename"': r'"code_filename":\s*"([^"]*)"',
"'tool_params'": r"'tool_params':\s*(\{[^{}]*\})", '"tool_params"': r'"tool_params":\s*(\{[^{}]*\})',
"'tool_name'": r"'tool_name':\s*'([^']*)'", '"tool_name"': r'"tool_name":\s*"([^"]*)"',
"'plans'": r"'plans':\s*(\[.*?\])", '"plans"': r'"plans":\s*(\[.*?\])',
"'content'": r"'content':\s*'([^']*)'", '"content"': r'"content":\s*"([^"]*)"',
}
# def get_extra_infos(self, message: Message) -> Message:
# ''''''
# if self.do_search:
# message = self.get_search_retrieval(message)
s_json = self._parse_json(content)
try:
if s_json and key in s_json:
return str(s_json[key])
except:
pass
# if self.do_doc_retrieval:
# message = self.get_doc_retrieval(message)
keystr = f"'{key}'" if f"'{key}'" in content else f'"{key}"'
return self._match(key2pattern.get(keystr, fr"'{key}':\s*'([^']*)'"), content, do_search=do_search, do_json=do_json)
# if self.do_tool_retrieval:
# message = self.get_tool_retrieval(message)
# return message
def _match(self, pattern, s, do_search=True, do_json=False):
try:
if do_search:
match = re.search(pattern, s)
if match:
value = match.group(1).replace("\\n", "\n")
if do_json:
value = json.loads(value)
else:
value = None
else:
match = re.findall(pattern, s, re.DOTALL)
if match:
value = match[0]
if do_json:
value = json.loads(value)
else:
value = None
except Exception as e:
logger.warning(f"{traceback.format_exc()}")
# def get_search_retrieval(self, message: Message,) -> Message:
# SEARCH_ENGINES = {"duckduckgo": DDGSTool}
# search_docs = []
# for idx, doc in enumerate(SEARCH_ENGINES["duckduckgo"].run(message.role_content, 3)):
# doc.update({"index": idx})
# search_docs.append(Doc(**doc))
# message.search_docs = search_docs
# return message
# def get_doc_retrieval(self, message: Message) -> Message:
# query = message.role_content
# knowledge_basename = message.doc_engine_name
# top_k = message.top_k
# score_threshold = message.score_threshold
# if knowledge_basename:
# docs = DocRetrieval.run(query, knowledge_basename, top_k, score_threshold)
# message.db_docs = [Doc(**doc) for doc in docs]
# return message
# def get_code_retrieval(self, message: Message) -> Message:
# # DocRetrieval.run("langchain是什么", "DSADSAD")
# query = message.input_query
# code_engine_name = message.code_engine_name
# history_node_list = message.history_node_list
# code_docs = CodeRetrieval.run(code_engine_name, query, code_limit=message.top_k, history_node_list=history_node_list)
# message.code_docs = [CodeDoc(**doc) for doc in code_docs]
# return message
# def get_tool_retrieval(self, message: Message) -> Message:
# return message
# def step_router(self, message: Message) -> tuple[Message, ...]:
# ''''''
# # message = self.parser(message)
# # logger.debug(f"message.action_status: {message.action_status}")
# observation_message = None
# if message.action_status == ActionStatus.CODING:
# message, observation_message = self.code_step(message)
# elif message.action_status == ActionStatus.TOOL_USING:
# message, observation_message = self.tool_step(message)
# logger.debug(f"pattern: {pattern}, s: {s}, match: {match}")
return value
# return message, observation_message
# def code_step(self, message: Message) -> Message:
# '''execute code'''
# # logger.debug(f"message.role_content: {message.role_content}, message.code_content: {message.code_content}")
# code_answer = self.codebox.chat('```python\n{}```'.format(message.code_content))
# code_prompt = f"执行上述代码后存在报错信息为 {code_answer.code_exe_response}，需要进行修复" \
# if code_answer.code_exe_type == "error" else f"执行上述代码后返回信息为 {code_answer.code_exe_response}"
# observation_message = Message(
# role_name="observation",
# role_type="func", #self.role.role_type,
# role_content="",
# step_content="",
# input_query=message.code_content,
# )
# uid = str(uuid.uuid1())
# if code_answer.code_exe_type == "image/png":
# message.figures[uid] = code_answer.code_exe_response
# message.code_answer = f"\n**Observation:**: 执行上述代码后生成一张图片, 图片名为{uid}\n"
# message.observation = f"\n**Observation:**: 执行上述代码后生成一张图片, 图片名为{uid}\n"
# message.step_content += f"\n**Observation:**: 执行上述代码后生成一张图片, 图片名为{uid}\n"
# message.step_contents += [f"\n**Observation:**: 执行上述代码后生成一张图片, 图片名为{uid}\n"]
# # message.role_content += f"\n**Observation:**:执行上述代码后生成一张图片, 图片名为{uid}\n"
# observation_message.role_content = f"\n**Observation:**: 执行上述代码后生成一张图片, 图片名为{uid}\n"
# observation_message.parsed_output = {"Observation": f"执行上述代码后生成一张图片, 图片名为{uid}"}
# else:
# message.code_answer = code_answer.code_exe_response
# message.observation = code_answer.code_exe_response
# message.step_content += f"\n**Observation:**: {code_prompt}\n"
# message.step_contents += [f"\n**Observation:**: {code_prompt}\n"]
# # message.role_content += f"\n**Observation:**: {code_prompt}\n"
# observation_message.role_content = f"\n**Observation:**: {code_prompt}\n"
# observation_message.parsed_output = {"Observation": code_prompt}
# # logger.info(f"**Observation:** {message.action_status}, {message.observation}")
# return message, observation_message
# def tool_step(self, message: Message) -> Message:
# '''execute tool'''
# # logger.debug(f"{message}")
# observation_message = Message(
# role_name="observation",
# role_type="function", #self.role.role_type,
# role_content="\n**Observation:** there is no tool can execute\n" ,
# step_content="",
# input_query=str(message.tool_params),
# tools=message.tools,
# )
# # logger.debug(f"message: {message.action_status}, {message.tool_name}, {message.tool_params}")
# tool_names = [tool.name for tool in message.tools]
# if message.tool_name not in tool_names:
# message.tool_answer = "\n**Observation:** there is no tool can execute\n"
# message.observation = "\n**Observation:** there is no tool can execute\n"
# # message.role_content += f"\n**Observation:**: 不存在可以执行的tool\n"
# message.step_content += f"\n**Observation:** there is no tool can execute\n"
# message.step_contents += [f"\n**Observation:** there is no tool can execute\n"]
# observation_message.role_content = f"\n**Observation:** there is no tool can execute\n"
# observation_message.parsed_output = {"Observation": "there is no tool can execute\n"}
# for tool in message.tools:
# if tool.name == message.tool_name:
# tool_res = tool.func(**message.tool_params.get("tool_params", {}))
# logger.debug(f"tool_res {tool_res}")
# message.tool_answer = tool_res
# message.observation = tool_res
# # message.role_content += f"\n**Observation:**: {tool_res}\n"
# message.step_content += f"\n**Observation:** {tool_res}\n"
# message.step_contents += [f"\n**Observation:** {tool_res}\n"]
# observation_message.role_content = f"\n**Observation:** {tool_res}\n"
# observation_message.parsed_output = {"Observation": tool_res}
# break
# # logger.info(f"**Observation:** {message.action_status}, {message.observation}")
# return message, observation_message
def _parse_json(self, s):
try:
pattern = r"```([^`]+)```"
match = re.findall(pattern, s)
if match:
return eval(match[0])
except:
pass
return None
# def parser(self, message: Message) -> Message:
# ''''''
# content = message.role_content
# parser_keys = ["action", "code_content", "code_filename", "tool_params", "plans"]
# try:
# s_json = self._parse_json(content)
# message.action_status = s_json.get("action")
# message.code_content = s_json.get("code_content")
# message.tool_params = s_json.get("tool_params")
# message.tool_name = s_json.get("tool_name")
# message.code_filename = s_json.get("code_filename")
# message.plans = s_json.get("plans")
# # for parser_key in parser_keys:
# # message.action_status = content.get(parser_key)
# except Exception as e:
# # logger.warning(f"{traceback.format_exc()}")
# def parse_text_to_dict(text):
# # Define a regular expression pattern to capture the key and value
# main_pattern = r"\*\*(.+?):\*\*\s*(.*?)\s*(?=\*\*|$)"
# list_pattern = r'```python\n(.*?)```'
# # Use re.findall to find all main matches in the text
# main_matches = re.findall(main_pattern, text, re.DOTALL)
# # Convert main matches to a dictionary
# parsed_dict = {key.strip(): value.strip() for key, value in main_matches}
# for k, v in parsed_dict.items():
# for pattern in [list_pattern]:
# if "PLAN" != k: continue
# match_value = re.search(pattern, v, re.DOTALL)
# if match_value:
# # Add the code block to the dictionary
# parsed_dict[k] = eval(match_value.group(1).strip())
# break
# return parsed_dict
# def extract_content_from_backticks(text):
# code_blocks = []
# lines = text.split('\n')
# is_code_block = False
# code_block = ''
# language = ''
# for line in lines:
# if line.startswith('```') and not is_code_block:
# is_code_block = True
# language = line[3:]
# code_block = ''
# elif line.startswith('```') and is_code_block:
# is_code_block = False
# code_blocks.append({language.strip(): code_block.strip()})
# elif is_code_block:
# code_block += line + '\n'
# return code_blocks
# def parse_dict_to_dict(parsed_dict):
# code_pattern = r'```python\n(.*?)```'
# tool_pattern = r'```tool_params\n(.*?)```'
# pattern_dict = {"code": code_pattern, "tool_params": tool_pattern}
# spec_parsed_dict = copy.deepcopy(parsed_dict)
# for key, pattern in pattern_dict.items():
# for k, text in parsed_dict.items():
# # Search for the code block
# if not isinstance(text, str): continue
# _match = re.search(pattern, text, re.DOTALL)
# if _match:
# # Add the code block to the dictionary
# try:
# spec_parsed_dict[key] = json.loads(_match.group(1).strip())
# except:
# spec_parsed_dict[key] = _match.group(1).strip()
# break
# return spec_parsed_dict
# def parse_dict_to_dict(parsed_dict):
# code_pattern = r'```python\n(.*?)```'
# tool_pattern = r'```json\n(.*?)```'
# pattern_dict = {"code": code_pattern, "json": tool_pattern}
# spec_parsed_dict = copy.deepcopy(parsed_dict)
# for key, pattern in pattern_dict.items():
# for k, text in parsed_dict.items():
# # Search for the code block
# if not isinstance(text, str): continue
# _match = re.search(pattern, text, re.DOTALL)
# if _match:
# # Add the code block to the dictionary
# logger.debug(f"dsadsa {text}")
# try:
# spec_parsed_dict[key] = json.loads(_match.group(1).strip())
# except:
# spec_parsed_dict[key] = _match.group(1).strip()
# break
# return spec_parsed_dict
# parsed_dict = parse_text_to_dict(content)
# spec_parsed_dict = parse_dict_to_dict(parsed_dict)
# action_value = parsed_dict.get('Action Status')
# if action_value:
# action_value = action_value.lower()
# logger.info(f'{self.role.role_name}: action_value: {action_value}')
# # action_value = self._match(r"'action':\s*'([^']*)'", content) if "'action'" in content else self._match(r'"action":\s*"([^"]*)"', content)
# code_content_value = spec_parsed_dict.get('code')
# # code_content_value = self._match(r"'code_content':\s*'([^']*)'", content) if "'code_content'" in content else self._match(r'"code_content":\s*"([^"]*)"', content)
# filename_value = self._match(r"'code_filename':\s*'([^']*)'", content) if "'code_filename'" in content else self._match(r'"code_filename":\s*"([^"]*)"', content)
# tool_params_value = spec_parsed_dict.get('tool_params')
# # tool_params_value = self._match(r"'tool_params':\s*(\{[^{}]*\})", content, do_json=True) if "'tool_params'" in content \
# # else self._match(r'"tool_params":\s*(\{[^{}]*\})', content, do_json=True)
# tool_name_value = self._match(r"'tool_name':\s*'([^']*)'", content) if "'tool_name'" in content else self._match(r'"tool_name":\s*"([^"]*)"', content)
# plans_value = self._match(r"'plans':\s*(\[.*?\])", content, do_search=False) if "'plans'" in content else self._match(r'"plans":\s*(\[.*?\])', content, do_search=False, )
# # re解析
# message.action_status = action_value or "default"
# message.code_content = code_content_value
# message.code_filename = filename_value
# message.tool_params = tool_params_value
# message.tool_name = tool_name_value
# message.plans = plans_value
# message.parsed_output = parsed_dict
# message.spec_parsed_output = spec_parsed_dict
# code_content_value = spec_parsed_dict.get('code')
# # code_content_value = self._match(r"'code_content':\s*'([^']*)'", content) if "'code_content'" in content else self._match(r'"code_content":\s*"([^"]*)"', content)
# filename_value = self._match(r"'code_filename':\s*'([^']*)'", content) if "'code_filename'" in content else self._match(r'"code_filename":\s*"([^"]*)"', content)
# logger.debug(spec_parsed_dict)
# if action_value == 'tool_using':
# tool_params_value = spec_parsed_dict.get('json')
# else:
# tool_params_value = None
# # tool_params_value = self._match(r"'tool_params':\s*(\{[^{}]*\})", content, do_json=True) if "'tool_params'" in content \
# # else self._match(r'"tool_params":\s*(\{[^{}]*\})', content, do_json=True)
# tool_name_value = self._match(r"'tool_name':\s*'([^']*)'", content) if "'tool_name'" in content else self._match(r'"tool_name":\s*"([^"]*)"', content)
# plans_value = self._match(r"'plans':\s*(\[.*?\])", content, do_search=False) if "'plans'" in content else self._match(r'"plans":\s*(\[.*?\])', content, do_search=False, )
# # re解析
# message.action_status = action_value or "default"
# message.code_content = code_content_value
# message.code_filename = filename_value
# message.tool_params = tool_params_value
# message.tool_name = tool_name_value
# message.plans = plans_value
# message.parsed_output = parsed_dict
# message.spec_parsed_output = spec_parsed_dict
# # logger.debug(f"确认当前的action: {message.action_status}")
# return message
# def parser_spec_key(self, content, key, do_search=True, do_json=False) -> str:
# ''''''
# key2pattern = {
# "'action'": r"'action':\s*'([^']*)'", '"action"': r'"action":\s*"([^"]*)"',
# "'code_content'": r"'code_content':\s*'([^']*)'", '"code_content"': r'"code_content":\s*"([^"]*)"',
# "'code_filename'": r"'code_filename':\s*'([^']*)'", '"code_filename"': r'"code_filename":\s*"([^"]*)"',
# "'tool_params'": r"'tool_params':\s*(\{[^{}]*\})", '"tool_params"': r'"tool_params":\s*(\{[^{}]*\})',
# "'tool_name'": r"'tool_name':\s*'([^']*)'", '"tool_name"': r'"tool_name":\s*"([^"]*)"',
# "'plans'": r"'plans':\s*(\[.*?\])", '"plans"': r'"plans":\s*(\[.*?\])',
# "'content'": r"'content':\s*'([^']*)'", '"content"': r'"content":\s*"([^"]*)"',
# }
# s_json = self._parse_json(content)
# try:
# if s_json and key in s_json:
# return str(s_json[key])
# except:
# pass
# keystr = f"'{key}'" if f"'{key}'" in content else f'"{key}"'
# return self._match(key2pattern.get(keystr, fr"'{key}':\s*'([^']*)'"), content, do_search=do_search, do_json=do_json)
# def _match(self, pattern, s, do_search=True, do_json=False):
# try:
# if do_search:
# match = re.search(pattern, s)
# if match:
# value = match.group(1).replace("\\n", "\n")
# if do_json:
# value = json.loads(value)
# else:
# value = None
# else:
# match = re.findall(pattern, s, re.DOTALL)
# if match:
# value = match[0]
# if do_json:
# value = json.loads(value)
# else:
# value = None
# except Exception as e:
# logger.warning(f"{traceback.format_exc()}")
# # logger.debug(f"pattern: {pattern}, s: {s}, match: {match}")
# return value
# def _parse_json(self, s):
# try:
# pattern = r"```([^`]+)```"
# match = re.findall(pattern, s)
# if match:
# return eval(match[0])
# except:
# pass
# return None
def get_memory(self, ):
def get_memory(self, content_key="role_content"):
return self.memory.to_tuple_messages(content_key="step_content")
def get_memory_str(self, ):
def get_memory_str(self, content_key="role_content"):
return "\n".join([": ".join(i) for i in self.memory.to_tuple_messages(content_key="step_content")])

110
dev_opsgpt/connector/agents/check_agent.py Normal file
View File

@ -0,0 +1,110 @@
from pydantic import BaseModel
from typing import List, Union
import re
import json
import traceback
import copy
from loguru import logger
from langchain.prompts.chat import ChatPromptTemplate
from dev_opsgpt.connector.schema import (
Memory, Task, Env, Role, Message, ActionStatus
)
from dev_opsgpt.llm_models import getChatModel
from dev_opsgpt.connector.configs.agent_config import REACT_PROMPT_INPUT, CONTEXT_PROMPT_INPUT, QUERY_CONTEXT_PROMPT_INPUT
from .base_agent import BaseAgent
class CheckAgent(BaseAgent):
def __init__(
self,
role: Role,
task: Task = None,
memory: Memory = None,
chat_turn: int = 1,
do_search: bool = False,
do_doc_retrieval: bool = False,
do_tool_retrieval: bool = False,
temperature: float = 0.2,
stop: Union[List[str], str] = None,
do_filter: bool = True,
do_use_self_memory: bool = True,
focus_agents: List[str] = [],
focus_message_keys: List[str] = [],
# prompt_mamnger: PromptManager
):
super().__init__(role, task, memory, chat_turn, do_search, do_doc_retrieval,
do_tool_retrieval, temperature, stop, do_filter,do_use_self_memory,
focus_agents, focus_message_keys
)
def create_prompt(
self, query: Message, memory: Memory =None, history: Memory = None, background: Memory = None, memory_pool: Memory=None, prompt_mamnger=None) -> str:
'''
role\task\tools\docs\memory
'''
#
doc_infos = self.create_doc_prompt(query)
code_infos = self.create_codedoc_prompt(query)
#
formatted_tools, tool_names = self.create_tools_prompt(query)
task_prompt = self.create_task_prompt(query)
background_prompt = self.create_background_prompt(background)
history_prompt = self.create_history_prompt(history)
selfmemory_prompt = self.create_selfmemory_prompt(memory, control_key="step_content")
# react 流程是自身迭代过程，另外二次触发的是需要作为历史对话信息
# input_query = react_memory.to_tuple_messages(content_key="step_content")
input_query = query.input_query
# logger.debug(f"{self.role.role_name} extra_system_prompt: {self.role.role_prompt}")
# logger.debug(f"{self.role.role_name} input_query: {input_query}")
# logger.debug(f"{self.role.role_name} doc_infos: {doc_infos}")
# logger.debug(f"{self.role.role_name} tool_names: {tool_names}")
# prompt += "\n" + CHECK_PROMPT_INPUT.format(**{"query": input_query})
# prompt.format(**{"query": input_query})
# extra_system_prompt = self.role.role_prompt
prompt = self.role.role_prompt.format(**{"query": input_query, "formatted_tools": formatted_tools, "tool_names": tool_names})
if "**Context:**" in self.role.role_prompt:
# logger.debug(f"parsed_output_list: {query.parsed_output_list}")
# input_query = "'''" + "\n".join([f"*{k}*\n{v}" for i in background.get_parserd_output_list() for k,v in i.items() if "Action Status" !=k]) + "'''"
context = "\n".join([f"*{k}*\n{v}" for i in background.get_parserd_output_list() for k,v in i.items() if "Action Status" !=k])
# logger.debug(context)
# logger.debug(f"parsed_output_list: {t}")
prompt += "\n" + QUERY_CONTEXT_PROMPT_INPUT.format(**{"query": query.origin_query, "context": context})
else:
prompt += "\n" + REACT_PROMPT_INPUT.format(**{"query": input_query})
task = query.task or self.task
if task_prompt is not None:
prompt += "\n" + task.task_prompt
# if doc_infos is not None and doc_infos!="" and doc_infos!="不存在知识库辅助信息":
# prompt += f"\n知识库信息: {doc_infos}"
# if code_infos is not None and code_infos!="" and code_infos!="不存在代码库辅助信息":
# prompt += f"\n代码库信息: {code_infos}"
# if background_prompt:
# prompt += "\n" + background_prompt
# if history_prompt:
# prompt += "\n" + history_prompt
# if selfmemory_prompt:
# prompt += "\n" + selfmemory_prompt
# prompt = extra_system_prompt.format(**{"query": input_query, "doc_infos": doc_infos, "formatted_tools": formatted_tools, "tool_names": tool_names})
while "{{" in prompt or "}}" in prompt:
prompt = prompt.replace("{{", "{")
prompt = prompt.replace("}}", "}")
# logger.debug(f"{self.role.role_name} prompt: {prompt}")
return prompt

214
dev_opsgpt/connector/agents/executor_agent.py Normal file
View File

@ -0,0 +1,214 @@
from pydantic import BaseModel
from typing import List, Union, Tuple, Any
import re
import json
import traceback
import copy
from loguru import logger
from langchain.prompts.chat import ChatPromptTemplate
from dev_opsgpt.connector.schema import (
Memory, Task, Env, Role, Message, ActionStatus
)
from dev_opsgpt.llm_models import getChatModel
from dev_opsgpt.connector.configs.prompts import EXECUTOR_PROMPT_INPUT, BEGIN_PROMPT_INPUT
from dev_opsgpt.connector.utils import parse_section
from .base_agent import BaseAgent
class ExecutorAgent(BaseAgent):
def __init__(
self,
role: Role,
task: Task = None,
memory: Memory = None,
chat_turn: int = 1,
do_search: bool = False,
do_doc_retrieval: bool = False,
do_tool_retrieval: bool = False,
temperature: float = 0.2,
stop: Union[List[str], str] = None,
do_filter: bool = True,
do_use_self_memory: bool = True,
focus_agents: List[str] = [],
focus_message_keys: List[str] = [],
# prompt_mamnger: PromptManager
):
super().__init__(role, task, memory, chat_turn, do_search, do_doc_retrieval,
do_tool_retrieval, temperature, stop, do_filter,do_use_self_memory,
focus_agents, focus_message_keys
)
self.do_all_task = True # run all tasks
def arun(self, query: Message, history: Memory = None, background: Memory = None, memory_pool: Memory=None) -> Message:
'''agent reponse from multi-message'''
# insert query into memory
task_executor_memory = Memory(messages=[])
# insert query
output_message = Message(
role_name=self.role.role_name,
role_type="ai", #self.role.role_type,
role_content=query.input_query,
step_content="",
input_query=query.input_query,
tools=query.tools,
parsed_output_list=[query.parsed_output]
)
self_memory = self.memory if self.do_use_self_memory else None
plan_step = int(query.parsed_output.get("PLAN_STEP", 0))
# 如果存在plan字段且plan字段为str的时候
if "PLAN" not in query.parsed_output or isinstance(query.parsed_output.get("PLAN", []), str) or plan_step >= len(query.parsed_output.get("PLAN", [])):
query_c = copy.deepcopy(query)
query_c.parsed_output = {"Question": query_c.input_query}
task_executor_memory.append(query_c)
for output_message, task_executor_memory in self._arun_step(output_message, query_c, self_memory, history, background, memory_pool, task_executor_memory):
pass
# task_executor_memory.append(query_c)
# content = "the execution step of the plan is exceed the planned scope."
# output_message.parsed_dict = {"Thought": content, "Action Status": "finished", "Action": content}
# task_executor_memory.append(output_message)
elif "PLAN" in query.parsed_output:
logger.debug(f"{query.parsed_output['PLAN']}")
if self.do_all_task:
# run all tasks step by step
for task_content in query.parsed_output["PLAN"][plan_step:]:
# create your llm prompt
query_c = copy.deepcopy(query)
query_c.parsed_output = {"Question": task_content}
task_executor_memory.append(query_c)
for output_message, task_executor_memory in self._arun_step(output_message, query_c, self_memory, history, background, memory_pool, task_executor_memory):
pass
yield output_message
else:
query_c = copy.deepcopy(query)
task_content = query_c.parsed_output["PLAN"][plan_step]
query_c.parsed_output = {"Question": task_content}
task_executor_memory.append(query_c)
for output_message, task_executor_memory in self._arun_step(output_message, query_c, self_memory, history, background, memory_pool, task_executor_memory):
pass
output_message.parsed_output.update({"CURRENT_STEP": plan_step})
# update self_memory
self.append_history(query)
self.append_history(output_message)
# logger.info(f"{self.role.role_name} currenct question: {output_message.input_query}\nllm_executor_run: {output_message.step_content}")
# logger.info(f"{self.role.role_name} currenct parserd_output_list: {output_message.parserd_output_list}")
output_message.input_query = output_message.role_content
# update memory pool
memory_pool.append(output_message)
yield output_message
def _arun_step(self, output_message: Message, query: Message, self_memory: Memory,
history: Memory, background: Memory, memory_pool: Memory,
react_memory: Memory) -> Union[Message, Memory]:
'''execute the llm predict by created prompt'''
prompt = self.create_prompt(query, self_memory, history, background, memory_pool=memory_pool, react_memory=react_memory)
content = self.llm.predict(prompt)
# logger.debug(f"{self.role.role_name} prompt: {prompt}")
logger.debug(f"{self.role.role_name} content: {content}")
output_message.role_content = content
output_message.role_contents += [content]
output_message.step_content += "\n"+output_message.role_content
output_message.step_contents + [output_message.role_content]
output_message = self.message_utils.parser(output_message)
# according the output to choose one action for code_content or tool_content
output_message, observation_message = self.message_utils.step_router(output_message)
# logger.debug(f"{self.role.role_name} content: {content}")
# update parserd_output_list
output_message.parsed_output_list.append(output_message.parsed_output)
react_message = copy.deepcopy(output_message)
react_memory.append(react_message)
if observation_message:
react_memory.append(observation_message)
output_message.parsed_output_list.append(observation_message.parsed_output)
logger.debug(f"{observation_message.role_name} content: {observation_message.role_content}")
yield output_message, react_memory
def create_prompt(
self, query: Message, memory: Memory =None, history: Memory = None, background: Memory = None, memory_pool: Memory=None, react_memory: Memory = None, prompt_mamnger=None) -> str:
'''
role\task\tools\docs\memory
'''
#
doc_infos = self.create_doc_prompt(query)
code_infos = self.create_codedoc_prompt(query)
#
formatted_tools, tool_names = self.create_tools_prompt(query)
task_prompt = self.create_task_prompt(query)
background_prompt = self.create_background_prompt(background, control_key="step_content")
history_prompt = self.create_history_prompt(history)
selfmemory_prompt = self.create_selfmemory_prompt(memory, control_key="step_content")
#
memory_pool_select_by_agent_key = self.select_memory_by_agent_key(memory_pool)
memory_pool_select_by_agent_key_context = '\n\n'.join([
f"*{k}*\n{v}" for parsed_output in memory_pool_select_by_agent_key.get_parserd_output_list() for k, v in parsed_output.items() if k not in ['Action Status']
])
DocInfos = ""
if doc_infos is not None and doc_infos!="" and doc_infos!="不存在知识库辅助信息":
DocInfos += f"\nDocument Information: {doc_infos}"
if code_infos is not None and code_infos!="" and code_infos!="不存在代码库辅助信息":
DocInfos += f"\nCodeBase Infomation: {code_infos}"
# extra_system_prompt = self.role.role_prompt
prompt = self.role.role_prompt.format(**{"formatted_tools": formatted_tools, "tool_names": tool_names})
# input_query = react_memory.to_tuple_messages(content_key="role_content")
# logger.debug(f"get_parserd_dict {react_memory.get_parserd_output()}")
input_query = "\n".join(["\n".join([f"**{k}:**\n{v}" for k,v in _dict.items()]) for _dict in react_memory.get_parserd_output()])
# input_query = query.input_query + "\n".join([f"{v}" for k, v in input_query if v])
last_agent_parsed_output = "\n".join(["\n".join([f"*{k}*\n{v}" for k,v in _dict.items()]) for _dict in query.parsed_output_list])
react_parsed_output = "\n".join(["\n".join([f"*{k}_context*\n{v}" for k,v in _dict.items()]) for _dict in react_memory.get_parserd_output()[:-1]])
#
prompt += "\n" + BEGIN_PROMPT_INPUT
input_keys = parse_section(self.role.role_prompt, 'Input Format')
if input_keys:
for input_key in input_keys:
if input_key == "Origin Query":
prompt += "\n**Origin Query:**\n" + query.origin_query
elif input_key == "DocInfos":
prompt += "\n**DocInfos:**\n" + DocInfos
elif input_key == "Context":
if self.focus_agents and memory_pool_select_by_agent_key_context:
context = memory_pool_select_by_agent_key_context
else:
context = last_agent_parsed_output
prompt += "\n**Context:**\n" + context + f"\n{react_parsed_output}"
elif input_key == "Question":
prompt += "\n**Question:**\n" + query.parsed_output.get("Question")
else:
prompt += "\n" + input_query
task = query.task or self.task
# if task_prompt is not None:
# prompt += "\n" + task.task_prompt
# if selfmemory_prompt:
# prompt += "\n" + selfmemory_prompt
# if background_prompt:
# prompt += "\n" + background_prompt
# if history_prompt:
# prompt += "\n" + history_prompt
# prompt = extra_system_prompt.format(**{"query": input_query, "doc_infos": doc_infos, "formatted_tools": formatted_tools, "tool_names": tool_names})
while "{{" in prompt or "}}" in prompt:
prompt = prompt.replace("{{", "{")
prompt = prompt.replace("}}", "}")
return prompt
def set_task(self, do_all_task):
'''set task exec type'''
self.do_all_task = do_all_task

115
dev_opsgpt/connector/agents/react_agent.py
View File

@ -1,15 +1,16 @@
from pydantic import BaseModel
from typing import List, Union
import re
import json
import traceback
import copy
from loguru import logger
from langchain.prompts.chat import ChatPromptTemplate
from dev_opsgpt.connector.connector_schema import Message
from dev_opsgpt.connector.shcema.memory import Memory
from dev_opsgpt.connector.connector_schema import Task, Env, Role, Message, ActionStatus
from dev_opsgpt.connector.schema import (
Memory, Task, Env, Role, Message, ActionStatus
)
from dev_opsgpt.llm_models import getChatModel
from dev_opsgpt.connector.configs.agent_config import REACT_PROMPT_INPUT
@ -27,63 +28,93 @@ class ReactAgent(BaseAgent):
do_doc_retrieval: bool = False,
do_tool_retrieval: bool = False,
temperature: float = 0.2,
stop: Union[List[str], str] = "观察",
stop: Union[List[str], str] = None,
do_filter: bool = True,
do_use_self_memory: bool = True,
# docs_prompt: str,
focus_agents: List[str] = [],
focus_message_keys: List[str] = [],
# prompt_mamnger: PromptManager
):
super().__init__(role, task, memory, chat_turn, do_search, do_doc_retrieval,
do_tool_retrieval, temperature, stop, do_filter,do_use_self_memory
do_tool_retrieval, temperature, stop, do_filter,do_use_self_memory,
focus_agents, focus_message_keys
)
def run(self, query: Message, history: Memory = None, background: Memory = None) -> Message:
def run(self, query: Message, history: Memory = None, background: Memory = None, memory_pool: Memory = None) -> Message:
'''agent reponse from multi-message'''
for message in self.arun(query, history, background, memory_pool):
pass
return message
def arun(self, query: Message, history: Memory = None, background: Memory = None, memory_pool: Memory = None) -> Message:
'''agent reponse from multi-message'''
step_nums = copy.deepcopy(self.chat_turn)
react_memory = Memory([])
# 问题插入
react_memory = Memory(messages=[])
# insert query
output_message = Message(
role_name=self.role.role_name,
role_type="ai", #self.role.role_type,
role_content=query.input_query,
step_content=query.input_query,
step_content="",
input_query=query.input_query,
tools=query.tools
tools=query.tools,
parsed_output_list=[query.parsed_output]
)
react_memory.append(output_message)
query_c = copy.deepcopy(query)
query_c.parsed_output = {"Question": "\n".join([f"{v}" for k, v in query.parsed_output.items() if k not in ["Action Status"]])}
react_memory.append(query_c)
self_memory = self.memory if self.do_use_self_memory else None
idx = 0
# start to react
while step_nums > 0:
output_message.role_content = output_message.step_content
self_memory = self.memory if self.do_use_self_memory else None
prompt = self.create_prompt(query, self_memory, history, background, react_memory)
prompt = self.create_prompt(query, self_memory, history, background, react_memory, memory_pool)
try:
content = self.llm.predict(prompt)
except Exception as e:
logger.warning(f"error prompt: {prompt}")
raise Exception(traceback.format_exc())
output_message.role_content = content
output_message.role_content = "\n"+content
output_message.role_contents += [content]
output_message.step_content += output_message.role_content
output_message.step_content += "\n"+output_message.role_content
output_message.step_contents + [output_message.role_content]
yield output_message
# logger.debug(f"{self.role.role_name}, {idx} iteration prompt: {prompt}")
# logger.info(f"{self.role.role_name}, {idx} iteration step_run: {output_message.role_content}")
logger.info(f"{self.role.role_name}, {idx} iteration step_run: {output_message.role_content}")
output_message = self.parser(output_message)
output_message = self.message_utils.parser(output_message)
# when get finished signal can stop early
if output_message.action_status == ActionStatus.FINISHED: break
# according the output to choose one action for code_content or tool_content
output_message = self.step_router(output_message)
logger.info(f"{self.role.role_name} react_run: {output_message.role_content}")
output_message, observation_message = self.message_utils.step_router(output_message)
output_message.parsed_output_list.append(output_message.parsed_output)
react_message = copy.deepcopy(output_message)
react_memory.append(react_message)
if observation_message:
react_memory.append(observation_message)
output_message.parsed_output_list.append(observation_message.parsed_output)
# logger.debug(f"{observation_message.role_name} content: {observation_message.role_content}")
# logger.info(f"{self.role.role_name} currenct question: {output_message.input_query}\nllm_react_run: {output_message.role_content}")
idx += 1
step_nums -= 1
yield output_message
# react' self_memory saved at last
self.append_history(output_message)
return output_message
# update memory pool
# memory_pool.append(output_message)
output_message.input_query = query.input_query
# update memory pool
memory_pool.append(output_message)
yield output_message
def create_prompt(
self, query: Message, memory: Memory =None, history: Memory = None, background: Memory = None, react_memory: Memory = None, prompt_mamnger=None) -> str:
self, query: Message, memory: Memory =None, history: Memory = None, background: Memory = None, react_memory: Memory = None, memory_pool: Memory= None,
prompt_mamnger=None) -> str:
'''
role\task\tools\docs\memory
'''
@ -100,35 +131,39 @@ class ReactAgent(BaseAgent):
# extra_system_prompt = self.role.role_prompt
prompt = self.role.role_prompt.format(**{"formatted_tools": formatted_tools, "tool_names": tool_names})
# react 流程是自身迭代过程，另外二次触发的是需要作为历史对话信息
# input_query = react_memory.to_tuple_messages(content_key="step_content")
# # input_query = query.input_query + "\n" + "\n".join([f"{v}" for k, v in input_query if v])
# input_query = "\n".join([f"{v}" for k, v in input_query if v])
input_query = "\n".join(["\n".join([f"**{k}:**\n{v}" for k,v in _dict.items()]) for _dict in react_memory.get_parserd_output()])
logger.debug(f"input_query: {input_query}")
prompt += "\n" + REACT_PROMPT_INPUT.format(**{"query": input_query})
task = query.task or self.task
if task_prompt is not None:
prompt += "\n" + task.task_prompt
# if task_prompt is not None:
# prompt += "\n" + task.task_prompt
if doc_infos is not None and doc_infos!="" and doc_infos!="不存在知识库辅助信息":
prompt += f"\n知识库信息: {doc_infos}"
# if doc_infos is not None and doc_infos!="" and doc_infos!="不存在知识库辅助信息":
# prompt += f"\n知识库信息: {doc_infos}"
if code_infos is not None and code_infos!="" and code_infos!="不存在代码库辅助信息":
prompt += f"\n代码库信息: {code_infos}"
# if code_infos is not None and code_infos!="" and code_infos!="不存在代码库辅助信息":
# prompt += f"\n代码库信息: {code_infos}"
if background_prompt:
prompt += "\n" + background_prompt
# if background_prompt:
# prompt += "\n" + background_prompt
if history_prompt:
prompt += "\n" + history_prompt
# if history_prompt:
# prompt += "\n" + history_prompt
if selfmemory_prompt:
prompt += "\n" + selfmemory_prompt
# react 流程是自身迭代过程，另外二次触发的是需要作为历史对话信息
input_query = react_memory.to_tuple_messages(content_key="step_content")
input_query = "\n".join([f"{v}" for k, v in input_query if v])
# if selfmemory_prompt:
# prompt += "\n" + selfmemory_prompt
# logger.debug(f"{self.role.role_name} extra_system_prompt: {self.role.role_prompt}")
# logger.debug(f"{self.role.role_name} input_query: {input_query}")
# logger.debug(f"{self.role.role_name} doc_infos: {doc_infos}")
# logger.debug(f"{self.role.role_name} tool_names: {tool_names}")
prompt += "\n" + REACT_PROMPT_INPUT.format(**{"query": input_query})
# prompt += "\n" + REACT_PROMPT_INPUT.format(**{"query": input_query})
# prompt = extra_system_prompt.format(**{"query": input_query, "doc_infos": doc_infos, "formatted_tools": formatted_tools, "tool_names": tool_names})
while "{{" in prompt or "}}" in prompt:

109
dev_opsgpt/connector/agents/selector_agent.py Normal file
View File

@ -0,0 +1,109 @@
from pydantic import BaseModel
from typing import List, Union
import re
import json
import traceback
import copy
from loguru import logger
from langchain.prompts.chat import ChatPromptTemplate
from dev_opsgpt.connector.schema import (
Memory, Task, Env, Role, Message, ActionStatus
)
from dev_opsgpt.llm_models import getChatModel
from dev_opsgpt.connector.configs.agent_config import REACT_PROMPT_INPUT, CONTEXT_PROMPT_INPUT, QUERY_CONTEXT_PROMPT_INPUT
from .base_agent import BaseAgent
class SelectorAgent(BaseAgent):
def __init__(
self,
role: Role,
task: Task = None,
memory: Memory = None,
chat_turn: int = 1,
do_search: bool = False,
do_doc_retrieval: bool = False,
do_tool_retrieval: bool = False,
temperature: float = 0.2,
stop: Union[List[str], str] = None,
do_filter: bool = True,
do_use_self_memory: bool = True,
focus_agents: List[str] = [],
focus_message_keys: List[str] = [],
# prompt_mamnger: PromptManager
):
super().__init__(role, task, memory, chat_turn, do_search, do_doc_retrieval,
do_tool_retrieval, temperature, stop, do_filter,do_use_self_memory,
focus_agents, focus_message_keys
)
def create_prompt(
self, query: Message, memory: Memory =None, history: Memory = None, background: Memory = None, memory_pool: Memory=None, prompt_mamnger=None) -> str:
'''
role\task\tools\docs\memory
'''
#
doc_infos = self.create_doc_prompt(query)
code_infos = self.create_codedoc_prompt(query)
#
formatted_tools, tool_names = self.create_tools_prompt(query)
task_prompt = self.create_task_prompt(query)
background_prompt = self.create_background_prompt(background)
history_prompt = self.create_history_prompt(history)
selfmemory_prompt = self.create_selfmemory_prompt(memory, control_key="step_content")
# react 流程是自身迭代过程，另外二次触发的是需要作为历史对话信息
# input_query = react_memory.to_tuple_messages(content_key="step_content")
input_query = query.input_query
# logger.debug(f"{self.role.role_name} extra_system_prompt: {self.role.role_prompt}")
logger.debug(f"{self.role.role_name} input_query: {input_query}")
# logger.debug(f"{self.role.role_name} doc_infos: {doc_infos}")
# logger.debug(f"{self.role.role_name} tool_names: {tool_names}")
# prompt += "\n" + CHECK_PROMPT_INPUT.format(**{"query": input_query})
# prompt.format(**{"query": input_query})
# extra_system_prompt = self.role.role_prompt
prompt = self.role.role_prompt.format(**{"query": input_query, "formatted_tools": formatted_tools, "tool_names": tool_names})
if "**Context:**" in self.role.role_prompt:
# logger.debug(f"parsed_output_list: {query.parsed_output_list}")
# input_query = "'''" + "\n".join([f"*{k}*\n{v}" for i in background.get_parserd_output_list() for k,v in i.items() if "Action Status" !=k]) + "'''"
context = "\n".join([f"*{k}*\n{v}" for i in background.get_parserd_output_list() for k,v in i.items() if "Action Status" !=k])
# logger.debug(f"parsed_output_list: {t}")
prompt += "\n" + QUERY_CONTEXT_PROMPT_INPUT.format(**{"query": query.origin_query, "context": context})
else:
prompt += "\n" + REACT_PROMPT_INPUT.format(**{"query": input_query})
task = query.task or self.task
if task_prompt is not None:
prompt += "\n" + task.task_prompt
# if doc_infos is not None and doc_infos!="" and doc_infos!="不存在知识库辅助信息":
# prompt += f"\n知识库信息: {doc_infos}"
# if code_infos is not None and code_infos!="" and code_infos!="不存在代码库辅助信息":
# prompt += f"\n代码库信息: {code_infos}"
# if background_prompt:
# prompt += "\n" + background_prompt
# if history_prompt:
# prompt += "\n" + history_prompt
# if selfmemory_prompt:
# prompt += "\n" + selfmemory_prompt
# prompt = extra_system_prompt.format(**{"query": input_query, "doc_infos": doc_infos, "formatted_tools": formatted_tools, "tool_names": tool_names})
while "{{" in prompt or "}}" in prompt:
prompt = prompt.replace("{{", "{")
prompt = prompt.replace("}}", "}")
# logger.debug(f"{self.role.role_name} prompt: {prompt}")
return prompt

481
dev_opsgpt/connector/chains/base_chain.py
View File

@ -7,13 +7,14 @@ import traceback
import uuid
import copy
from dev_opsgpt.connector.agents import BaseAgent
from dev_opsgpt.connector.agents import BaseAgent, CheckAgent
from dev_opsgpt.tools.base_tool import BaseTools, Tool
from dev_opsgpt.connector.shcema.memory import Memory
from dev_opsgpt.connector.connector_schema import (
Role, Message, ActionStatus, ChainConfig,
from dev_opsgpt.connector.schema import (
Memory, Role, Message, ActionStatus, ChainConfig,
load_role_configs
)
from dev_opsgpt.connector.message_process import MessageUtils
from dev_opsgpt.sandbox import PyCodeBox, CodeBoxResponse
@ -37,7 +38,7 @@ class BaseChain:
self.agents = agents
self.chat_turn = chat_turn
self.do_checker = do_checker
self.checker = BaseAgent(role=role_configs["checker"].role,
self.checker = CheckAgent(role=role_configs["checker"].role,
task = None,
memory = None,
do_search = role_configs["checker"].do_search,
@ -45,37 +46,64 @@ class BaseChain:
do_tool_retrieval = role_configs["checker"].do_tool_retrieval,
do_filter=False, do_use_self_memory=False)
self.global_memory = Memory([])
self.local_memory = Memory([])
self.do_code_exec = do_code_exec
self.codebox = PyCodeBox(
remote_url=SANDBOX_SERVER["url"],
remote_ip=SANDBOX_SERVER["host"],
remote_port=SANDBOX_SERVER["port"],
token="mytoken",
do_code_exe=True,
do_remote=SANDBOX_SERVER["do_remote"],
do_check_net=False
)
self.do_agent_selector = False
self.agent_selector = CheckAgent(role=role_configs["checker"].role,
task = None,
memory = None,
do_search = role_configs["checker"].do_search,
do_doc_retrieval = role_configs["checker"].do_doc_retrieval,
do_tool_retrieval = role_configs["checker"].do_tool_retrieval,
do_filter=False, do_use_self_memory=False)
self.messageUtils = MessageUtils()
# all memory created by agent until instance deleted
self.global_memory = Memory(messages=[])
# self.do_code_exec = do_code_exec
# self.codebox = PyCodeBox(
# remote_url=SANDBOX_SERVER["url"],
# remote_ip=SANDBOX_SERVER["host"],
# remote_port=SANDBOX_SERVER["port"],
# token="mytoken",
# do_code_exe=True,
# do_remote=SANDBOX_SERVER["do_remote"],
# do_check_net=False
# )
def step(self, query: Message, history: Memory = None, background: Memory = None) -> Message:
def step(self, query: Message, history: Memory = None, background: Memory = None, memory_pool: Memory = None) -> Message:
'''execute chain'''
local_memory = Memory([])
for output_message, local_memory in self.astep(query, history, background, memory_pool):
pass
return output_message, local_memory
def astep(self, query: Message, history: Memory = None, background: Memory = None, memory_pool: Memory = None) -> Message:
'''execute chain'''
local_memory = Memory(messages=[])
input_message = copy.deepcopy(query)
step_nums = copy.deepcopy(self.chat_turn)
check_message = None
self.global_memory.append(input_message)
local_memory.append(input_message)
# local_memory.append(input_message)
while step_nums > 0:
for agent in self.agents:
output_message = agent.run(input_message, history, background=background)
output_message = self.inherit_extrainfo(input_message, output_message)
if self.do_agent_selector:
agent_message = copy.deepcopy(query)
agent_message.agents = self.agents
for selectory_message in self.agent_selector.arun(query, background=self.global_memory, memory_pool=memory_pool):
pass
selectory_message = self.messageUtils.parser(selectory_message)
selectory_message = self.messageUtils.filter(selectory_message)
agent = self.agents[selectory_message.agent_index]
# selector agent to execure next task
for output_message in agent.arun(input_message, history, background=background, memory_pool=memory_pool):
# logger.debug(f"local_memory {local_memory + output_message}")
yield output_message, local_memory + output_message
output_message = self.messageUtils.inherit_extrainfo(input_message, output_message)
# according the output to choose one action for code_content or tool_content
logger.info(f"{agent.role.role_name} message: {output_message.role_content}")
output_message = self.parser(output_message)
# output_message = self.step_router(output_message)
# logger.info(f"{agent.role.role_name}\nmessage: {output_message.step_content}\nquery: {output_message.input_query}")
output_message = self.messageUtils.parser(output_message)
yield output_message, local_memory + output_message
input_message = output_message
self.global_memory.append(output_message)
@ -84,192 +112,243 @@ class BaseChain:
# when get finished signal can stop early
if output_message.action_status == ActionStatus.FINISHED:
break
else:
for agent in self.agents:
for output_message in agent.arun(input_message, history, background=background, memory_pool=memory_pool):
# logger.debug(f"local_memory {local_memory + output_message}")
yield output_message, local_memory + output_message
output_message = self.messageUtils.inherit_extrainfo(input_message, output_message)
# according the output to choose one action for code_content or tool_content
# logger.info(f"{agent.role.role_name} currenct message: {output_message.step_content}\n next llm question: {output_message.input_query}")
output_message = self.messageUtils.parser(output_message)
yield output_message, local_memory + output_message
# output_message = self.step_router(output_message)
if self.do_checker:
logger.debug(f"{self.checker.role.role_name} input global memory: {self.global_memory.to_str_messages(content_key='step_content')}")
check_message = self.checker.run(query, background=self.global_memory)
check_message = self.parser(check_message)
check_message = self.filter(check_message)
check_message = self.inherit_extrainfo(output_message, check_message)
logger.debug(f"{self.checker.role.role_name}: {check_message.role_content}")
input_message = output_message
self.global_memory.append(output_message)
if check_message.action_status == ActionStatus.FINISHED:
self.global_memory.append(check_message)
break
local_memory.append(output_message)
# when get finished signal can stop early
if output_message.action_status == ActionStatus.FINISHED:
action_status = False
break
if self.do_checker and self.chat_turn > 1:
# logger.debug(f"{self.checker.role.role_name} input global memory: {self.global_memory.to_str_messages(content_key='step_content', return_all=False)}")
for check_message in self.checker.arun(query, background=local_memory, memory_pool=memory_pool):
pass
check_message = self.messageUtils.parser(check_message)
check_message = self.messageUtils.filter(check_message)
check_message = self.messageUtils.inherit_extrainfo(output_message, check_message)
logger.debug(f"{self.checker.role.role_name}: {check_message.role_content}")
if check_message.action_status == ActionStatus.FINISHED:
self.global_memory.append(check_message)
break
step_nums -= 1
return check_message or output_message, local_memory
def step_router(self, message: Message) -> Message:
''''''
# message = self.parser(message)
# logger.debug(f"message.action_status: {message.action_status}")
if message.action_status == ActionStatus.CODING:
message = self.code_step(message)
elif message.action_status == ActionStatus.TOOL_USING:
message = self.tool_step(message)
return message
def code_step(self, message: Message) -> Message:
'''execute code'''
# logger.debug(f"message.role_content: {message.role_content}, message.code_content: {message.code_content}")
code_answer = self.codebox.chat('```python\n{}```'.format(message.code_content))
uid = str(uuid.uuid1())
if code_answer.code_exe_type == "image/png":
message.figures[uid] = code_answer.code_exe_response
message.code_answer = f"\n观察: 执行代码后获得输出一张图片, 文件名为{uid}\n"
message.observation = f"\n观察: 执行代码后获得输出一张图片, 文件名为{uid}\n"
message.step_content += f"\n观察: 执行代码后获得输出一张图片, 文件名为{uid}\n"
message.step_contents += [f"\n观察: 执行代码后获得输出一张图片, 文件名为{uid}\n"]
message.role_content += f"\n执行代码后获得输出一张图片, 文件名为{uid}\n"
else:
message.code_answer = code_answer.code_exe_response
message.observation = code_answer.code_exe_response
message.step_content += f"\n观察: 执行代码后获得输出是 {code_answer.code_exe_response}\n"
message.step_contents += [f"\n观察: 执行代码后获得输出是 {code_answer.code_exe_response}\n"]
message.role_content += f"\n观察: 执行代码后获得输出是 {code_answer.code_exe_response}\n"
logger.info(f"观察: {message.action_status}, {message.observation}")
return message
def tool_step(self, message: Message) -> Message:
'''execute tool'''
# logger.debug(f"message: {message.action_status}, {message.tool_name}, {message.tool_params}")
tool_names = [tool.name for tool in message.tools]
if message.tool_name not in tool_names:
message.tool_answer = "不存在可以执行的tool"
message.observation = "不存在可以执行的tool"
message.role_content += f"\n观察: 不存在可以执行的tool\n"
message.step_content += f"\n观察: 不存在可以执行的tool\n"
message.step_contents += [f"\n观察: 不存在可以执行的tool\n"]
for tool in message.tools:
if tool.name == message.tool_name:
tool_res = tool.func(**message.tool_params)
message.tool_answer = tool_res
message.observation = tool_res
message.role_content += f"\n观察: {tool_res}\n"
message.step_content += f"\n观察: {tool_res}\n"
message.step_contents += [f"\n观察: {tool_res}\n"]
return message
def filter(self, message: Message, stop=None) -> Message:
tool_params = self.parser_spec_key(message.role_content, "tool_params")
code_content = self.parser_spec_key(message.role_content, "code_content")
plan = self.parser_spec_key(message.role_content, "plan")
plans = self.parser_spec_key(message.role_content, "plans", do_search=False)
content = self.parser_spec_key(message.role_content, "content", do_search=False)
# logger.debug(f"tool_params: {tool_params}, code_content: {code_content}, plan: {plan}, plans: {plans}, content: {content}")
role_content = tool_params or code_content or plan or plans or content
message.role_content = role_content or message.role_content
return message
def parser(self, message: Message) -> Message:
''''''
content = message.role_content
parser_keys = ["action", "code_content", "code_filename", "tool_params", "plans"]
try:
s_json = self._parse_json(content)
message.action_status = s_json.get("action")
message.code_content = s_json.get("code_content")
message.tool_params = s_json.get("tool_params")
message.tool_name = s_json.get("tool_name")
message.code_filename = s_json.get("code_filename")
message.plans = s_json.get("plans")
# for parser_key in parser_keys:
# message.action_status = content.get(parser_key)
except Exception as e:
# logger.warning(f"{traceback.format_exc()}")
action_value = self._match(r"'action':\s*'([^']*)'", content) if "'action'" in content else self._match(r'"action":\s*"([^"]*)"', content)
code_content_value = self._match(r"'code_content':\s*'([^']*)'", content) if "'code_content'" in content else self._match(r'"code_content":\s*"([^"]*)"', content)
filename_value = self._match(r"'code_filename':\s*'([^']*)'", content) if "'code_filename'" in content else self._match(r'"code_filename":\s*"([^"]*)"', content)
tool_params_value = self._match(r"'tool_params':\s*(\{[^{}]*\})", content, do_json=True) if "'tool_params'" in content \
else self._match(r'"tool_params":\s*(\{[^{}]*\})', content, do_json=True)
tool_name_value = self._match(r"'tool_name':\s*'([^']*)'", content) if "'tool_name'" in content else self._match(r'"tool_name":\s*"([^"]*)"', content)
plans_value = self._match(r"'plans':\s*(\[.*?\])", content, do_search=False) if "'plans'" in content else self._match(r'"plans":\s*(\[.*?\])', content, do_search=False, )
# re解析
message.action_status = action_value or "default"
message.code_content = code_content_value
message.code_filename = filename_value
message.tool_params = tool_params_value
message.tool_name = tool_name_value
message.plans = plans_value
logger.debug(f"确认当前的action: {message.action_status}")
return message
def parser_spec_key(self, content, key, do_search=True, do_json=False) -> str:
''''''
key2pattern = {
"'action'": r"'action':\s*'([^']*)'", '"action"': r'"action":\s*"([^"]*)"',
"'code_content'": r"'code_content':\s*'([^']*)'", '"code_content"': r'"code_content":\s*"([^"]*)"',
"'code_filename'": r"'code_filename':\s*'([^']*)'", '"code_filename"': r'"code_filename":\s*"([^"]*)"',
"'tool_params'": r"'tool_params':\s*(\{[^{}]*\})", '"tool_params"': r'"tool_params":\s*(\{[^{}]*\})',
"'tool_name'": r"'tool_name':\s*'([^']*)'", '"tool_name"': r'"tool_name":\s*"([^"]*)"',
"'plans'": r"'plans':\s*(\[.*?\])", '"plans"': r'"plans":\s*(\[.*?\])',
"'content'": r"'content':\s*'([^']*)'", '"content"': r'"content":\s*"([^"]*)"',
}
#
output_message = check_message or output_message # 返回chain和checker的结果
output_message.input_query = query.input_query # chain和chain之间消息通信不改变问题
yield output_message, local_memory
s_json = self._parse_json(content)
try:
if s_json and key in s_json:
return str(s_json[key])
except:
pass
# def step_router(self, message: Message) -> Message:
# ''''''
# # message = self.parser(message)
# # logger.debug(f"message.action_status: {message.action_status}")
# if message.action_status == ActionStatus.CODING:
# message = self.code_step(message)
# elif message.action_status == ActionStatus.TOOL_USING:
# message = self.tool_step(message)
keystr = f"'{key}'" if f"'{key}'" in content else f'"{key}"'
return self._match(key2pattern.get(keystr, fr"'{key}':\s*'([^']*)'"), content, do_search=do_search, do_json=do_json)
def _match(self, pattern, s, do_search=True, do_json=False):
try:
if do_search:
match = re.search(pattern, s)
if match:
value = match.group(1).replace("\\n", "\n")
if do_json:
value = json.loads(value)
else:
value = None
else:
match = re.findall(pattern, s, re.DOTALL)
if match:
value = match[0]
if do_json:
value = json.loads(value)
else:
value = None
except Exception as e:
logger.warning(f"{traceback.format_exc()}")
# return message
# logger.debug(f"pattern: {pattern}, s: {s}, match: {match}")
return value
def _parse_json(self, s):
try:
pattern = r"```([^`]+)```"
match = re.findall(pattern, s)
if match:
return eval(match[0])
except:
pass
return None
# def code_step(self, message: Message) -> Message:
# '''execute code'''
# # logger.debug(f"message.role_content: {message.role_content}, message.code_content: {message.code_content}")
# code_answer = self.codebox.chat('```python\n{}```'.format(message.code_content))
# uid = str(uuid.uuid1())
# if code_answer.code_exe_type == "image/png":
# message.figures[uid] = code_answer.code_exe_response
# message.code_answer = f"\n观察: 执行代码后获得输出一张图片, 文件名为{uid}\n"
# message.observation = f"\n观察: 执行代码后获得输出一张图片, 文件名为{uid}\n"
# message.step_content += f"\n观察: 执行代码后获得输出一张图片, 文件名为{uid}\n"
# message.step_contents += [f"\n观察: 执行代码后获得输出一张图片, 文件名为{uid}\n"]
# message.role_content += f"\n执行代码后获得输出一张图片, 文件名为{uid}\n"
# else:
# message.code_answer = code_answer.code_exe_response
# message.observation = code_answer.code_exe_response
# message.step_content += f"\n观察: 执行代码后获得输出是 {code_answer.code_exe_response}\n"
# message.step_contents += [f"\n观察: 执行代码后获得输出是 {code_answer.code_exe_response}\n"]
# message.role_content += f"\n观察: 执行代码后获得输出是 {code_answer.code_exe_response}\n"
# logger.info(f"观察: {message.action_status}, {message.observation}")
# return message
def inherit_extrainfo(self, input_message: Message, output_message: Message):
output_message.db_docs = input_message.db_docs
output_message.search_docs = input_message.search_docs
output_message.code_docs = input_message.code_docs
output_message.figures.update(input_message.figures)
return output_message
# def tool_step(self, message: Message) -> Message:
# '''execute tool'''
# # logger.debug(f"message: {message.action_status}, {message.tool_name}, {message.tool_params}")
# tool_names = [tool.name for tool in message.tools]
# if message.tool_name not in tool_names:
# message.tool_answer = "不存在可以执行的tool"
# message.observation = "不存在可以执行的tool"
# message.role_content += f"\n观察: 不存在可以执行的tool\n"
# message.step_content += f"\n观察: 不存在可以执行的tool\n"
# message.step_contents += [f"\n观察: 不存在可以执行的tool\n"]
# for tool in message.tools:
# if tool.name == message.tool_name:
# tool_res = tool.func(**message.tool_params)
# message.tool_answer = tool_res
# message.observation = tool_res
# message.role_content += f"\n观察: {tool_res}\n"
# message.step_content += f"\n观察: {tool_res}\n"
# message.step_contents += [f"\n观察: {tool_res}\n"]
# return message
# def filter(self, message: Message, stop=None) -> Message:
# tool_params = self.parser_spec_key(message.role_content, "tool_params")
# code_content = self.parser_spec_key(message.role_content, "code_content")
# plan = self.parser_spec_key(message.role_content, "plan")
# plans = self.parser_spec_key(message.role_content, "plans", do_search=False)
# content = self.parser_spec_key(message.role_content, "content", do_search=False)
# # logger.debug(f"tool_params: {tool_params}, code_content: {code_content}, plan: {plan}, plans: {plans}, content: {content}")
# role_content = tool_params or code_content or plan or plans or content
# message.role_content = role_content or message.role_content
# return message
def get_memory(self, do_all_memory=True, content_key="role_content") -> Memory:
memory = self.global_memory if do_all_memory else self.local_memory
# def parser(self, message: Message) -> Message:
# ''''''
# content = message.role_content
# parser_keys = ["action", "code_content", "code_filename", "tool_params", "plans"]
# try:
# s_json = self._parse_json(content)
# message.action_status = s_json.get("action")
# message.code_content = s_json.get("code_content")
# message.tool_params = s_json.get("tool_params")
# message.tool_name = s_json.get("tool_name")
# message.code_filename = s_json.get("code_filename")
# message.plans = s_json.get("plans")
# # for parser_key in parser_keys:
# # message.action_status = content.get(parser_key)
# except Exception as e:
# # logger.warning(f"{traceback.format_exc()}")
# def parse_text_to_dict(text):
# # Define a regular expression pattern to capture the key and value
# main_pattern = r"\*\*(.+?):\*\*\s*(.*?)\s*(?=\*\*|$)"
# code_pattern = r'```python\n(.*?)```'
# # Use re.findall to find all main matches in the text
# main_matches = re.findall(main_pattern, text, re.DOTALL)
# # Convert main matches to a dictionary
# parsed_dict = {key.strip(): value.strip() for key, value in main_matches}
# # Search for the code block
# code_match = re.search(code_pattern, text, re.DOTALL)
# if code_match:
# # Add the code block to the dictionary
# parsed_dict['code'] = code_match.group(1).strip()
# return parsed_dict
# parsed_dict = parse_text_to_dict(content)
# action_value = parsed_dict.get('Action Status')
# if action_value:
# action_value = action_value.lower()
# logger.debug(f'action_value: {action_value}')
# # action_value = self._match(r"'action':\s*'([^']*)'", content) if "'action'" in content else self._match(r'"action":\s*"([^"]*)"', content)
# code_content_value = parsed_dict.get('code')
# # code_content_value = self._match(r"'code_content':\s*'([^']*)'", content) if "'code_content'" in content else self._match(r'"code_content":\s*"([^"]*)"', content)
# filename_value = self._match(r"'code_filename':\s*'([^']*)'", content) if "'code_filename'" in content else self._match(r'"code_filename":\s*"([^"]*)"', content)
# tool_params_value = self._match(r"'tool_params':\s*(\{[^{}]*\})", content, do_json=True) if "'tool_params'" in content \
# else self._match(r'"tool_params":\s*(\{[^{}]*\})', content, do_json=True)
# tool_name_value = self._match(r"'tool_name':\s*'([^']*)'", content) if "'tool_name'" in content else self._match(r'"tool_name":\s*"([^"]*)"', content)
# plans_value = self._match(r"'plans':\s*(\[.*?\])", content, do_search=False) if "'plans'" in content else self._match(r'"plans":\s*(\[.*?\])', content, do_search=False, )
# # re解析
# message.action_status = action_value or "default"
# message.code_content = code_content_value
# message.code_filename = filename_value
# message.tool_params = tool_params_value
# message.tool_name = tool_name_value
# message.plans = plans_value
# # logger.debug(f"确认当前的action: {message.action_status}")
# return message
# def parser_spec_key(self, content, key, do_search=True, do_json=False) -> str:
# ''''''
# key2pattern = {
# "'action'": r"'action':\s*'([^']*)'", '"action"': r'"action":\s*"([^"]*)"',
# "'code_content'": r"'code_content':\s*'([^']*)'", '"code_content"': r'"code_content":\s*"([^"]*)"',
# "'code_filename'": r"'code_filename':\s*'([^']*)'", '"code_filename"': r'"code_filename":\s*"([^"]*)"',
# "'tool_params'": r"'tool_params':\s*(\{[^{}]*\})", '"tool_params"': r'"tool_params":\s*(\{[^{}]*\})',
# "'tool_name'": r"'tool_name':\s*'([^']*)'", '"tool_name"': r'"tool_name":\s*"([^"]*)"',
# "'plans'": r"'plans':\s*(\[.*?\])", '"plans"': r'"plans":\s*(\[.*?\])',
# "'content'": r"'content':\s*'([^']*)'", '"content"': r'"content":\s*"([^"]*)"',
# }
# s_json = self._parse_json(content)
# try:
# if s_json and key in s_json:
# return str(s_json[key])
# except:
# pass
# keystr = f"'{key}'" if f"'{key}'" in content else f'"{key}"'
# return self._match(key2pattern.get(keystr, fr"'{key}':\s*'([^']*)'"), content, do_search=do_search, do_json=do_json)
# def _match(self, pattern, s, do_search=True, do_json=False):
# try:
# if do_search:
# match = re.search(pattern, s)
# if match:
# value = match.group(1).replace("\\n", "\n")
# if do_json:
# value = json.loads(value)
# else:
# value = None
# else:
# match = re.findall(pattern, s, re.DOTALL)
# if match:
# value = match[0]
# if do_json:
# value = json.loads(value)
# else:
# value = None
# except Exception as e:
# logger.warning(f"{traceback.format_exc()}")
# # logger.debug(f"pattern: {pattern}, s: {s}, match: {match}")
# return value
# def _parse_json(self, s):
# try:
# pattern = r"```([^`]+)```"
# match = re.findall(pattern, s)
# if match:
# return eval(match[0])
# except:
# pass
# return None
# def inherit_extrainfo(self, input_message: Message, output_message: Message):
# output_message.db_docs = input_message.db_docs
# output_message.search_docs = input_message.search_docs
# output_message.code_docs = input_message.code_docs
# output_message.origin_query = input_message.origin_query
# output_message.figures.update(input_message.figures)
# return output_message
def get_memory(self, content_key="role_content") -> Memory:
memory = self.global_memory
return memory.to_tuple_messages(content_key=content_key)
def get_memory_str(self, do_all_memory=True, content_key="role_content") -> Memory:
memory = self.global_memory if do_all_memory else self.local_memory
def get_memory_str(self, content_key="role_content") -> Memory:
memory = self.global_memory
# for i in memory.to_tuple_messages(content_key=content_key):
# logger.debug(f"{i}")
return "\n".join([": ".join(i) for i in memory.to_tuple_messages(content_key=content_key)])

28
dev_opsgpt/connector/chains/chains.py
View File

@ -1,28 +1,18 @@
from typing import List
from loguru import logger
import copy
from dev_opsgpt.connector.agents import BaseAgent
from .base_chain import BaseChain
from dev_opsgpt.connector.agents import BaseAgent, CheckAgent
from dev_opsgpt.connector.schema import (
Memory, Role, Message, ActionStatus, ChainConfig,
load_role_configs
)
class simpleChatChain(BaseChain):
def __init__(self, agents: List[BaseAgent], do_code_exec: bool = False) -> None:
super().__init__(agents, do_code_exec)
class toolChatChain(BaseChain):
def __init__(self, agents: List[BaseAgent], do_code_exec: bool = False) -> None:
super().__init__(agents, do_code_exec)
class dataAnalystChain(BaseChain):
def __init__(self, agents: List[BaseAgent], do_code_exec: bool = False) -> None:
super().__init__(agents, do_code_exec)
class plannerChain(BaseChain):
class ExecutorRefineChain(BaseChain):
def __init__(self, agents: List[BaseAgent], do_code_exec: bool = False) -> None:
super().__init__(agents, do_code_exec)

359
dev_opsgpt/connector/configs/agent_config.py
View File

@ -1,213 +1,33 @@
from enum import Enum
from .prompts import (
REACT_PROMPT_INPUT, CHECK_PROMPT_INPUT, EXECUTOR_PROMPT_INPUT, CONTEXT_PROMPT_INPUT, QUERY_CONTEXT_PROMPT_INPUT,PLAN_PROMPT_INPUT,
RECOGNIZE_INTENTION_PROMPT,
CHECKER_TEMPLATE_PROMPT,
CONV_SUMMARY_PROMPT,
QA_PROMPT, CODE_QA_PROMPT, QA_TEMPLATE_PROMPT,
EXECUTOR_TEMPLATE_PROMPT,
REFINE_TEMPLATE_PROMPT,
PLANNER_TEMPLATE_PROMPT, GENERAL_PLANNER_PROMPT, DATA_PLANNER_PROMPT, TOOL_PLANNER_PROMPT,
PRD_WRITER_METAGPT_PROMPT, DESIGN_WRITER_METAGPT_PROMPT, TASK_WRITER_METAGPT_PROMPT, CODE_WRITER_METAGPT_PROMPT,
REACT_TEMPLATE_PROMPT,
REACT_TOOL_PROMPT, REACT_CODE_PROMPT, REACT_TOOL_AND_CODE_PLANNER_PROMPT, REACT_TOOL_AND_CODE_PROMPT
)
class AgentType:
REACT = "ReactAgent"
EXECUTOR = "ExecutorAgent"
ONE_STEP = "BaseAgent"
DEFAULT = "BaseAgent"
REACT_TOOL_PROMPT = """尽可能地以有帮助和准确的方式回应人类。您可以使用以下工具:
{formatted_tools}
使用json blob来指定一个工具，提供一个action关键字（工具名称）和一个tool_params关键字（工具输入）。
有效的"action"值为："finished" 或 "tool_using" (使用工具来回答问题)
有效的"tool_name"值为：{tool_names}
请仅在每个$JSON_BLOB中提供一个action，如下所示：
```
{{{{
"action": $ACTION,
"tool_name": $TOOL_NAME
"tool_params": $INPUT
}}}}
```
按照以下格式进行回应：
问题：输入问题以回答
思考：考虑之前和之后的步骤
行动：
```
$JSON_BLOB
```
观察：行动结果
...（重复思考/行动/观察N次）
思考：我知道该如何回应
行动：
```
{{{{
"action": "finished",
"tool_name": "notool"
"tool_params": "最终返回答案给到用户"
}}}}
```
"""
REACT_PROMPT_INPUT = '''下面开始！记住根据问题进行返回需要生成的答案
问题: {query}'''
REACT_CODE_PROMPT = """尽可能地以有帮助和准确的方式回应人类，能够逐步编写可执行并打印变量的代码来解决问题
使用 JSON Blob 来指定一个返回的内容，提供一个 action（行动）和一个 code （生成代码）。
有效的 'action' 值为：'coding'(结合总结下述思维链过程编写下一步的可执行代码) or 'finished' （总结下述思维链过程可回答问题）。
在每个 $JSON_BLOB 中仅提供一个 action，如下所示：
```
{{{{'action': $ACTION,'code_content': $CODE}}}}
```
按照以下思维链格式进行回应：
问题：输入问题以回答
思考：考虑之前和之后的步骤
行动：
```
$JSON_BLOB
```
观察：行动结果
...（重复思考/行动/观察N次）
思考：我知道该如何回应
行动：
```
{{{{
"action": "finished",
"code_content": "总结上述思维链过程回答问题"
}}}}
```
"""
GENERAL_PLANNER_PROMPT = """你是一个通用计划拆解助手，将问题拆解问题成各个详细明确的步骤计划或直接回答问题，尽可能地以有帮助和准确的方式回应人类，
使用 JSON Blob 来指定一个返回的内容，提供一个 action（行动）和一个 plans （生成的计划）。
有效的 'action' 值为：'planning'(拆解计划) or 'only_answer' （不需要拆解问题即可直接回答问题）。
有效的 'plans' 值为: 一个任务列表，按顺序写出需要执行的计划
在每个 $JSON_BLOB 中仅提供一个 action，如下所示：
```
{{'action': 'planning', 'plans': [$PLAN1, $PLAN2, $PLAN3, ..., $PLANN], }}
或者
{{'action': 'only_answer', 'plans': "直接回答问题", }}
```
按照以下格式进行回应：
问题：输入问题以回答
行动：
```
$JSON_BLOB
```
"""
DATA_PLANNER_PROMPT = """你是一个数据分析助手，能够根据问题来制定一个详细明确的数据分析计划，尽可能地以有帮助和准确的方式回应人类，
使用 JSON Blob 来指定一个返回的内容，提供一个 action（行动）和一个 plans （生成的计划）。
有效的 'action' 值为：'planning'(拆解计划) or 'only_answer' （不需要拆解问题即可直接回答问题）。
有效的 'plans' 值为: 一份数据分析计划清单，按顺序排列，用文本表示
在每个 $JSON_BLOB 中仅提供一个 action，如下所示：
```
{{'action': 'planning', 'plans': '$PLAN1, $PLAN2, ..., $PLAN3' }}
```
按照以下格式进行回应：
问题：输入问题以回答
行动：
```
$JSON_BLOB
```
"""
TOOL_PLANNER_PROMPT = """你是一个工具使用过程的计划拆解助手，将问题拆解为一系列的工具使用计划，若没有可用工具则直接回答问题，尽可能地以有帮助和准确的方式回应人类，你可以使用以下工具:
{formatted_tools}
使用 JSON Blob 来指定一个返回的内容，提供一个 action（行动）和一个 plans （生成的计划）。
有效的 'action' 值为：'planning'(拆解计划) or 'only_answer' （不需要拆解问题即可直接回答问题）。
有效的 'plans' 值为: 一个任务列表，按顺序写出需要使用的工具和使用该工具的理由
在每个 $JSON_BLOB 中仅提供一个 action，如下两个示例所示：
```
{{'action': 'planning', 'plans': [$PLAN1, $PLAN2, $PLAN3, ..., $PLANN], }}
```
或者 若无法通过以上工具解决问题，则直接回答问题
```
{{'action': 'only_answer', 'plans': "直接回答问题", }}
```
按照以下格式进行回应：
问题：输入问题以回答
行动：
```
$JSON_BLOB
```
"""
RECOGNIZE_INTENTION_PROMPT = """你是一个任务决策助手，能够将理解用户意图并决策采取最合适的行动，尽可能地以有帮助和准确的方式回应人类，
使用 JSON Blob 来指定一个返回的内容，提供一个 action（行动）。
有效的 'action' 值为：'planning'(需要先进行拆解计划) or 'only_answer' （不需要拆解问题即可直接回答问题）or "tool_using" (使用工具来回答问题) or 'coding'(生成可执行的代码)。
在每个 $JSON_BLOB 中仅提供一个 action，如下所示：
```
{{'action': $ACTION}}
```
按照以下格式进行回应：
问题：输入问题以回答
行动：$ACTION
```
$JSON_BLOB
```
"""
CHECKER_PROMPT = """尽可能地以有帮助和准确的方式回应人类，判断问题是否得到解答，同时展现解答的过程和内容
使用 JSON Blob 来指定一个返回的内容，提供一个 action（行动）。
有效的 'action' 值为：'finished'(任务已经可以通过“背景信息”和“对话信息”回答问题) or 'continue' （“背景信息”和“对话信息”不足以回答问题）。
在每个 $JSON_BLOB 中仅提供一个 action，如下所示：
```
{{'action': $ACTION, 'content': '提取“背景信息”和“对话信息”中信息来回答问题'}}
```
按照以下格式进行回应：
问题：输入问题以回答
行动：$ACTION
```
$JSON_BLOB
```
"""
CONV_SUMMARY_PROMPT = """尽可能地以有帮助和准确的方式回应人类，根据“背景信息”中的有效信息回答问题，
使用 JSON Blob 来指定一个返回的内容，提供一个 action（行动）。
有效的 'action' 值为：'finished'(任务已经可以通过上下文信息可以回答) or 'continue' （根据背景信息回答问题）。
在每个 $JSON_BLOB 中仅提供一个 action，如下所示：
```
{{'action': $ACTION, 'content': '根据背景信息回答问题'}}
```
按照以下格式进行回应：
问题：输入问题以回答
行动：
```
$JSON_BLOB
```
"""
CONV_SUMMARY_PROMPT = """尽可能地以有帮助和准确的方式回应人类
根据“背景信息”中的有效信息回答问题，同时展现解答的过程和内容
若能根“背景信息”回答问题，则直接回答
否则，总结“背景信息”的内容
"""
QA_PROMPT = """根据已知信息，简洁和专业的来回答问题。如果无法从中得到答案，请说 “根据已知信息无法回答该问题”，不允许在答案中添加编造成分，答案请使用中文。
使用 JSON Blob 来指定一个返回的内容，提供一个 action（行动）。
有效的 'action' 值为：'finished'(任务已经可以通过上下文信息可以回答) or 'continue' （上下文信息不足以回答问题）。
在每个 $JSON_BLOB 中仅提供一个 action，如下所示：
```
{{'action': $ACTION, 'content': '总结对话内容'}}
```
按照以下格式进行回应：
问题：输入问题以回答
行动：$ACTION
```
$JSON_BLOB
```
"""
CODE_QA_PROMPT = """【指令】根据已知信息来回答问题"""
AGETN_CONFIGS = {
"checker": {
"role": {
"role_prompt": CHECKER_PROMPT,
"role_type": "ai",
"role_prompt": CHECKER_TEMPLATE_PROMPT,
"role_type": "assistant",
"role_name": "checker",
"role_desc": "",
"agent_type": "BaseAgent"
@ -220,7 +40,7 @@ AGETN_CONFIGS = {
"conv_summary": {
"role": {
"role_prompt": CONV_SUMMARY_PROMPT,
"role_type": "ai",
"role_type": "assistant",
"role_name": "conv_summary",
"role_desc": "",
"agent_type": "BaseAgent"
@ -232,8 +52,8 @@ AGETN_CONFIGS = {
},
"general_planner": {
"role": {
"role_prompt": GENERAL_PLANNER_PROMPT,
"role_type": "ai",
"role_prompt": PLANNER_TEMPLATE_PROMPT,
"role_type": "assistant",
"role_name": "general_planner",
"role_desc": "",
"agent_type": "BaseAgent"
@ -243,10 +63,37 @@ AGETN_CONFIGS = {
"do_doc_retrieval": False,
"do_tool_retrieval": False
},
"executor": {
"role": {
"role_prompt": EXECUTOR_TEMPLATE_PROMPT,
"role_type": "assistant",
"role_name": "executor",
"role_desc": "",
"agent_type": "ExecutorAgent",
},
"stop": "\n**Observation:**",
"chat_turn": 1,
"do_search": False,
"do_doc_retrieval": False,
"do_tool_retrieval": False
},
"base_refiner": {
"role": {
"role_prompt": REFINE_TEMPLATE_PROMPT,
"role_type": "assistant",
"role_name": "base_refiner",
"role_desc": "",
"agent_type": "BaseAgent"
},
"chat_turn": 1,
"do_search": False,
"do_doc_retrieval": False,
"do_tool_retrieval": False
},
"planner": {
"role": {
"role_prompt": DATA_PLANNER_PROMPT,
"role_type": "ai",
"role_type": "assistant",
"role_name": "planner",
"role_desc": "",
"agent_type": "BaseAgent"
@ -259,7 +106,7 @@ AGETN_CONFIGS = {
"intention_recognizer": {
"role": {
"role_prompt": RECOGNIZE_INTENTION_PROMPT,
"role_type": "ai",
"role_type": "assistant",
"role_name": "intention_recognizer",
"role_desc": "",
"agent_type": "BaseAgent"
@ -272,7 +119,7 @@ AGETN_CONFIGS = {
"tool_planner": {
"role": {
"role_prompt": TOOL_PLANNER_PROMPT,
"role_type": "ai",
"role_type": "assistant",
"role_name": "tool_planner",
"role_desc": "",
"agent_type": "BaseAgent"
@ -282,10 +129,37 @@ AGETN_CONFIGS = {
"do_doc_retrieval": False,
"do_tool_retrieval": False
},
"tool_and_code_react": {
"role": {
"role_prompt": REACT_TOOL_AND_CODE_PROMPT,
"role_type": "assistant",
"role_name": "tool_and_code_react",
"role_desc": "",
"agent_type": "ReactAgent",
},
"stop": "\n**Observation:**",
"chat_turn": 7,
"do_search": False,
"do_doc_retrieval": False,
"do_tool_retrieval": False
},
"tool_and_code_planner": {
"role": {
"role_prompt": REACT_TOOL_AND_CODE_PLANNER_PROMPT,
"role_type": "assistant",
"role_name": "tool_and_code_planner",
"role_desc": "",
"agent_type": "BaseAgent"
},
"chat_turn": 1,
"do_search": False,
"do_doc_retrieval": False,
"do_tool_retrieval": False
},
"tool_react": {
"role": {
"role_prompt": REACT_TOOL_PROMPT,
"role_type": "ai",
"role_type": "assistant",
"role_name": "tool_react",
"role_desc": "",
"agent_type": "ReactAgent"
@ -294,12 +168,12 @@ AGETN_CONFIGS = {
"do_search": False,
"do_doc_retrieval": False,
"do_tool_retrieval": False,
"stop": "观察"
"stop": "\n**Observation:**"
},
"code_react": {
"role": {
"role_prompt": REACT_CODE_PROMPT,
"role_type": "ai",
"role_type": "assistant",
"role_name": "code_react",
"role_desc": "",
"agent_type": "ReactAgent"
@ -308,25 +182,25 @@ AGETN_CONFIGS = {
"do_search": False,
"do_doc_retrieval": False,
"do_tool_retrieval": False,
"stop": "观察"
"stop": "\n**Observation:**"
},
"qaer": {
"role": {
"role_prompt": QA_PROMPT,
"role_type": "ai",
"role_prompt": QA_TEMPLATE_PROMPT,
"role_type": "assistant",
"role_name": "qaer",
"role_desc": "",
"agent_type": "BaseAgent"
},
"chat_turn": 1,
"do_search": False,
"do_doc_retrieval": True,
"do_doc_retrieval": False,
"do_tool_retrieval": False
},
"code_qaer": {
"role": {
"role_prompt": CODE_QA_PROMPT ,
"role_type": "ai",
"role_type": "assistant",
"role_name": "code_qaer",
"role_desc": "",
"agent_type": "BaseAgent"
@ -338,8 +212,8 @@ AGETN_CONFIGS = {
},
"searcher": {
"role": {
"role_prompt": QA_PROMPT,
"role_type": "ai",
"role_prompt": QA_TEMPLATE_PROMPT,
"role_type": "assistant",
"role_name": "searcher",
"role_desc": "",
"agent_type": "BaseAgent"
@ -349,62 +223,65 @@ AGETN_CONFIGS = {
"do_doc_retrieval": False,
"do_tool_retrieval": False
},
"answer": {
"metaGPT_PRD": {
"role": {
"role_prompt": "",
"role_type": "ai",
"role_name": "answer",
"role_prompt": PRD_WRITER_METAGPT_PROMPT,
"role_type": "assistant",
"role_name": "metaGPT_PRD",
"role_desc": "",
"agent_type": "BaseAgent"
},
"chat_turn": 1,
"do_search": False,
"do_doc_retrieval": False,
"do_tool_retrieval": False
"do_tool_retrieval": False,
"focus_agents": [],
"focus_message_keys": [],
},
"data_analyst": {
"metaGPT_DESIGN": {
"role": {
"role_prompt": """你是一个数据分析的代码开发助手，能够编写可执行的代码来完成相关的数据分析问题，使用 JSON Blob 来指定一个返回的内容，通过提供一个 action（行动）和一个 code （生成代码）和 一个 file_name （指定保存文件）。\
有效的 'action' 值为：'coding'(生成可执行的代码) or 'finished' （不生成代码并直接返回答案）。在每个 $JSON_BLOB 中仅提供一个 action，如下所示：\
```\n{{'action': $ACTION,'code_content': $CODE, 'code_filename': $FILE_NAME}}```\
下面开始！记住根据问题进行返回需要生成的答案，格式为 ```JSON_BLOB```""",
"role_type": "ai",
"role_name": "data_analyst",
"role_prompt": DESIGN_WRITER_METAGPT_PROMPT,
"role_type": "assistant",
"role_name": "metaGPT_DESIGN",
"role_desc": "",
"agent_type": "BaseAgent"
},
"chat_turn": 1,
"do_search": False,
"do_doc_retrieval": False,
"do_tool_retrieval": False
"do_tool_retrieval": False,
"focus_agents": ["metaGPT_PRD"],
"focus_message_keys": [],
},
"deveploer": {
"metaGPT_TASK": {
"role": {
"role_prompt": """你是一个代码开发助手，能够编写可执行的代码来完成问题，使用 JSON Blob 来指定一个返回的内容，通过提供一个 action（行动）和一个 code （生成代码）和 一个 file_name （指定保存文件）。\
有效的 'action' 值为：'coding'(生成可执行的代码) or 'finished' （不生成代码并直接返回答案）。在每个 $JSON_BLOB 中仅提供一个 action，如下所示：\
```\n{{'action': $ACTION,'code_content': $CODE, 'code_filename': $FILE_NAME}}```\
下面开始！记住根据问题进行返回需要生成的答案，格式为 ```JSON_BLOB```""",
"role_type": "ai",
"role_name": "deveploer",
"role_prompt": TASK_WRITER_METAGPT_PROMPT,
"role_type": "assistant",
"role_name": "metaGPT_TASK",
"role_desc": "",
"agent_type": "BaseAgent"
},
"chat_turn": 1,
"do_search": False,
"do_doc_retrieval": False,
"do_tool_retrieval": False
"do_tool_retrieval": False,
"focus_agents": ["metaGPT_DESIGN"],
"focus_message_keys": [],
},
"tester": {
"metaGPT_CODER": {
"role": {
"role_prompt": "你是一个QA问答的助手，能够尽可能准确地回答问题，下面请逐步思考问题并回答",
"role_type": "ai",
"role_name": "tester",
"role_prompt": CODE_WRITER_METAGPT_PROMPT,
"role_type": "assistant",
"role_name": "metaGPT_CODER",
"role_desc": "",
"agent_type": "BaseAgent"
"agent_type": "ExecutorAgent"
},
"chat_turn": 1,
"do_search": False,
"do_doc_retrieval": False,
"do_tool_retrieval": False
}
"do_tool_retrieval": False,
"focus_agents": ["metaGPT_DESIGN", "metaGPT_TASK"],
"focus_message_keys": [],
},
}

99
dev_opsgpt/connector/configs/agent_prompt/design_writer.yaml Normal file
View File

@ -0,0 +1,99 @@
You are a Architect, named Bob, your goal is Design a concise, usable, complete python system, and the constraint is Try to specify good open source tools as much as possible.
# Context
## Original Requirements:
Create a snake game.
## Product Goals:
Develop a highly addictive and engaging snake game.
Provide a user-friendly and intuitive user interface.
Implement various levels and challenges to keep the players entertained.
## User Stories:
As a user, I want to be able to control the snake's movement using arrow keys or touch gestures.
As a user, I want to see my score and progress displayed on the screen.
As a user, I want to be able to pause and resume the game at any time.
As a user, I want to be challenged with different obstacles and levels as I progress.
As a user, I want to have the option to compete with other players and compare my scores.
## Competitive Analysis:
Python Snake Game: A simple snake game implemented in Python with basic features and limited levels.
Snake.io: A multiplayer online snake game with competitive gameplay and high engagement.
Slither.io: Another multiplayer online snake game with a larger player base and addictive gameplay.
Snake Zone: A mobile snake game with various power-ups and challenges.
Snake Mania: A classic snake game with modern graphics and smooth controls.
Snake Rush: A fast-paced snake game with time-limited challenges.
Snake Master: A snake game with unique themes and customizable snakes.
## Requirement Analysis:
The product should be a highly addictive and engaging snake game with a user-friendly interface. It should provide various levels and challenges to keep the players entertained. The game should have smooth controls and allow the users to compete with each other.
## Requirement Pool:
```
[
["Implement different levels with increasing difficulty", "P0"],
["Allow users to control the snake using arrow keys or touch gestures", "P0"],
["Display the score and progress on the screen", "P1"],
["Provide an option to pause and resume the game", "P1"],
["Integrate leaderboards to enable competition among players", "P2"]
]
```
## UI Design draft:
The game will have a simple and clean interface. The main screen will display the snake, obstacles, and the score. The snake's movement can be controlled using arrow keys or touch gestures. There will be buttons to pause and resume the game. The level and difficulty will be indicated on the screen. The design will have a modern and visually appealing style with smooth animations.
## Anything UNCLEAR:
There are no unclear points.
## Format example
---
## Implementation approach
We will ...
## Python package name
```python
"snake_game"
```
## File list
```python
[
"main.py",
]
```
## Data structures and interface definitions
```mermaid
classDiagram
class Game{
+int score
}
...
Game "1" -- "1" Food: has
```
## Program call flow
```mermaid
sequenceDiagram
participant M as Main
...
G->>M: end game
```
## Anything UNCLEAR
The requirement is clear to me.
---
-----
Role: You are an architect; the goal is to design a SOTA PEP8-compliant python system; make the best use of good open source tools
Requirement: Fill in the following missing information based on the context, note that all sections are response with code form separately
Max Output: 8192 chars or 2048 tokens. Try to use them up.
Attention: Use '##' to split sections, not '#', and '## <SECTION_NAME>' SHOULD WRITE BEFORE the code and triple quote.
## Implementation approach: Provide as Plain text. Analyze the difficult points of the requirements, select the appropriate open-source framework.
## Python package name: Provide as Python str with python triple quoto, concise and clear, characters only use a combination of all lowercase and underscores
## File list: Provided as Python list[str], the list of ONLY REQUIRED files needed to write the program(LESS IS MORE!). Only need relative paths, comply with PEP8 standards. ALWAYS write a main.py or app.py here
## Data structures and interface definitions: Use mermaid classDiagram code syntax, including classes (INCLUDING __init__ method) and functions (with type annotations), CLEARLY MARK the RELATIONSHIPS between classes, and comply with PEP8 standards. The data structures SHOULD BE VERY DETAILED and the API should be comprehensive with a complete design.
## Program call flow: Use sequenceDiagram code syntax, COMPLETE and VERY DETAILED, using CLASSES AND API DEFINED ABOVE accurately, covering the CRUD AND INIT of each object, SYNTAX MUST BE CORRECT.
## Anything UNCLEAR: Provide as Plain text. Make clear here.

101
dev_opsgpt/connector/configs/agent_prompt/prd_writer.yaml Normal file
View File

@ -0,0 +1,101 @@
You are a Product Manager, named Alice, your goal is Efficiently create a successful product, and the constraint is .
# Context
## Original Requirements
Create a snake game.
## Search Information
### Search Results
### Search Summary
## mermaid quadrantChart code syntax example. DONT USE QUOTO IN CODE DUE TO INVALID SYNTAX. Replace the <Campain X> with REAL COMPETITOR NAME
```mermaid
quadrantChart
title Reach and engagement of campaigns
x-axis Low Reach --> High Reach
y-axis Low Engagement --> High Engagement
quadrant-1 We should expand
quadrant-2 Need to promote
quadrant-3 Re-evaluate
quadrant-4 May be improved
"Campaign: A": [0.3, 0.6]
"Campaign B": [0.45, 0.23]
"Campaign C": [0.57, 0.69]
"Campaign D": [0.78, 0.34]
"Campaign E": [0.40, 0.34]
"Campaign F": [0.35, 0.78]
"Our Target Product": [0.5, 0.6]
```
## Format example
---
## Original Requirements
The boss ...
## Product Goals
```python
[
"Create a ...",
]
```
## User Stories
```python
[
"As a user, ...",
]
```
## Competitive Analysis
```python
[
"Python Snake Game: ...",
]
```
## Competitive Quadrant Chart
```mermaid
quadrantChart
title Reach and engagement of campaigns
...
"Our Target Product": [0.6, 0.7]
```
## Requirement Analysis
The product should be a ...
## Requirement Pool
```python
[
["End game ...", "P0"]
]
```
## UI Design draft
Give a basic function description, and a draft
## Anything UNCLEAR
There are no unclear points.
---
-----
Role: You are a professional product manager; the goal is to design a concise, usable, efficient product
Requirements: According to the context, fill in the following missing information, note that each sections are returned in Python code triple quote form seperatedly. If the requirements are unclear, ensure minimum viability and avoid excessive design
ATTENTION: Use '##' to SPLIT SECTIONS, not '#'. AND '## <SECTION_NAME>' SHOULD WRITE BEFORE the code and triple quote. Output carefully referenced "Format example" in format.
## Original Requirements: Provide as Plain text, place the polished complete original requirements here
## Product Goals: Provided as Python list[str], up to 3 clear, orthogonal product goals. If the requirement itself is simple, the goal should also be simple
## User Stories: Provided as Python list[str], up to 5 scenario-based user stories, If the requirement itself is simple, the user stories should also be less
## Competitive Analysis: Provided as Python list[str], up to 7 competitive product analyses, consider as similar competitors as possible
## Competitive Quadrant Chart: Use mermaid quadrantChart code syntax. up to 14 competitive products. Translation: Distribute these competitor scores evenly between 0 and 1, trying to conform to a normal distribution centered around 0.5 as much as possible.
## Requirement Analysis: Provide as Plain text. Be simple. LESS IS MORE. Make your requirements less dumb. Delete the parts unnessasery.
## Requirement Pool: Provided as Python list[list[str], the parameters are requirement description, priority(P0/P1/P2), respectively, comply with PEP standards; no more than 5 requirements and consider to make its difficulty lower
## UI Design draft: Provide as Plain text. Be simple. Describe the elements and functions, also provide a simple style description and layout description.
## Anything UNCLEAR: Provide as Plain text. Make clear here.

177
dev_opsgpt/connector/configs/agent_prompt/review_code.yaml Normal file
View File

@ -0,0 +1,177 @@
NOTICE
Role: You are a professional software engineer, and your main task is to review the code. You need to ensure that the code conforms to the PEP8 standards, is elegantly designed and modularized, easy to read and maintain, and is written in Python 3.9 (or in another programming language).
ATTENTION: Use '##' to SPLIT SECTIONS, not '#'. Output format carefully referenced "Format example".
## Code Review: Based on the following context and code, and following the check list, Provide key, clear, concise, and specific code modification suggestions, up to 5.
```
1. Check 0: Is the code implemented as per the requirements?
2. Check 1: Are there any issues with the code logic?
3. Check 2: Does the existing code follow the "Data structures and interface definitions"?
4. Check 3: Is there a function in the code that is omitted or not fully implemented that needs to be implemented?
5. Check 4: Does the code have unnecessary or lack dependencies?
```
## Rewrite Code: point.py Base on "Code Review" and the source code, rewrite code with triple quotes. Do your utmost to optimize THIS SINGLE FILE.
-----
# Context
## Implementation approach
For the snake game, we can use the Pygame library, which is an open-source and easy-to-use library for game development in Python. Pygame provides a simple and efficient way to handle graphics, sound, and user input, making it suitable for developing a snake game.
## Python package name
```
"snake_game"
```
## File list
````
[
"main.py",
]
```
## Data structures and interface definitions
```
classDiagram
class Game:
-int score
-bool paused
+__init__()
+start_game()
+handle_input(key: int)
+update_game()
+draw_game()
+game_over()
class Snake:
-list[Point] body
-Point dir
-bool alive
+__init__(start_pos: Point)
+move()
+change_direction(dir: Point)
+grow()
+get_head() -> Point
+get_body() -> list[Point]
+is_alive() -> bool
class Point:
-int x
-int y
+__init__(x: int, y: int)
+set_coordinate(x: int, y: int)
+get_coordinate() -> tuple[int, int]
class Food:
-Point pos
-bool active
+__init__()
+generate_new_food()
+get_position() -> Point
+is_active() -> bool
Game "1" -- "1" Snake: contains
Game "1" -- "1" Food: has
```
## Program call flow
```
sequenceDiagram
participant M as Main
participant G as Game
participant S as Snake
participant F as Food
M->>G: Start game
G->>G: Initialize game
loop
M->>G: Handle user input
G->>S: Handle input
G->>F: Check if snake eats food
G->>S: Update snake movement
G->>G: Check game over condition
G->>G: Update score
G->>G: Draw game
M->>G: Update display
end
G->>G: Game over
```
## Required Python third-party packages
```
"""
pygame==2.0.1
"""
```
## Required Other language third-party packages
```
"""
No third-party packages required for other languages.
"""
```
## Logic Analysis
```
[
["main.py", "Main"],
["game.py", "Game"],
["snake.py", "Snake"],
["point.py", "Point"],
["food.py", "Food"]
]
```
## Task list
```
[
"point.py",
"food.py",
"snake.py",
"game.py",
"main.py"
]
```
## Shared Knowledge
```
"""
The 'point.py' module contains the implementation of the Point class, which represents a point in a 2D coordinate system.
The 'food.py' module contains the implementation of the Food class, which represents the food in the game.
The 'snake.py' module contains the implementation of the Snake class, which represents the snake in the game.
The 'game.py' module contains the implementation of the Game class, which manages the game logic.
The 'main.py' module is the entry point of the application and starts the game.
"""
```
## Anything UNCLEAR
We need to clarify the main entry point of the application and ensure that all required third-party libraries are properly initialized.
## Code: point.py
```
class Point:
def __init__(self, x: int, y: int):
self.x = x
self.y = y
def set_coordinate(self, x: int, y: int):
self.x = x
self.y = y
def get_coordinate(self) -> tuple[int, int]:
return self.x, self.y
```
-----
## Format example
-----
## Code Review
1. The code ...
2. ...
3. ...
4. ...
5. ...
## Rewrite Code: point.py
```python
## point.py
...
```
-----

148
dev_opsgpt/connector/configs/agent_prompt/task_write.yaml Normal file
View File

@ -0,0 +1,148 @@
You are a Project Manager, named Eve, your goal isImprove team efficiency and deliver with quality and quantity, and the constraint is .
# Context
## Implementation approach
For the snake game, we can use the Pygame library, which is an open-source and easy-to-use library for game development in Python. Pygame provides a simple and efficient way to handle graphics, sound, and user input, making it suitable for developing a snake game.
## Python package name
```
"snake_game"
```
## File list
````
[
"main.py",
"game.py",
"snake.py",
"food.py"
]
```
## Data structures and interface definitions
```
classDiagram
class Game{
-int score
-bool game_over
+start_game() : void
+end_game() : void
+update() : void
+draw() : void
+handle_events() : void
}
class Snake{
-list[Tuple[int, int]] body
-Tuple[int, int] direction
+move() : void
+change_direction(direction: Tuple[int, int]) : void
+is_collision() : bool
+grow() : void
+draw() : void
}
class Food{
-Tuple[int, int] position
+generate() : void
+draw() : void
}
class Main{
-Game game
+run() : void
}
Game "1" -- "1" Snake: contains
Game "1" -- "1" Food: has
Main "1" -- "1" Game: has
```
## Program call flow
```
sequenceDiagram
participant M as Main
participant G as Game
participant S as Snake
participant F as Food
M->G: run()
G->G: start_game()
G->G: handle_events()
G->G: update()
G->G: draw()
G->G: end_game()
G->S: move()
S->S: change_direction()
S->S: is_collision()
S->S: grow()
S->S: draw()
G->F: generate()
F->F: draw()
```
## Anything UNCLEAR
The design and implementation of the snake game are clear based on the given requirements.
## Format example
---
## Required Python third-party packages
```python
"""
flask==1.1.2
bcrypt==3.2.0
"""
```
## Required Other language third-party packages
```python
"""
No third-party ...
"""
```
## Full API spec
```python
"""
openapi: 3.0.0
...
description: A JSON object ...
"""
```
## Logic Analysis
```python
[
["game.py", "Contains ..."],
]
```
## Task list
```python
[
"game.py",
]
```
## Shared Knowledge
```python
"""
'game.py' contains ...
"""
```
## Anything UNCLEAR
We need ... how to start.
---
-----
Role: You are a project manager; the goal is to break down tasks according to PRD/technical design, give a task list, and analyze task dependencies to start with the prerequisite modules
Requirements: Based on the context, fill in the following missing information, note that all sections are returned in Python code triple quote form seperatedly. Here the granularity of the task is a file, if there are any missing files, you can supplement them
Attention: Use '##' to split sections, not '#', and '## <SECTION_NAME>' SHOULD WRITE BEFORE the code and triple quote.
## Required Python third-party packages: Provided in requirements.txt format
## Required Other language third-party packages: Provided in requirements.txt format
## Full API spec: Use OpenAPI 3.0. Describe all APIs that may be used by both frontend and backend.
## Logic Analysis: Provided as a Python list[list[str]. the first is filename, the second is class/method/function should be implemented in this file. Analyze the dependencies between the files, which work should be done first
## Task list: Provided as Python list[str]. Each str is a filename, the more at the beginning, the more it is a prerequisite dependency, should be done first
## Shared Knowledge: Anything that should be public like utils' functions, config's variables details that should make clear first.
## Anything UNCLEAR: Provide as Plain text. Make clear here. For example, don't forget a main entry. don't forget to init 3rd party libs.

147
dev_opsgpt/connector/configs/agent_prompt/write_code.yaml Normal file
View File

@ -0,0 +1,147 @@
NOTICE
Role: You are a professional engineer; the main goal is to write PEP8 compliant, elegant, modular, easy to read and maintain Python 3.9 code (but you can also use other programming language)
ATTENTION: Use '##' to SPLIT SECTIONS, not '#'. Output format carefully referenced "Format example".
## Code: snake.py Write code with triple quoto, based on the following list and context.
1. Do your best to implement THIS ONLY ONE FILE. ONLY USE EXISTING API. IF NO API, IMPLEMENT IT.
2. Requirement: Based on the context, implement one following code file, note to return only in code form, your code will be part of the entire project, so please implement complete, reliable, reusable code snippets
3. Attention1: If there is any setting, ALWAYS SET A DEFAULT VALUE, ALWAYS USE STRONG TYPE AND EXPLICIT VARIABLE.
4. Attention2: YOU MUST FOLLOW "Data structures and interface definitions". DONT CHANGE ANY DESIGN.
5. Think before writing: What should be implemented and provided in this document?
6. CAREFULLY CHECK THAT YOU DONT MISS ANY NECESSARY CLASS/FUNCTION IN THIS FILE.
7. Do not use public member functions that do not exist in your design.
-----
# Context
## Implementation approach
For the snake game, we can use the Pygame library, which is an open-source and easy-to-use library for game development in Python. Pygame provides a simple and efficient way to handle graphics, sound, and user input, making it suitable for developing a snake game.
## Python package name
```
"snake_game"
```
## File list
````
[
"main.py",
"game.py",
"snake.py",
"food.py"
]
```
## Data structures and interface definitions
```
classDiagram
class Game{
-int score
-bool game_over
+start_game() : void
+end_game() : void
+update() : void
+draw() : void
+handle_events() : void
}
class Snake{
-list[Tuple[int, int]] body
-Tuple[int, int] direction
+move() : void
+change_direction(direction: Tuple[int, int]) : void
+is_collision() : bool
+grow() : void
+draw() : void
}
class Food{
-Tuple[int, int] position
+generate() : void
+draw() : void
}
class Main{
-Game game
+run() : void
}
Game "1" -- "1" Snake: contains
Game "1" -- "1" Food: has
Main "1" -- "1" Game: has
```
## Program call flow
```
sequenceDiagram
participant M as Main
participant G as Game
participant S as Snake
participant F as Food
M->G: run()
G->G: start_game()
G->G: handle_events()
G->G: update()
G->G: draw()
G->G: end_game()
G->S: move()
S->S: change_direction()
S->S: is_collision()
S->S: grow()
S->S: draw()
G->F: generate()
F->F: draw()
```
## Anything UNCLEAR
The design and implementation of the snake game are clear based on the given requirements.
## Required Python third-party packages
```
"""
pygame==2.0.1
"""
```
## Required Other language third-party packages
```
"""
No third-party packages required for other languages.
"""
```
## Logic Analysis
```
[
["main.py", "Main"],
["game.py", "Game"],
["snake.py", "Snake"],
["food.py", "Food"]
]
```
## Task list
```
[
"snake.py",
"food.py",
"game.py",
"main.py"
]
```
## Shared Knowledge
```
"""
'game.py' contains the main logic for the snake game, including starting the game, handling user input, updating the game state, and drawing the game state.
'snake.py' contains the logic for the snake, including moving the snake, changing its direction, checking for collisions, growing the snake, and drawing the snake.
'food.py' contains the logic for the food, including generating a new food position and drawing the food.
'main.py' initializes the game and runs the game loop.
"""
```
## Anything UNCLEAR
We need to clarify the main entry point of the application and ensure that all required third-party libraries are properly initialized.
-----
## Format example
-----
## Code: snake.py
```python
## snake.py
...
```
-----

107
dev_opsgpt/connector/configs/chain_config.py
View File

@ -1,17 +1,16 @@
from enum import Enum
# from .prompts import PLANNER_TEMPLATE_PROMPT
CHAIN_CONFIGS = {
"chatChain": {
"chain_name": "chatChain",
"chain_type": "BaseChain",
"agents": ["answer"],
"agents": ["qaer"],
"chat_turn": 1,
"do_checker": False,
"clear_structure": "True",
"brainstorming": "False",
"gui_design": "True",
"git_management": "False",
"self_improve": "False"
"chain_prompt": ""
},
"docChatChain": {
"chain_name": "docChatChain",
@ -19,11 +18,7 @@ CHAIN_CONFIGS = {
"agents": ["qaer"],
"chat_turn": 1,
"do_checker": False,
"clear_structure": "True",
"brainstorming": "False",
"gui_design": "True",
"git_management": "False",
"self_improve": "False"
"chain_prompt": ""
},
"searchChatChain": {
"chain_name": "searchChatChain",
@ -31,11 +26,7 @@ CHAIN_CONFIGS = {
"agents": ["searcher"],
"chat_turn": 1,
"do_checker": False,
"clear_structure": "True",
"brainstorming": "False",
"gui_design": "True",
"git_management": "False",
"self_improve": "False"
"chain_prompt": ""
},
"codeChatChain": {
"chain_name": "codehChatChain",
@ -43,11 +34,7 @@ CHAIN_CONFIGS = {
"agents": ["code_qaer"],
"chat_turn": 1,
"do_checker": False,
"clear_structure": "True",
"brainstorming": "False",
"gui_design": "True",
"git_management": "False",
"self_improve": "False"
"chain_prompt": ""
},
"toolReactChain": {
"chain_name": "toolReactChain",
@ -55,34 +42,70 @@ CHAIN_CONFIGS = {
"agents": ["tool_planner", "tool_react"],
"chat_turn": 2,
"do_checker": True,
"clear_structure": "True",
"brainstorming": "False",
"gui_design": "True",
"git_management": "False",
"self_improve": "False"
"chain_prompt": ""
},
"codePlannerChain": {
"chain_name": "codePlannerChain",
"chain_type": "BaseChain",
"agents": ["planner"],
"chat_turn": 1,
"do_checker": True,
"chain_prompt": ""
},
"codeReactChain": {
"chain_name": "codeReactChain",
"chain_type": "BaseChain",
"agents": ["planner", "code_react"],
"chat_turn": 2,
"agents": ["code_react"],
"chat_turn": 6,
"do_checker": True,
"clear_structure": "True",
"brainstorming": "False",
"gui_design": "True",
"git_management": "False",
"self_improve": "False"
"chain_prompt": ""
},
"dataAnalystChain": {
"chain_name": "dataAnalystChain",
"codeToolPlanChain": {
"chain_name": "codeToolPlanChain",
"chain_type": "BaseChain",
"agents": ["planner", "code_react"],
"chat_turn": 2,
"agents": ["tool_and_code_planner"],
"chat_turn": 1,
"do_checker": False,
"chain_prompt": ""
},
"codeToolReactChain": {
"chain_name": "codeToolReactChain",
"chain_type": "BaseChain",
"agents": ["tool_and_code_react"],
"chat_turn": 3,
"do_checker": True,
"clear_structure": "True",
"brainstorming": "False",
"gui_design": "True",
"git_management": "False",
"self_improve": "False"
"chain_prompt": ""
},
"planChain": {
"chain_name": "planChain",
"chain_type": "BaseChain",
"agents": ["general_planner"],
"chat_turn": 1,
"do_checker": False,
"chain_prompt": ""
},
"executorChain": {
"chain_name": "executorChain",
"chain_type": "BaseChain",
"agents": ["executor"],
"chat_turn": 1,
"do_checker": True,
"chain_prompt": ""
},
"executorRefineChain": {
"chain_name": "executorRefineChain",
"chain_type": "BaseChain",
"agents": ["executor", "base_refiner"],
"chat_turn": 3,
"do_checker": True,
"chain_prompt": ""
},
"metagptChain": {
"chain_name": "metagptChain",
"chain_type": "BaseChain",
"agents": ["metaGPT_PRD", "metaGPT_DESIGN", "metaGPT_TASK", "metaGPT_CODER"],
"chat_turn": 1,
"do_checker": False,
"chain_prompt": ""
},
}

37
dev_opsgpt/connector/configs/phase_config.py
View File

@ -55,9 +55,10 @@ PHASE_CONFIGS = {
"do_using_tool": True
},
"codeReactPhase": {
"phase_name": "codeReacttPhase",
"phase_name": "codeReactPhase",
"phase_type": "BasePhase",
"chains": ["codeReactChain"],
# "chains": ["codePlannerChain", "codeReactChain"],
"chains": ["planChain", "codeReactChain"],
"do_summary": False,
"do_search": False,
"do_doc_retrieval": False,
@ -65,15 +66,37 @@ PHASE_CONFIGS = {
"do_tool_retrieval": False,
"do_using_tool": False
},
"dataReactPhase": {
"phase_name": "dataReactPhase",
"codeToolReactPhase": {
"phase_name": "codeToolReactPhase",
"phase_type": "BasePhase",
"chains": ["dataAnalystChain"],
"do_summary": True,
"chains": ["codeToolPlanChain", "codeToolReactChain"],
"do_summary": False,
"do_search": False,
"do_doc_retrieval": False,
"do_code_retrieval": False,
"do_tool_retrieval": False,
"do_using_tool": True
},
"baseTaskPhase": {
"phase_name": "baseTaskPhase",
"phase_type": "BasePhase",
"chains": ["planChain", "executorChain"],
"do_summary": False,
"do_search": False,
"do_doc_retrieval": False,
"do_code_retrieval": False,
"do_tool_retrieval": False,
"do_using_tool": False
}
},
"metagpt_code_devlop": {
"phase_name": "metagpt_code_devlop",
"phase_type": "BasePhase",
"chains": ["metagptChain",],
"do_summary": False,
"do_search": False,
"do_doc_retrieval": False,
"do_code_retrieval": False,
"do_tool_retrieval": False,
"do_using_tool": False
},
}

38
dev_opsgpt/connector/configs/prompts/__init__.py Normal file
View File

@ -0,0 +1,38 @@
from .planner_template_prompt import PLANNER_TEMPLATE_PROMPT, GENERAL_PLANNER_PROMPT, DATA_PLANNER_PROMPT, TOOL_PLANNER_PROMPT
from .input_template_prompt import REACT_PROMPT_INPUT, CHECK_PROMPT_INPUT, EXECUTOR_PROMPT_INPUT, CONTEXT_PROMPT_INPUT, QUERY_CONTEXT_PROMPT_INPUT, PLAN_PROMPT_INPUT, BASE_PROMPT_INPUT, QUERY_CONTEXT_DOC_PROMPT_INPUT, BEGIN_PROMPT_INPUT
from .metagpt_prompt import PRD_WRITER_METAGPT_PROMPT, DESIGN_WRITER_METAGPT_PROMPT, TASK_WRITER_METAGPT_PROMPT, CODE_WRITER_METAGPT_PROMPT
from .intention_template_prompt import RECOGNIZE_INTENTION_PROMPT
from .checker_template_prompt import CHECKER_PROMPT, CHECKER_TEMPLATE_PROMPT
from .summary_template_prompt import CONV_SUMMARY_PROMPT
from .qa_template_prompt import QA_PROMPT, CODE_QA_PROMPT, QA_TEMPLATE_PROMPT
from .executor_template_prompt import EXECUTOR_TEMPLATE_PROMPT
from .refine_template_prompt import REFINE_TEMPLATE_PROMPT
from .react_template_prompt import REACT_TEMPLATE_PROMPT
from .react_code_prompt import REACT_CODE_PROMPT
from .react_tool_prompt import REACT_TOOL_PROMPT
from .react_tool_code_prompt import REACT_TOOL_AND_CODE_PROMPT
from .react_tool_code_planner_prompt import REACT_TOOL_AND_CODE_PLANNER_PROMPT
__all__ = [
"REACT_PROMPT_INPUT", "CHECK_PROMPT_INPUT", "EXECUTOR_PROMPT_INPUT", "CONTEXT_PROMPT_INPUT", "QUERY_CONTEXT_PROMPT_INPUT", "PLAN_PROMPT_INPUT", "BASE_PROMPT_INPUT", "QUERY_CONTEXT_DOC_PROMPT_INPUT", "BEGIN_PROMPT_INPUT",
"RECOGNIZE_INTENTION_PROMPT",
"PRD_WRITER_METAGPT_PROMPT", "DESIGN_WRITER_METAGPT_PROMPT", "TASK_WRITER_METAGPT_PROMPT", "CODE_WRITER_METAGPT_PROMPT",
"CHECKER_PROMPT", "CHECKER_TEMPLATE_PROMPT",
"CONV_SUMMARY_PROMPT",
"QA_PROMPT", "CODE_QA_PROMPT", "QA_TEMPLATE_PROMPT",
"EXECUTOR_TEMPLATE_PROMPT",
"REFINE_TEMPLATE_PROMPT",
"PLANNER_TEMPLATE_PROMPT", "GENERAL_PLANNER_PROMPT", "DATA_PLANNER_PROMPT", "TOOL_PLANNER_PROMPT",
"REACT_TEMPLATE_PROMPT",
"REACT_CODE_PROMPT", "REACT_TOOL_PROMPT", "REACT_TOOL_AND_CODE_PROMPT", "REACT_TOOL_AND_CODE_PLANNER_PROMPT"
]

37
dev_opsgpt/connector/configs/prompts/checker_template_prompt.py Normal file
View File

@ -0,0 +1,37 @@
CHECKER_TEMPLATE_PROMPT = """#### Checker Assistance Guidance
When users have completed a sequence of tasks or if there is clear evidence that no further actions are required, your role is to confirm the completion.
Your task is to assess the current situation based on the context and determine whether all objectives have been met.
Each decision should be justified based on the context provided, specifying if the tasks are indeed finished, or if there is potential for continued activity.
#### Input Format
**Origin Query:** the initial question or objective that the user wanted to achieve
**Context:** the current status and history of the tasks to determine if Origin Query has been achieved.
#### Response Output Format
**REASON:** Justify the decision of choosing 'finished' and 'continued' by evaluating the progress step by step.
Consider all relevant information. If the tasks were aimed at an ongoing process, assess whether it has reached a satisfactory conclusion.
**Action Status:** Set to 'finished' or 'continued'.
If it's 'finished', the context can answer the origin query.
If it's 'continued', the context cant answer the origin query.
"""
CHECKER_PROMPT = """尽可能地以有帮助和准确的方式回应人类，判断问题是否得到解答，同时展现解答的过程和内容。
用户的问题：{query}
使用 JSON Blob 来指定一个返回的内容，提供一个 action（行动）。
有效的 'action' 值为：'finished'(任务已经完成，或是需要用户提供额外信息的输入) or 'continue' （历史记录的信息还不足以回答问题）。
在每个 $JSON_BLOB 中仅提供一个 action，如下所示：
```
{{'content': '提取“背景信息”和“对话信息”中信息来回答问题', 'reason': '解释$ACTION的原因', 'action': $ACTION}}
```
按照以下格式进行回应：
问题：输入问题以回答
行动：
```
$JSON_BLOB
```
"""

33
dev_opsgpt/connector/configs/prompts/executor_template_prompt.py Normal file
View File

@ -0,0 +1,33 @@
EXECUTOR_TEMPLATE_PROMPT = """#### Writing Code Assistance Guidance
When users need help with coding, your role is to provide precise and effective guidance.
Write the code step by step, showing only the part necessary to solve the current problem.
Each reply should contain only the code required for the current step.
#### Response Process
**Question:** First, clarify the problem to be solved.
**Thoughts:** Based on the question and observations above, provide the plan for executing this step.
**Action Status:** Set to 'finished' or 'coding'. If it's 'finished', the next action is to provide the final answer to the original question. If it's 'coding', the next step is to write the code.
**Action:** Code according to your thoughts. Use this format for code:
```python
# Write your code here
```
**Observation:** Check the results and effects of the executed code.
... (Repeat this Question/Thoughts/Action/Observation cycle as needed)
**Thoughts:** I now know the final answer
**Action Status:** Set to 'finished'
**Action:** The final answer to the original input question
"""

40
dev_opsgpt/connector/configs/prompts/input_template_prompt.py Normal file
View File

@ -0,0 +1,40 @@
BASE_PROMPT_INPUT = '''#### Begin!!!
'''
PLAN_PROMPT_INPUT = '''#### Begin!!!
**Question:** {query}
'''
REACT_PROMPT_INPUT = '''#### Begin!!!
{query}
'''
CONTEXT_PROMPT_INPUT = '''#### Begin!!!
**Context:** {context}
'''
QUERY_CONTEXT_DOC_PROMPT_INPUT = '''#### Begin!!!
**Origin Query:** {query}
**Context:** {context}
**DocInfos:** {DocInfos}
'''
QUERY_CONTEXT_PROMPT_INPUT = '''#### Begin!!!
**Origin Query:** {query}
**Context:** {context}
'''
EXECUTOR_PROMPT_INPUT = '''#### Begin!!!
{query}
'''
BEGIN_PROMPT_INPUT = '''#### Begin!!!
'''
CHECK_PROMPT_INPUT = '''下面是用户的原始问题：{query}'''

14
dev_opsgpt/connector/configs/prompts/intention_template_prompt.py Normal file
View File

@ -0,0 +1,14 @@
RECOGNIZE_INTENTION_PROMPT = """你是一个任务决策助手，能够将理解用户意图并决策采取最合适的行动，尽可能地以有帮助和准确的方式回应人类，
使用 JSON Blob 来指定一个返回的内容，提供一个 action（行动）。
有效的 'action' 值为：'planning'(需要先进行拆解计划) or 'only_answer' （不需要拆解问题即可直接回答问题）or "tool_using" (使用工具来回答问题) or 'coding'(生成可执行的代码)。
在每个 $JSON_BLOB 中仅提供一个 action，如下所示：
```
{{'action': $ACTION}}
```
按照以下格式进行回应：
问题：输入问题以回答
行动：$ACTION
```
$JSON_BLOB
```
"""

218
dev_opsgpt/connector/configs/prompts/metagpt_prompt.py Normal file
View File

@ -0,0 +1,218 @@
PRD_WRITER_METAGPT_PROMPT = """#### PRD Writer Assistance Guidance
You are a professional Product Manager, your goal is to design a concise, usable, efficient product.
According to the context, fill in the following missing information, note that each sections are returned in Python code triple quote form seperatedly.
If the Origin Query are unclear, ensure minimum viability and avoid excessive design.
ATTENTION: response carefully referenced "Response Output Format" in format.
#### Input Format
**Origin Query:** the initial question or objective that the user wanted to achieve
**Context:** the current status and history of the tasks to determine if Origin Query has been achieved.
#### Response Output Format
**Original Requirements:**
The boss ...
**Product Goals:**
```python
[
"Create a ...",
]
```
**User Stories:**
```python
[
"As a user, ...",
]
```
**Competitive Analysis:**
```python
[
"Python Snake Game: ...",
]
```
**Requirement Analysis:**
The product should be a ...
**Requirement Pool:**
```python
[
["End game ...", "P0"]
]
```
**UI Design draft:**
Give a basic function description, and a draft
**Anything UNCLEAR:**
There are no unclear points.'''
"""
DESIGN_WRITER_METAGPT_PROMPT = """#### PRD Writer Assistance Guidance
You are an architect; the goal is to design a SOTA PEP8-compliant python system; make the best use of good open source tools.
Fill in the following missing information based on the context, note that all sections are response with code form separately.
8192 chars or 2048 tokens. Try to use them up.
ATTENTION: response carefully referenced "Response Format" in format.
#### Input Format
**Origin Query:** the initial question or objective that the user wanted to achieve
**Context:** the current status and history of the tasks to determine if Origin Query has been achieved.
#### Response Format
**Implementation approach:**
Provide as Plain text. Analyze the difficult points of the requirements, select the appropriate open-source framework.
**Python package name:**
Provide as Python str with python triple quoto, concise and clear, characters only use a combination of all lowercase and underscores
```python
"snake_game"
```
**File list:**
Provided as Python list[str], the list of ONLY REQUIRED files needed to write the program(LESS IS MORE!). Only need relative paths, comply with PEP8 standards. ALWAYS write a main.py or app.py here
```python
[
"main.py",
...
]
```
**Data structures and interface definitions:**
Use mermaid classDiagram code syntax, including classes (INCLUDING __init__ method) and functions (with type annotations),
CLEARLY MARK the RELATIONSHIPS between classes, and comply with PEP8 standards. The data structures SHOULD BE VERY DETAILED and the API should be comprehensive with a complete design.
```mermaid
classDiagram
class Game {{
+int score
}}
...
Game "1" -- "1" Food: has
```
**Program call flow:**
Use sequenceDiagram code syntax, COMPLETE and VERY DETAILED, using CLASSES AND API DEFINED ABOVE accurately, covering the CRUD AND INIT of each object, SYNTAX MUST BE CORRECT.
```mermaid
sequenceDiagram
participant M as Main
...
G->>M: end game
```
**Anything UNCLEAR:**
Provide as Plain text. Make clear here.
"""
TASK_WRITER_METAGPT_PROMPT = """#### Task Plan Assistance Guidance
You are a project manager, the goal is to break down tasks according to PRD/technical design, give a task list, and analyze task dependencies to start with the prerequisite modules
Based on the context, fill in the following missing information, note that all sections are returned in Python code triple quote form seperatedly.
Here the granularity of the task is a file, if there are any missing files, you can supplement them
8192 chars or 2048 tokens. Try to use them up.
ATTENTION: response carefully referenced "Response Output Format" in format.
#### Input Format
**Origin Query:** the initial question or objective that the user wanted to achieve
**Context:** the current status and history of the tasks to determine if Origin Query has been achieved.
#### Response Output Format
**Required Python third-party packages:** Provided in requirements.txt format
```python
flask==1.1.2
bcrypt==3.2.0
...
```
**Required Other language third-party packages:** Provided in requirements.txt format
```python
No third-party ...
```
**Full API spec:** Use OpenAPI 3.0. Describe all APIs that may be used by both frontend and backend.
```python
openapi: 3.0.0
...
description: A JSON object ...
```
**Logic Analysis:** Provided as a Python list[list[str]. the first is filename, the second is class/method/function should be implemented in this file. Analyze the dependencies between the files, which work should be done first
```python
[
["game.py", "Contains ..."],
]
```
**PLAN:** Provided as Python list[str]. Each str is a filename, the more at the beginning, the more it is a prerequisite dependency, should be done first
```python
[
"game.py",
]
```
**Shared Knowledge:** Anything that should be public like utils' functions, config's variables details that should make clear first.
```python
'game.py' contains ...
```
**Anything UNCLEAR:**
Provide as Plain text. Make clear here. For example, don't forget a main entry. don't forget to init 3rd party libs.
"""
CODE_WRITER_METAGPT_PROMPT = """#### Code Writer Assistance Guidance
You are a professional engineer; the main goal is to write PEP8 compliant, elegant, modular, easy to read and maintain Python 3.9 code (but you can also use other programming language)
Code: Write code with triple quoto, based on the following list and context.
1. Do your best to implement THIS ONLY ONE FILE. ONLY USE EXISTING API. IF NO API, IMPLEMENT IT.
2. Requirement: Based on the context, implement one following code file, note to return only in code form, your code will be part of the entire project, so please implement complete, reliable, reusable code snippets
3. Attention1: If there is any setting, ALWAYS SET A DEFAULT VALUE, ALWAYS USE STRONG TYPE AND EXPLICIT VARIABLE.
4. Attention2: YOU MUST FOLLOW "Data structures and interface definitions". DONT CHANGE ANY DESIGN.
5. Think before writing: What should be implemented and provided in this document?
6. CAREFULLY CHECK THAT YOU DONT MISS ANY NECESSARY CLASS/FUNCTION IN THIS FILE.
7. Do not use public member functions that do not exist in your design.
8. **$key:** is Input format or Output format, *$key* is the context infomation, they are different.
8192 chars or 2048 tokens. Try to use them up.
ATTENTION: response carefully referenced "Response Output Format" in format **$key:**.
#### Input Format
**Origin Query:** the user's origin query you should to be solved
**Context:** the current status and history of the tasks to determine if Origin Query has been achieved.
**Question:** clarify the current question to be solved
#### Response Output Format
**Action Status:** Coding2File
**SaveFileName** construct a local file name based on Question and Context, such as
```python
$projectname/$filename.py
```
**Code:** Write your code here
```python
# Write your code here
```
"""

114
dev_opsgpt/connector/configs/prompts/planner_template_prompt.py Normal file
View File

@ -0,0 +1,114 @@
PLANNER_TEMPLATE_PROMPT = """#### Planner Assistance Guidance
When users need assistance with generating a sequence of achievable tasks, your role is to provide a coherent and continuous plan.
Design the plan step by step, ensuring each task builds on the completion of the previous one.
Each instruction should be actionable and directly follow from the outcome of the preceding step.
ATTENTION: response carefully referenced "Response Output Format" in format.
#### Input Format
**Question:** First, clarify the problem to be solved.
#### Response Output Format
**Action Status:** Set to 'finished' or 'planning'.
If it's 'finished', the PLAN is to provide the final answer to the original question.
If it's 'planning', the PLAN is to provide a Python list[str] of achievable tasks.
**PLAN:**
```list
[
"First, we should ...",
]
```
"""
TOOL_PLANNER_PROMPT = """#### Tool Planner Assistance Guidance
Helps user to break down a process of tool usage into a series of plans.
If there are no available tools, can directly answer the question.
Rrespond to humans in the most helpful and accurate way possible.
You can use the following tool: {formatted_tools}
#### Input Format
**Origin Query:** the initial question or objective that the user wanted to achieve
**Context:** the current status and history of the tasks to determine if Origin Query has been achieved.
#### Response Output Format
**Action Status:** Set to 'finished' or 'planning'. If it's 'finished', the PLAN is to provide the final answer to the original question. If it's 'planning', the PLAN is to provide a sequence of achievable tasks.
**PLAN:**
```python
[
"First, we should ...",
]
```
"""
GENERAL_PLANNER_PROMPT = """你是一个通用计划拆解助手，将问题拆解问题成各个详细明确的步骤计划或直接回答问题，尽可能地以有帮助和准确的方式回应人类，
使用 JSON Blob 来指定一个返回的内容，提供一个 action（行动）和一个 plans （生成的计划）。
有效的 'action' 值为：'planning'(拆解计划) or 'only_answer' （不需要拆解问题即可直接回答问题）。
有效的 'plans' 值为: 一个任务列表，按顺序写出需要执行的计划
在每个 $JSON_BLOB 中仅提供一个 action，如下所示：
```
{{'action': 'planning', 'plans': [$PLAN1, $PLAN2, $PLAN3, ..., $PLANN], }}
或者
{{'action': 'only_answer', 'plans': "直接回答问题", }}
```
按照以下格式进行回应：
问题：输入问题以回答
行动：
```
$JSON_BLOB
```
"""
DATA_PLANNER_PROMPT = """你是一个数据分析助手，能够根据问题来制定一个详细明确的数据分析计划，尽可能地以有帮助和准确的方式回应人类，
使用 JSON Blob 来指定一个返回的内容，提供一个 action（行动）和一个 plans （生成的计划）。
有效的 'action' 值为：'planning'(拆解计划) or 'only_answer' （不需要拆解问题即可直接回答问题）。
有效的 'plans' 值为: 一份数据分析计划清单，按顺序排列，用文本表示
在每个 $JSON_BLOB 中仅提供一个 action，如下所示：
```
{{'action': 'planning', 'plans': '$PLAN1, $PLAN2, ..., $PLAN3' }}
```
按照以下格式进行回应：
问题：输入问题以回答
行动：
```
$JSON_BLOB
```
"""
# TOOL_PLANNER_PROMPT = """你是一个工具使用过程的计划拆解助手，将问题拆解为一系列的工具使用计划，若没有可用工具则直接回答问题，尽可能地以有帮助和准确的方式回应人类，你可以使用以下工具:
# {formatted_tools}
# 使用 JSON Blob 来指定一个返回的内容，提供一个 action（行动）和一个 plans （生成的计划）。
# 有效的 'action' 值为：'planning'(拆解计划) or 'only_answer' （不需要拆解问题即可直接回答问题）。
# 有效的 'plans' 值为: 一个任务列表，按顺序写出需要使用的工具和使用该工具的理由
# 在每个 $JSON_BLOB 中仅提供一个 action，如下两个示例所示：
# ```
# {{'action': 'planning', 'plans': [$PLAN1, $PLAN2, $PLAN3, ..., $PLANN], }}
# ```
# 或者 若无法通过以上工具解决问题，则直接回答问题
# ```
# {{'action': 'only_answer', 'plans': "直接回答问题", }}
# ```
# 按照以下格式进行回应：
# 问题：输入问题以回答
# 行动：
# ```
# $JSON_BLOB
# ```
# """

52
dev_opsgpt/connector/configs/prompts/qa_template_prompt.py Normal file
View File

@ -0,0 +1,52 @@
QA_TEMPLATE_PROMPT = """#### Question Answer Assistance Guidance
Based on the information provided, please answer the origin query concisely and professionally.
If the answer cannot be derived from the given Context and DocInfos, please say 'The question cannot be answered based on the information provided' and do not add any fabricated elements to the answer.
Attention: Follow the input format and response output format
#### Input Format
**Origin Query:** the initial question or objective that the user wanted to achieve
**Context:** the current status and history of the tasks to determine if Origin Query has been achieved.
**DocInfos:**: the relevant doc information or code information, if this is empty, don't refer to this.
#### Response Output Format
**Answer:** Response to the user's origin query based on Context and DocInfos. If DocInfos is empty, you can ignore it.
"""
CODE_QA_PROMPT = """#### Code Answer Assistance Guidance
Based on the information provided, please answer the origin query concisely and professionally.
If the answer cannot be derived from the given Context and DocInfos, please say 'The question cannot be answered based on the information provided' and do not add any fabricated elements to the answer.
Attention: Follow the input format and response output format
#### Input Format
**Origin Query:** the initial question or objective that the user wanted to achieve
**DocInfos:**: the relevant doc information or code information, if this is empty, don't refer to this.
#### Response Output Format
**Answer:** Response to the user's origin query based on DocInfos. If DocInfos is empty, you can ignore it.
"""
QA_PROMPT = """根据已知信息，简洁和专业的来回答问题。如果无法从中得到答案，请说 “根据已知信息无法回答该问题”，不允许在答案中添加编造成分，答案请使用中文。
使用 JSON Blob 来指定一个返回的内容，提供一个 action（行动）。
有效的 'action' 值为：'finished'(任务已经可以通过上下文信息可以回答) or 'continue' （上下文信息不足以回答问题）。
在每个 $JSON_BLOB 中仅提供一个 action，如下所示：
```
{{'action': $ACTION, 'content': '总结对话内容'}}
```
按照以下格式进行回应：
问题：输入问题以回答
行动：$ACTION
```
$JSON_BLOB
```
"""
# CODE_QA_PROMPT = """【指令】根据已知信息来回答问"""

31
dev_opsgpt/connector/configs/prompts/react_code_prompt.py Normal file
View File

@ -0,0 +1,31 @@
REACT_CODE_PROMPT = """#### Writing Code Assistance Guidance
When users need help with coding, your role is to provide precise and effective guidance. Write the code step by step, showing only the part necessary to solve the current problem. Each reply should contain only the code required for the current step.
#### Response Process
**Question:** First, clarify the problem to be solved.
**Thoughts:** Based on the question and observations above, provide the plan for executing this step.
**Action Status:** Set to 'finished' or 'coding'. If it's 'finished', the next action is to provide the final answer to the original question. If it's 'coding', the next step is to write the code.
**Action:** Code according to your thoughts. (Please note that only the content printed out by the executed code can be observed in the subsequent observation.) Use this format for code:
```python
# Write your code here
```
**Observation:** Check the results and effects of the executed code.
... (Repeat this Thoughts/Action/Observation cycle as needed)
**Thoughts:** I now know the final answer
**Action Status:** Set to 'finished'
**Action:** The final answer to the original input question
"""

31
dev_opsgpt/connector/configs/prompts/react_template_prompt.py Normal file
View File

@ -0,0 +1,31 @@
REACT_TEMPLATE_PROMPT = """#### Writing Code Assistance Guidance
When users need help with coding, your role is to provide precise and effective guidance. Write the code step by step, showing only the part necessary to solve the current problem. Each reply should contain only the code required for the current step.
#### Response Process
**Question:** First, clarify the problem to be solved.
**Thoughts:** Based on the question and observations above, provide the plan for executing this step.
**Action Status:** Set to 'finished' or 'coding'. If it's 'finished', the next action is to provide the final answer to the original question. If it's 'coding', the next step is to write the code.
**Action:** Code according to your thoughts. Use this format for code:
```python
# Write your code here
```
**Observation:** Check the results and effects of the executed code.
... (Repeat this Thoughts/Action/Observation cycle as needed)
**Thoughts:** I now know the final answer
**Action Status:** Set to 'finished'
**Action:** The final answer to the original input question
"""

49
dev_opsgpt/connector/configs/prompts/react_tool_code_planner_prompt.py Normal file
View File

@ -0,0 +1,49 @@
REACT_TOOL_AND_CODE_PLANNER_PROMPT = """#### Tool and Code Sequence Breakdown Assistant
When users need assistance with deconstructing problems into a series of actionable plans using tools or code, your role is to provide a structured plan or a direct solution.
You may use the following tools:
{formatted_tools}
Depending on the user's query, the response will either be a plan detailing the use of tools and reasoning, or a direct answer if the problem does not require breaking down.
#### Input Format
**Origin Query:** user's query
#### Follow this Response Format
**Action Status:** Set to 'planning' to provide a sequence of tasks, or 'only_answer' to provide a direct response without a plan.
**Action:**
For planning:
```list
[
"First step of the plan using a specified tool or a outline plan for code...",
"Next step in the plan...",
// Continue with additional steps as necessary
]
```
Or, provide the direct answer.
"""
# REACT_TOOL_AND_CODE_PLANNER_PROMPT = """你是一个工具和代码使用过程的计划拆解助手，将问题拆解为一系列的工具使用计划，若没有可用工具则使用代码，尽可能地以有帮助和准确的方式回应人类，你可以使用以下工具:
# {formatted_tools}
# 使用 JSON Blob 来指定一个返回的内容，提供一个 action（行动）和一个 plans （生成的计划）。
# 有效的 'action' 值为：'planning'(拆解计划) or 'only_answer' （不需要拆解问题即可直接回答问题）。
# 有效的 'plans' 值为: 一个任务列表，按顺序写出需要使用的工具和使用该工具的理由
# 在每个 $JSON_BLOB 中仅提供一个 action，如下两个示例所示：
# ```
# {{'action': 'planning', 'plans': [$PLAN1, $PLAN2, $PLAN3, ..., $PLANN], }}
# ```
# 或者 若无法通过以上工具或者代码解决问题，则直接回答问题
# ```
# {{'action': 'only_answer', 'plans': "直接回答问题", }}
# ```
# 按照以下格式进行回应（$JSON_BLOB要求符合上述规定）：
# 问题：输入问题以回答
# 行动：
# ```
# $JSON_BLOB
# ```
# """

81
dev_opsgpt/connector/configs/prompts/react_tool_code_prompt.py Normal file
View File

@ -0,0 +1,81 @@
REACT_TOOL_AND_CODE_PROMPT = """#### Code and Tool Agent Assistance Guidance
When users need help with coding or using tools, your role is to provide precise and effective guidance. Use the tools provided if they can solve the problem, otherwise, write the code step by step, showing only the part necessary to solve the current problem. Each reply should contain only the guidance required for the current step either by tool usage or code.
#### Tool Infomation
You can use these tools:\n{formatted_tools}
Valid "tool_name" value:\n{tool_names}
#### Response Process
**Question:** Start by understanding the input question to be answered.
**Thoughts:** Considering the user's question, previously executed steps, and the plan, decide whether the current step requires the use of a tool or coding. Solve the problem step by step, only displaying the thought process necessary for the current step of solving the problem. If a tool can be used, provide its name and parameters. If coding is required, outline the plan for executing this step.
**Action Status:** finished, tool_using, or coding. (Choose one from these three statuses. If the task is done, set it to 'finished'. If using a tool, set it to 'tool_using'. If writing code, set it to 'coding'.)
**Action:**
If using a tool, output the following format to call the tool:
```json
{{
"tool_name": "$TOOL_NAME",
"tool_params": "$INPUT"
}}
```
If the problem cannot be solved with a tool at the moment, then proceed to solve the issue using code. Output the following format to execute the code:
```python
# Write your code here
```
**Observation:** Check the results and effects of the executed action.
... (Repeat this Thoughts/Action/Observation cycle as needed)
**Thoughts:** Conclude the final response to the input question.
**Action Status:** finished
**Action:** The final answer or guidance to the original input question.
"""
# REACT_TOOL_AND_CODE_PROMPT = """你是一个使用工具与代码的助手。
# 如果现有工具不足以完成整个任务，请不要添加不存在的工具，只使用现有工具完成可能的部分。
# 如果当前步骤不能使用工具完成，将由代码来完成。
# 有效的"action"值为："finished"（已经完成用户的任务） 、 "tool_using" (使用工具来回答问题) 或 'coding'(结合总结下述思维链过程编写下一步的可执行代码)。
# 尽可能地以有帮助和准确的方式回应人类，你可以使用以下工具:
# {formatted_tools}
# 如果现在的步骤可以用工具解决问题，请仅在每个$JSON_BLOB中提供一个action，如下所示：
# ```
# {{{{
# "action": $ACTION,
# "tool_name": $TOOL_NAME
# "tool_params": $INPUT
# }}}}
# ```
# 若当前无法通过工具解决问题，则使用代码解决问题
# 请仅在每个$JSON_BLOB中提供一个action，如下所示：
# ```
# {{{{'action': $ACTION,'code_content': $CODE}}}}
# ```
# 按照以下思维链格式进行回应（$JSON_BLOB要求符合上述规定）：
# 问题：输入问题以回答
# 思考：考虑之前和之后的步骤
# 行动：
# ```
# $JSON_BLOB
# ```
# 观察：行动结果
# ...（重复思考/行动/观察N次）
# 思考：我知道该如何回应
# 行动：
# ```
# $JSON_BLOB
# ```
# """

81
dev_opsgpt/connector/configs/prompts/react_tool_prompt.py Normal file
View File

@ -0,0 +1,81 @@
REACT_TOOL_PROMPT = """#### Tool Agent Assistance Guidance
When interacting with users, your role is to respond in a helpful and accurate manner using the tools available. Follow the steps below to ensure efficient and effective use of the tools.
Please note that all the tools you can use are listed below. You can only choose from these tools for use. If there are no suitable tools, please do not invent any tools. Just let the user know that you do not have suitable tools to use.
#### Tool List
you can use these tools:\n{formatted_tools}
valid "tool_name" value is:\n{tool_names}
#### Response Process
**Question:** Start by understanding the input question to be answered.
**Thoughts:** Based on the question and previous observations, plan the approach for using the tool effectively.
**Action Status:** Set to either 'finished' or 'tool_using'. If 'finished', provide the final response to the original question. If 'tool_using', proceed with using the specified tool.
**Action:** Use the tools by formatting the tool action in JSON. The format should be:
```json
{{
"tool_name": "$TOOL_NAME",
"tool_params": "$INPUT"
}}
```
**Observation:** Evaluate the outcome of the tool's usage.
... (Repeat this Thoughts/Action/Observation cycle as needed)
**Thoughts:** Determine the final response based on the results.
**Action Status:** Set to 'finished'
**Action:** Conclude with the final response to the original question in this format:
```json
{{
"tool_params": "Final response to be provided to the user",
"tool_name": "notool",
}}
```
"""
# REACT_TOOL_PROMPT = """尽可能地以有帮助和准确的方式回应人类。您可以使用以下工具:
# {formatted_tools}
# 使用json blob来指定一个工具，提供一个action关键字（工具名称）和一个tool_params关键字（工具输入）。
# 有效的"action"值为："finished" 或 "tool_using" (使用工具来回答问题)
# 有效的"tool_name"值为：{tool_names}
# 请仅在每个$JSON_BLOB中提供一个action，如下所示：
# ```
# {{{{
# "action": $ACTION,
# "tool_name": $TOOL_NAME,
# "tool_params": $INPUT
# }}}}
# ```
# 按照以下格式进行回应：
# 问题：输入问题以回答
# 思考：考虑之前和之后的步骤
# 行动：
# ```
# $JSON_BLOB
# ```
# 观察：行动结果
# ...（重复思考/行动/观察N次）
# 思考：我知道该如何回应
# 行动：
# ```
# {{{{
# "action": "finished",
# "tool_name": "notool",
# "tool_params": "最终返回答案给到用户"
# }}}}
# ```
# """

30
dev_opsgpt/connector/configs/prompts/refine_template_prompt.py Normal file
View File

@ -0,0 +1,30 @@
REFINE_TEMPLATE_PROMPT = """#### Refiner Assistance Guidance
When users have a sequence of tasks that require optimization or adjustment based on feedback from the context, your role is to refine the existing plan.
Your task is to identify where improvements can be made and provide a revised plan that is more efficient or effective.
Each instruction should be an enhancement of the existing plan and should specify the step from which the changes should be implemented.
#### Input Format
**Context:** Review the history of the plan and feedback to identify areas for improvement.
Take into consideration all feedback information from the current step. If there is no existing plan, generate a new one.
#### Response Output Format
**REASON:** think the reason of why choose 'finished', 'unchanged' or 'adjusted' step by step.
**Action Status:** Set to 'finished', 'unchanged' or 'adjusted'.
If it's 'finished', all tasks are accomplished, and no adjustments are needed, so PLAN_STEP is set to -1.
If it's 'unchanged', this PLAN has no problem, just set PLAN_STEP to CURRENT_STEP+1.
If it's 'adjusted', the PLAN is to provide an optimized version of the original plan.
**PLAN:**
```list
[
"First, we should ...",
]
```
**PLAN_STEP:** Set to the plan index from which the changes should start. Index range from 0 to n-1 or -1
If it's 'finished', the PLAN_STEP is -1. If it's 'adjusted', the PLAN_STEP is the index of the first revised task in the sequence.
"""

20
dev_opsgpt/connector/configs/prompts/summary_template_prompt.py Normal file
View File

@ -0,0 +1,20 @@
CONV_SUMMARY_PROMPT = """尽可能地以有帮助和准确的方式回应人类，根据“背景信息”中的有效信息回答问题，
使用 JSON Blob 来指定一个返回的内容，提供一个 action（行动）。
有效的 'action' 值为：'finished'(任务已经可以通过上下文信息可以回答) or 'continue' （根据背景信息回答问题）。
在每个 $JSON_BLOB 中仅提供一个 action，如下所示：
```
{{'action': $ACTION, 'content': '根据背景信息回答问题'}}
```
按照以下格式进行回应：
问题：输入问题以回答
行动：
```
$JSON_BLOB
```
"""
CONV_SUMMARY_PROMPT = """尽可能地以有帮助和准确的方式回应人类
根据“背景信息”中的有效信息回答问题，同时展现解答的过程和内容
若能根“背景信息”回答问题，则直接回答
否则，总结“背景信息”的内容
"""

364
dev_opsgpt/connector/message_process.py Normal file
View File

@ -0,0 +1,364 @@
import re, traceback, uuid, copy, json, os
from loguru import logger
from configs.server_config import SANDBOX_SERVER
from configs.model_config import JUPYTER_WORK_PATH
from dev_opsgpt.connector.schema import (
Memory, Task, Env, Role, Message, ActionStatus, CodeDoc, Doc
)
from dev_opsgpt.tools import DDGSTool, DocRetrieval, CodeRetrieval
from dev_opsgpt.sandbox import PyCodeBox, CodeBoxResponse
class MessageUtils:
def __init__(self, role: Role = None) -> None:
self.role = role
self.codebox = PyCodeBox(
remote_url=SANDBOX_SERVER["url"],
remote_ip=SANDBOX_SERVER["host"],
remote_port=SANDBOX_SERVER["port"],
token="mytoken",
do_code_exe=True,
do_remote=SANDBOX_SERVER["do_remote"],
do_check_net=False
)
def filter(self, message: Message, stop=None) -> Message:
tool_params = self.parser_spec_key(message.role_content, "tool_params")
code_content = self.parser_spec_key(message.role_content, "code_content")
plan = self.parser_spec_key(message.role_content, "plan")
plans = self.parser_spec_key(message.role_content, "plans", do_search=False)
content = self.parser_spec_key(message.role_content, "content", do_search=False)
# logger.debug(f"tool_params: {tool_params}, code_content: {code_content}, plan: {plan}, plans: {plans}, content: {content}")
role_content = tool_params or code_content or plan or plans or content
message.role_content = role_content or message.role_content
return message
def inherit_extrainfo(self, input_message: Message, output_message: Message):
output_message.db_docs = input_message.db_docs
output_message.search_docs = input_message.search_docs
output_message.code_docs = input_message.code_docs
output_message.figures.update(input_message.figures)
output_message.origin_query = input_message.origin_query
return output_message
def get_extrainfo_step(self, message: Message, do_search, do_doc_retrieval, do_code_retrieval, do_tool_retrieval) -> Message:
''''''
if do_search:
message = self.get_search_retrieval(message)
if do_doc_retrieval:
message = self.get_doc_retrieval(message)
if do_code_retrieval:
input_message = self.get_code_retrieval(message)
if do_tool_retrieval:
message = self.get_tool_retrieval(message)
return message
def get_search_retrieval(self, message: Message,) -> Message:
SEARCH_ENGINES = {"duckduckgo": DDGSTool}
search_docs = []
for idx, doc in enumerate(SEARCH_ENGINES["duckduckgo"].run(message.role_content, 3)):
doc.update({"index": idx})
search_docs.append(Doc(**doc))
message.search_docs = search_docs
return message
def get_doc_retrieval(self, message: Message) -> Message:
query = message.role_content
knowledge_basename = message.doc_engine_name
top_k = message.top_k
score_threshold = message.score_threshold
if knowledge_basename:
docs = DocRetrieval.run(query, knowledge_basename, top_k, score_threshold)
message.db_docs = [Doc(**doc) for doc in docs]
return message
def get_code_retrieval(self, message: Message) -> Message:
# DocRetrieval.run("langchain是什么", "DSADSAD")
query = message.input_query
code_engine_name = message.code_engine_name
history_node_list = message.history_node_list
code_docs = CodeRetrieval.run(code_engine_name, query, code_limit=message.top_k, history_node_list=history_node_list, search_type=message.cb_search_type)
message.code_docs = [CodeDoc(**doc) for doc in code_docs]
return message
def get_tool_retrieval(self, message: Message) -> Message:
return message
def step_router(self, message: Message) -> tuple[Message, ...]:
''''''
# message = self.parser(message)
# logger.debug(f"message.action_status: {message.action_status}")
observation_message = None
if message.action_status == ActionStatus.CODING:
message, observation_message = self.code_step(message)
elif message.action_status == ActionStatus.TOOL_USING:
message, observation_message = self.tool_step(message)
elif message.action_status == ActionStatus.CODING2FILE:
self.save_code2file(message)
return message, observation_message
def code_step(self, message: Message) -> Message:
'''execute code'''
# logger.debug(f"message.role_content: {message.role_content}, message.code_content: {message.code_content}")
code_answer = self.codebox.chat('```python\n{}```'.format(message.code_content))
code_prompt = f"The return error after executing the above code is {code_answer.code_exe_response}，need to recover" \
if code_answer.code_exe_type == "error" else f"The return information after executing the above code is {code_answer.code_exe_response}"
observation_message = Message(
role_name="observation",
role_type="func", #self.role.role_type,
role_content="",
step_content="",
input_query=message.code_content,
)
uid = str(uuid.uuid1())
if code_answer.code_exe_type == "image/png":
message.figures[uid] = code_answer.code_exe_response
message.code_answer = f"\n**Observation:**: The return figure name is {uid} after executing the above code.\n"
message.observation = f"\n**Observation:**: The return figure name is {uid} after executing the above code.\n"
message.step_content += f"\n**Observation:**: The return figure name is {uid} after executing the above code.\n"
message.step_contents += [f"\n**Observation:**:The return figure name is {uid} after executing the above code.\n"]
# message.role_content += f"\n**Observation:**:执行上述代码后生成一张图片, 图片名为{uid}\n"
observation_message.role_content = f"\n**Observation:**: The return figure name is {uid} after executing the above code.\n"
observation_message.parsed_output = {"Observation": f"The return figure name is {uid} after executing the above code."}
else:
message.code_answer = code_answer.code_exe_response
message.observation = code_answer.code_exe_response
message.step_content += f"\n**Observation:**: {code_prompt}\n"
message.step_contents += [f"\n**Observation:**: {code_prompt}\n"]
# message.role_content += f"\n**Observation:**: {code_prompt}\n"
observation_message.role_content = f"\n**Observation:**: {code_prompt}\n"
observation_message.parsed_output = {"Observation": code_prompt}
# logger.info(f"**Observation:** {message.action_status}, {message.observation}")
return message, observation_message
def tool_step(self, message: Message) -> Message:
'''execute tool'''
# logger.debug(f"{message}")
observation_message = Message(
role_name="observation",
role_type="function", #self.role.role_type,
role_content="\n**Observation:** there is no tool can execute\n" ,
step_content="",
input_query=str(message.tool_params),
tools=message.tools,
)
# logger.debug(f"message: {message.action_status}, {message.tool_name}, {message.tool_params}")
tool_names = [tool.name for tool in message.tools]
if message.tool_name not in tool_names:
message.tool_answer = "\n**Observation:** there is no tool can execute\n"
message.observation = "\n**Observation:** there is no tool can execute\n"
# message.role_content += f"\n**Observation:**: 不存在可以执行的tool\n"
message.step_content += f"\n**Observation:** there is no tool can execute\n"
message.step_contents += [f"\n**Observation:** there is no tool can execute\n"]
observation_message.role_content = f"\n**Observation:** there is no tool can execute\n"
observation_message.parsed_output = {"Observation": "there is no tool can execute\n"}
for tool in message.tools:
if tool.name == message.tool_name:
tool_res = tool.func(**message.tool_params.get("tool_params", {}))
logger.debug(f"tool_res {tool_res}")
message.tool_answer = tool_res
message.observation = tool_res
# message.role_content += f"\n**Observation:**: {tool_res}\n"
message.step_content += f"\n**Observation:** {tool_res}\n"
message.step_contents += [f"\n**Observation:** {tool_res}\n"]
observation_message.role_content = f"\n**Observation:** {tool_res}\n"
observation_message.parsed_output = {"Observation": tool_res}
break
# logger.info(f"**Observation:** {message.action_status}, {message.observation}")
return message, observation_message
def parser(self, message: Message) -> Message:
''''''
content = message.role_content
parser_keys = ["action", "code_content", "code_filename", "tool_params", "plans"]
try:
s_json = self._parse_json(content)
message.action_status = s_json.get("action")
message.code_content = s_json.get("code_content")
message.tool_params = s_json.get("tool_params")
message.tool_name = s_json.get("tool_name")
message.code_filename = s_json.get("code_filename")
message.plans = s_json.get("plans")
# for parser_key in parser_keys:
# message.action_status = content.get(parser_key)
except Exception as e:
# logger.warning(f"{traceback.format_exc()}")
def parse_text_to_dict(text):
# Define a regular expression pattern to capture the key and value
main_pattern = r"\*\*(.+?):\*\*\s*(.*?)\s*(?=\*\*|$)"
list_pattern = r'```python\n(.*?)```'
# Use re.findall to find all main matches in the text
main_matches = re.findall(main_pattern, text, re.DOTALL)
# Convert main matches to a dictionary
parsed_dict = {key.strip(): value.strip() for key, value in main_matches}
for k, v in parsed_dict.items():
for pattern in [list_pattern]:
if "PLAN" != k: continue
v = v.replace("```list", "```python")
match_value = re.search(pattern, v, re.DOTALL)
if match_value:
# Add the code block to the dictionary
parsed_dict[k] = eval(match_value.group(1).strip())
break
return parsed_dict
def extract_content_from_backticks(text):
code_blocks = []
lines = text.split('\n')
is_code_block = False
code_block = ''
language = ''
for line in lines:
if line.startswith('```') and not is_code_block:
is_code_block = True
language = line[3:]
code_block = ''
elif line.startswith('```') and is_code_block:
is_code_block = False
code_blocks.append({language.strip(): code_block.strip()})
elif is_code_block:
code_block += line + '\n'
return code_blocks
def parse_dict_to_dict(parsed_dict) -> dict:
code_pattern = r'```python\n(.*?)```'
tool_pattern = r'```json\n(.*?)```'
pattern_dict = {"code": code_pattern, "json": tool_pattern}
spec_parsed_dict = copy.deepcopy(parsed_dict)
for key, pattern in pattern_dict.items():
for k, text in parsed_dict.items():
# Search for the code block
if not isinstance(text, str): continue
_match = re.search(pattern, text, re.DOTALL)
if _match:
# Add the code block to the dictionary
try:
spec_parsed_dict[key] = json.loads(_match.group(1).strip())
except:
spec_parsed_dict[key] = _match.group(1).strip()
break
return spec_parsed_dict
parsed_dict = parse_text_to_dict(content)
spec_parsed_dict = parse_dict_to_dict(parsed_dict)
action_value = parsed_dict.get('Action Status')
if action_value:
action_value = action_value.lower()
logger.info(f'{message.role_name}: action_value: {action_value}')
# action_value = self._match(r"'action':\s*'([^']*)'", content) if "'action'" in content else self._match(r'"action":\s*"([^"]*)"', content)
code_content_value = spec_parsed_dict.get('code')
# code_content_value = self._match(r"'code_content':\s*'([^']*)'", content) if "'code_content'" in content else self._match(r'"code_content":\s*"([^"]*)"', content)
filename_value = self._match(r"'code_filename':\s*'([^']*)'", content) if "'code_filename'" in content else self._match(r'"code_filename":\s*"([^"]*)"', content)
if action_value == 'tool_using':
tool_params_value = spec_parsed_dict.get('json')
else:
tool_params_value = None
# tool_params_value = spec_parsed_dict.get('tool_params')
# tool_params_value = self._match(r"'tool_params':\s*(\{[^{}]*\})", content, do_json=True) if "'tool_params'" in content \
# else self._match(r'"tool_params":\s*(\{[^{}]*\})', content, do_json=True)
tool_name_value = self._match(r"'tool_name':\s*'([^']*)'", content) if "'tool_name'" in content else self._match(r'"tool_name":\s*"([^"]*)"', content)
plans_value = self._match(r"'plans':\s*(\[.*?\])", content, do_search=False) if "'plans'" in content else self._match(r'"plans":\s*(\[.*?\])', content, do_search=False, )
# re解析
message.action_status = action_value or "default"
message.code_content = code_content_value
message.code_filename = filename_value
message.tool_params = tool_params_value
message.tool_name = tool_name_value
message.plans = plans_value
message.parsed_output = parsed_dict
message.spec_parsed_output = spec_parsed_dict
# logger.debug(f"确认当前的action: {message.action_status}")
return message
def parser_spec_key(self, content, key, do_search=True, do_json=False) -> str:
''''''
key2pattern = {
"'action'": r"'action':\s*'([^']*)'", '"action"': r'"action":\s*"([^"]*)"',
"'code_content'": r"'code_content':\s*'([^']*)'", '"code_content"': r'"code_content":\s*"([^"]*)"',
"'code_filename'": r"'code_filename':\s*'([^']*)'", '"code_filename"': r'"code_filename":\s*"([^"]*)"',
"'tool_params'": r"'tool_params':\s*(\{[^{}]*\})", '"tool_params"': r'"tool_params":\s*(\{[^{}]*\})',
"'tool_name'": r"'tool_name':\s*'([^']*)'", '"tool_name"': r'"tool_name":\s*"([^"]*)"',
"'plans'": r"'plans':\s*(\[.*?\])", '"plans"': r'"plans":\s*(\[.*?\])',
"'content'": r"'content':\s*'([^']*)'", '"content"': r'"content":\s*"([^"]*)"',
}
s_json = self._parse_json(content)
try:
if s_json and key in s_json:
return str(s_json[key])
except:
pass
keystr = f"'{key}'" if f"'{key}'" in content else f'"{key}"'
return self._match(key2pattern.get(keystr, fr"'{key}':\s*'([^']*)'"), content, do_search=do_search, do_json=do_json)
def _match(self, pattern, s, do_search=True, do_json=False):
try:
if do_search:
match = re.search(pattern, s)
if match:
value = match.group(1).replace("\\n", "\n")
if do_json:
value = json.loads(value)
else:
value = None
else:
match = re.findall(pattern, s, re.DOTALL)
if match:
value = match[0]
if do_json:
value = json.loads(value)
else:
value = None
except Exception as e:
logger.warning(f"{traceback.format_exc()}")
# logger.debug(f"pattern: {pattern}, s: {s}, match: {match}")
return value
def _parse_json(self, s):
try:
pattern = r"```([^`]+)```"
match = re.findall(pattern, s)
if match:
return eval(match[0])
except:
pass
return None
def save_code2file(self, message: Message, project_dir=JUPYTER_WORK_PATH):
filename = message.parsed_output.get("SaveFileName")
code = message.spec_parsed_output.get("code")
for k, v in {"&gt;": ">", "&ge;": ">=", "&lt;": "<", "&le;": "<="}.items():
code = code.replace(k, v)
file_path = os.path.join(project_dir, filename)
if not os.path.exists(file_path):
os.makedirs(os.path.dirname(file_path), exist_ok=True)
with open(file_path, "w") as f:
f.write(code)

198
dev_opsgpt/connector/phase/base_phase.py
View File

@ -8,14 +8,13 @@ from loguru import logger
from dev_opsgpt.connector.agents import BaseAgent
from dev_opsgpt.connector.chains import BaseChain
from dev_opsgpt.tools.base_tool import BaseTools, Tool
from dev_opsgpt.connector.shcema.memory import Memory
from dev_opsgpt.connector.connector_schema import (
Task, Env, Role, Message, Doc, Docs, AgentConfig, ChainConfig, PhaseConfig, CodeDoc,
from dev_opsgpt.connector.schema import (
Memory, Task, Env, Role, Message, Doc, Docs, AgentConfig, ChainConfig, PhaseConfig, CodeDoc,
load_chain_configs, load_phase_configs, load_role_configs
)
from dev_opsgpt.connector.configs import AGETN_CONFIGS, CHAIN_CONFIGS, PHASE_CONFIGS
from dev_opsgpt.tools import DDGSTool, DocRetrieval, CodeRetrieval
from dev_opsgpt.connector.message_process import MessageUtils
role_configs = load_role_configs(AGETN_CONFIGS)
chain_configs = load_chain_configs(CHAIN_CONFIGS)
@ -56,57 +55,71 @@ class BasePhase:
chain_config = chain_config,
role_config = role_config,
)
self.message_utils = MessageUtils()
self.phase_name = phase_name
self.do_summary = do_summary
self.do_search = do_search
self.do_code_retrieval = do_code_retrieval
self.do_doc_retrieval = do_doc_retrieval
self.do_tool_retrieval = do_tool_retrieval
self.global_message = Memory([])
#
self.global_memory = Memory(messages=[])
# self.chain_message = Memory([])
self.phase_memory: List[Memory] = []
# memory_pool dont have specific order
self.memory_pool = Memory(messages=[])
def step(self, query: Message, history: Memory = None) -> Tuple[Message, Memory]:
def astep(self, query: Message, history: Memory = None) -> Tuple[Message, Memory]:
summary_message = None
chain_message = Memory([])
local_memory = Memory([])
chain_message = Memory(messages=[])
local_phase_memory = Memory(messages=[])
# do_search、do_doc_search、do_code_search
query = self.get_extrainfo_step(query)
query = self.message_utils.get_extrainfo_step(query, self.do_search, self.do_doc_retrieval, self.do_code_retrieval, self.do_tool_retrieval)
input_message = copy.deepcopy(query)
self.global_message.append(input_message)
self.global_memory.append(input_message)
local_phase_memory.append(input_message)
for chain in self.chains:
# chain can supply background and query to next chain
output_message, chain_memory = chain.step(input_message, history, background=chain_message)
output_message = self.inherit_extrainfo(input_message, output_message)
for output_message, local_chain_memory in chain.astep(input_message, history, background=chain_message, memory_pool=self.memory_pool):
# logger.debug(f"local_memory: {local_memory + chain_memory}")
yield output_message, local_phase_memory + local_chain_memory
output_message = self.message_utils.inherit_extrainfo(input_message, output_message)
input_message = output_message
logger.info(f"{chain.chainConfig.chain_name} phase_step: {output_message.role_content}")
# 这一段也有问题
self.global_memory.extend(local_chain_memory)
local_phase_memory.extend(local_chain_memory)
self.global_message.append(output_message)
local_memory.extend(chain_memory)
# whether use summary_llm
# whether to use summary_llm
if self.do_summary:
logger.info(f"{self.conv_summary_agent.role.role_name} input global memory: {self.global_message.to_str_messages(content_key='step_content')}")
logger.info(f"{self.conv_summary_agent.role.role_name} input global memory: {self.global_message.to_str_messages(content_key='role_content')}")
summary_message = self.conv_summary_agent.run(query, background=self.global_message)
logger.info(f"{self.conv_summary_agent.role.role_name} input global memory: {local_phase_memory.to_str_messages(content_key='step_content')}")
for summary_message in self.conv_summary_agent.arun(query, background=local_phase_memory, memory_pool=self.memory_pool):
pass
# summary_message = Message(**summary_message)
summary_message.role_name = chain.chainConfig.chain_name
summary_message = self.conv_summary_agent.parser(summary_message)
summary_message = self.conv_summary_agent.filter(summary_message)
summary_message = self.inherit_extrainfo(output_message, summary_message)
summary_message = self.conv_summary_agent.message_utils.parser(summary_message)
summary_message = self.conv_summary_agent.message_utils.filter(summary_message)
summary_message = self.message_utils.inherit_extrainfo(output_message, summary_message)
chain_message.append(summary_message)
message = summary_message or output_message
yield message, local_phase_memory
# 由于不会存在多轮chain执行，所以直接保留memory即可
for chain in self.chains:
self.phase_memory.append(chain.global_memory)
# TODO：local_memory缺少添加summary的过程
message = summary_message or output_message
message.role_name = self.phase_name
# message.db_docs = query.db_docs
# message.code_docs = query.code_docs
# message.search_docs = query.search_docs
return summary_message or output_message, local_memory
yield message, local_phase_memory
def step(self, query: Message, history: Memory = None) -> Tuple[Message, Memory]:
for message, local_phase_memory in self.astep(query, history=history):
pass
return message, local_phase_memory
def init_chains(self, phase_name, phase_config, chain_config,
role_config, task=None, memory=None) -> List[BaseChain]:
@ -118,25 +131,33 @@ class BasePhase:
chains = []
self.chain_module = importlib.import_module("dev_opsgpt.connector.chains")
self.agent_module = importlib.import_module("dev_opsgpt.connector.agents")
phase = phase_configs.get(phase_name)
for chain_name in phase.chains:
logger.info(f"chain_name: {chain_name}")
# chain_class = getattr(self.chain_module, chain_name)
logger.debug(f"{chain_configs.keys()}")
chain_config = chain_configs[chain_name]
agents = [
getattr(self.agent_module, role_configs[agent_name].role.agent_type)(
role_configs[agent_name].role,
phase = phase_configs.get(phase_name)
logger.info(f"start to init the phase, the phase_name is {phase_name}, it contains these chains such as {phase.chains}")
for chain_name in phase.chains:
# logger.debug(f"{chain_configs.keys()}")
chain_config = chain_configs[chain_name]
logger.info(f"start to init the chain, the chain_name is {chain_name}, it contains these agents such as {chain_config.agents}")
agents = []
for agent_name in chain_config.agents:
agent_config = role_configs[agent_name]
baseAgent: BaseAgent = getattr(self.agent_module, agent_config.role.agent_type)
base_agent = baseAgent(
agent_config.role,
task = task,
memory = memory,
chat_turn=role_configs[agent_name].chat_turn,
do_search = role_configs[agent_name].do_search,
do_doc_retrieval = role_configs[agent_name].do_doc_retrieval,
do_tool_retrieval = role_configs[agent_name].do_tool_retrieval,
chat_turn=agent_config.chat_turn,
do_search = agent_config.do_search,
do_doc_retrieval = agent_config.do_doc_retrieval,
do_tool_retrieval = agent_config.do_tool_retrieval,
stop= agent_config.stop,
focus_agents=agent_config.focus_agents,
focus_message_keys=agent_config.focus_message_keys,
)
for agent_name in chain_config.agents
]
agents.append(base_agent)
chain_instance = BaseChain(
chain_config, agents, chain_config.chat_turn,
do_checker=chain_configs[chain_name].do_checker,
@ -145,56 +166,57 @@ class BasePhase:
return chains
def get_extrainfo_step(self, input_message):
if self.do_doc_retrieval:
input_message = self.get_doc_retrieval(input_message)
# def get_extrainfo_step(self, input_message):
# if self.do_doc_retrieval:
# input_message = self.get_doc_retrieval(input_message)
logger.debug(F"self.do_code_retrieval: {self.do_code_retrieval}")
if self.do_code_retrieval:
input_message = self.get_code_retrieval(input_message)
# # logger.debug(F"self.do_code_retrieval: {self.do_code_retrieval}")
# if self.do_code_retrieval:
# input_message = self.get_code_retrieval(input_message)
if self.do_search:
input_message = self.get_search_retrieval(input_message)
# if self.do_search:
# input_message = self.get_search_retrieval(input_message)
return input_message
# return input_message
def inherit_extrainfo(self, input_message: Message, output_message: Message):
output_message.db_docs = input_message.db_docs
output_message.search_docs = input_message.search_docs
output_message.code_docs = input_message.code_docs
output_message.figures.update(input_message.figures)
return output_message
# def inherit_extrainfo(self, input_message: Message, output_message: Message):
# output_message.db_docs = input_message.db_docs
# output_message.search_docs = input_message.search_docs
# output_message.code_docs = input_message.code_docs
# output_message.figures.update(input_message.figures)
# output_message.origin_query = input_message.origin_query
# return output_message
def get_search_retrieval(self, message: Message,) -> Message:
SEARCH_ENGINES = {"duckduckgo": DDGSTool}
search_docs = []
for idx, doc in enumerate(SEARCH_ENGINES["duckduckgo"].run(message.role_content, 3)):
doc.update({"index": idx})
search_docs.append(Doc(**doc))
message.search_docs = search_docs
return message
# def get_search_retrieval(self, message: Message,) -> Message:
# SEARCH_ENGINES = {"duckduckgo": DDGSTool}
# search_docs = []
# for idx, doc in enumerate(SEARCH_ENGINES["duckduckgo"].run(message.role_content, 3)):
# doc.update({"index": idx})
# search_docs.append(Doc(**doc))
# message.search_docs = search_docs
# return message
def get_doc_retrieval(self, message: Message) -> Message:
query = message.role_content
knowledge_basename = message.doc_engine_name
top_k = message.top_k
score_threshold = message.score_threshold
if knowledge_basename:
docs = DocRetrieval.run(query, knowledge_basename, top_k, score_threshold)
message.db_docs = [Doc(**doc) for doc in docs]
return message
# def get_doc_retrieval(self, message: Message) -> Message:
# query = message.role_content
# knowledge_basename = message.doc_engine_name
# top_k = message.top_k
# score_threshold = message.score_threshold
# if knowledge_basename:
# docs = DocRetrieval.run(query, knowledge_basename, top_k, score_threshold)
# message.db_docs = [Doc(**doc) for doc in docs]
# return message
def get_code_retrieval(self, message: Message) -> Message:
# DocRetrieval.run("langchain是什么", "DSADSAD")
query = message.input_query
code_engine_name = message.code_engine_name
history_node_list = message.history_node_list
code_docs = CodeRetrieval.run(code_engine_name, query, code_limit=message.top_k, history_node_list=history_node_list)
message.code_docs = [CodeDoc(**doc) for doc in code_docs]
return message
# def get_code_retrieval(self, message: Message) -> Message:
# # DocRetrieval.run("langchain是什么", "DSADSAD")
# query = message.input_query
# code_engine_name = message.code_engine_name
# history_node_list = message.history_node_list
# code_docs = CodeRetrieval.run(code_engine_name, query, code_limit=message.top_k, history_node_list=history_node_list)
# message.code_docs = [CodeDoc(**doc) for doc in code_docs]
# return message
def get_tool_retrieval(self, message: Message) -> Message:
return message
# def get_tool_retrieval(self, message: Message) -> Message:
# return message
def update(self) -> Memory:
pass
@ -205,7 +227,7 @@ class BasePhase:
)
def get_memory_str(self, do_all_memory=True, content_key="role_content") -> str:
memory = self.global_message if do_all_memory else self.phase_memory
memory = self.global_memory if do_all_memory else self.phase_memory
return "\n".join([": ".join(i) for i in memory.to_tuple_messages(content_key=content_key)])
def get_chains_memory(self, content_key="role_content") -> List[Tuple]:

9
dev_opsgpt/connector/schema/__init__.py Normal file
View File

@ -0,0 +1,9 @@
from .memory import Memory
from .general_schema import *
from .message import Message
__all__ = [
"Memory", "ActionStatus", "Doc", "CodeDoc", "Task",
"Env", "Role", "ChainConfig", "AgentConfig", "PhaseConfig", "Message",
"load_role_configs", "load_chain_configs", "load_phase_configs"
]

128
dev_opsgpt/connector/connector_schema.py → dev_opsgpt/connector/schema/general_schema.py
View File

@ -16,11 +16,50 @@ class ActionStatus(Enum):
EXECUTING_CODE = "executing_code"
EXECUTING_TOOL = "executing_tool"
DEFAUILT = "default"
CODING2FILE = "coding2file"
def __eq__(self, other):
if isinstance(other, str):
return self.value == other
return super().__eq__(other)
class RoleTypeEnums(Enum):
SYSTEM = "system"
USER = "user"
ASSISTANT = "assistant"
FUNCTION = "function"
OBSERVATION = "observation"
def __eq__(self, other):
if isinstance(other, str):
return self.value == other
return super().__eq__(other)
class InputKeyEnums(Enum):
# Origin Query is ui's user question
ORIGIN_QUERY = "origin_query"
# agent's input from last agent
CURRENT_QUESTION = "current_question"
# ui memory contaisn (user and assistants)
UI_MEMORY = "ui_memory"
# agent's memory
SELF_MEMORY = "self_memory"
# chain memory
CHAIN_MEMORY = "chain_memory"
# agent's memory
SELF_ONE_MEMORY = "self_one_memory"
# chain memory
CHAIN_ONE_MEMORY = "chain_one_memory"
# Doc Infomations contains (Doc\Code\Search)
DOC_INFOS = "doc_infos"
def __eq__(self, other):
if isinstance(other, str):
return self.value == other
return super().__eq__(other)
class Doc(BaseModel):
title: str
@ -117,10 +156,13 @@ class ChainConfig(BaseModel):
class AgentConfig(BaseModel):
role: Role
stop: str = None
chat_turn: int = 1
do_search: bool = False
do_doc_retrieval: bool = False
do_tool_retrieval: bool = False
focus_agents: List = []
focus_message_keys: List = []
class PhaseConfig(BaseModel):
@ -133,92 +175,6 @@ class PhaseConfig(BaseModel):
do_code_retrieval: bool = False
do_tool_retrieval: bool = False
class Message(BaseModel):
role_name: str
role_type: str
role_prompt: str = None
input_query: str = None
# 模型最终返回
role_content: str = None
role_contents: List[str] = []
step_content: str = None
step_contents: List[str] = []
chain_content: str = None
chain_contents: List[str] = []
# 模型结果解析
plans: List[str] = None
code_content: str = None
code_filename: str = None
tool_params: str = None
tool_name: str = None
# 执行结果
action_status: str = ActionStatus.DEFAUILT
code_answer: str = None
tool_answer: str = None
observation: str = None
figures: Dict[str, str] = {}
# 辅助信息
tools: List[BaseTool] = []
task: Task = None
db_docs: List['Doc'] = []
code_docs: List['CodeDoc'] = []
search_docs: List['Doc'] = []
# 执行输入
phase_name: str = None
chain_name: str = None
do_search: bool = False
doc_engine_name: str = None
code_engine_name: str = None
search_engine_name: str = None
top_k: int = 3
score_threshold: float = 1.0
do_doc_retrieval: bool = False
do_code_retrieval: bool = False
do_tool_retrieval: bool = False
history_node_list: List[str] = []
def to_tuple_message(self, return_all: bool = False, content_key="role_content"):
if content_key == "role_content":
role_content = self.role_content
elif content_key == "step_content":
role_content = self.step_content or self.role_content
else:
role_content =self.role_content
if return_all:
return (self.role_name, self.role_type, role_content)
else:
return (self.role_name, role_content)
return (self.role_type, re.sub("}", "}}", re.sub("{", "{{", str(self.role_content))))
def to_dict_message(self, return_all: bool = False, content_key="role_content"):
if content_key == "role_content":
role_content =self.role_content
elif content_key == "step_content":
role_content = self.step_content or self.role_content
else:
role_content =self.role_content
if return_all:
return vars(self)
else:
return {"role": self.role_name, "content": role_content}
def is_system_role(self,):
return self.role_type == "system"
def __str__(self) -> str:
# key_str = '\n'.join([k for k, v in vars(self).items()])
# logger.debug(f"{key_str}")
return "\n".join([": ".join([k, str(v)]) for k, v in vars(self).items()])
def load_role_configs(config) -> Dict[str, AgentConfig]:
if isinstance(config, str):

62
dev_opsgpt/connector/shcema/memory.py → dev_opsgpt/connector/schema/memory.py
View File

@ -1,17 +1,18 @@
from pydantic import BaseModel
from typing import List
from typing import List, Union
from loguru import logger
from dev_opsgpt.connector.connector_schema import Message
from .message import Message
from dev_opsgpt.utils.common_utils import (
save_to_jsonl_file, save_to_json_file, read_json_file, read_jsonl_file
)
class Memory:
def __init__(self, messages: List[Message] = []):
self.messages = messages
class Memory(BaseModel):
messages: List[Message] = []
# def __init__(self, messages: List[Message] = []):
# self.messages = messages
def append(self, message: Message):
self.messages.append(message)
@ -28,8 +29,9 @@ class Memory:
def delete(self, ):
pass
def get_messages(self, ) -> List[Message]:
return self.messages
def get_messages(self, k=0) -> List[Message]:
"""Return the most recent k memories, return all when k=0"""
return self.messages[-k:]
def save(self, file_type="jsonl", return_all=True):
try:
@ -57,32 +59,56 @@ class Memory:
return False
def to_tuple_messages(self, return_system: bool = False, return_all: bool = False, content_key="role_content"):
def to_tuple_messages(self, return_all: bool = True, content_key="role_content", filter_roles=[]):
# logger.debug(f"{[message.to_tuple_message(return_all, content_key) for message in self.messages ]}")
return [
message.to_tuple_message(return_all, content_key) for message in self.messages
if not message.is_system_role() | return_system
if message.role_name not in filter_roles
]
def to_dict_messages(self, return_system: bool = False, return_all: bool = False, content_key="role_content"):
def to_dict_messages(self, return_all: bool = True, content_key="role_content", filter_roles=[]):
return [
message.to_dict_message(return_all, content_key) for message in self.messages
if not message.is_system_role() | return_system
if message.role_name not in filter_roles
]
def to_str_messages(self, return_system: bool = False, return_all: bool = False, content_key="role_content"):
def to_str_messages(self, return_all: bool = True, content_key="role_content", filter_roles=[]):
# for message in self.messages:
# logger.debug(f"{message.to_tuple_message(return_all, content_key)}")
# logger.debug(f"{[message.to_tuple_message(return_all, content_key) for message in self.messages ]}")
return "\n".join([
": ".join(message.to_tuple_message(return_all, content_key)) for message in self.messages
if not message.is_system_role() | return_system
return "\n\n".join([message.to_str_content(return_all, content_key) for message in self.messages
if message.role_name not in filter_roles
])
def get_parserd_output(self, ):
return [message.parsed_output for message in self.messages]
def get_parserd_output_list(self, ):
# for message in self.messages:
# logger.debug(f"{message.role_name}: {message.parsed_output_list}")
return [parsed_output for message in self.messages for parsed_output in message.parsed_output_list[1:]]
def get_rolenames(self, ):
''''''
return [message.role_name for message in self.messages]
@classmethod
def from_memory_list(cls, memorys: List['Memory']) -> 'Memory':
return cls([message for memory in memorys for message in memory.get_messages()])
return cls(messages=[message for memory in memorys for message in memory.get_messages()])
def __len__(self, ):
return len(self.messages)
def __str__(self) -> str:
return "\n".join([":".join(i) for i in self.to_tuple_messages()])
return "\n".join([":".join(i) for i in self.to_tuple_messages()])
def __add__(self, other: Union[Message, 'Memory']) -> 'Memory':
if isinstance(other, Message):
return Memory(messages=self.messages + [other])
elif isinstance(other, Memory):
return Memory(messages=self.messages + other.messages)
else:
raise ValueError(f"cant add unspecified type like as {type(other)}")

98
dev_opsgpt/connector/schema/message.py Normal file
View File

@ -0,0 +1,98 @@
from pydantic import BaseModel
from loguru import logger
from .general_schema import *
class Message(BaseModel):
chat_index: str = None
role_name: str
role_type: str
role_prompt: str = None
input_query: str = None
origin_query: str = None
# 模型最终返回
role_content: str = None
role_contents: List[str] = []
step_content: str = None
step_contents: List[str] = []
chain_content: str = None
chain_contents: List[str] = []
# 模型结果解析
plans: List[str] = None
code_content: str = None
code_filename: str = None
tool_params: str = None
tool_name: str = None
parsed_output: dict = {}
spec_parsed_output: dict = {}
parsed_output_list: List[Dict] = []
# 执行结果
action_status: str = ActionStatus.DEFAUILT
agent_index: int = None
code_answer: str = None
tool_answer: str = None
observation: str = None
figures: Dict[str, str] = {}
# 辅助信息
tools: List[BaseTool] = []
task: Task = None
db_docs: List['Doc'] = []
code_docs: List['CodeDoc'] = []
search_docs: List['Doc'] = []
agents: List = []
# 执行输入
phase_name: str = None
chain_name: str = None
do_search: bool = False
doc_engine_name: str = None
code_engine_name: str = None
cb_search_type: str = None
search_engine_name: str = None
top_k: int = 3
score_threshold: float = 1.0
do_doc_retrieval: bool = False
do_code_retrieval: bool = False
do_tool_retrieval: bool = False
history_node_list: List[str] = []
def to_tuple_message(self, return_all: bool = True, content_key="role_content"):
role_content = self.to_str_content(False, content_key)
if return_all:
return (self.role_name, role_content)
else:
return (role_content)
def to_dict_message(self, return_all: bool = True, content_key="role_content"):
role_content = self.to_str_content(False, content_key)
if return_all:
return {"role": self.role_name, "content": role_content}
else:
return vars(self)
def to_str_content(self, return_all: bool = True, content_key="role_content"):
if content_key == "role_content":
role_content = self.role_content or self.input_query
elif content_key == "step_content":
role_content = self.step_content or self.role_content or self.input_query
else:
role_content = self.role_content or self.input_query
if return_all:
return f"{self.role_name}: {role_content}"
else:
return role_content
def is_system_role(self,):
return self.role_type == "system"
def __str__(self) -> str:
# key_str = '\n'.join([k for k, v in vars(self).items()])
# logger.debug(f"{key_str}")
return "\n".join([": ".join([k, str(v)]) for k, v in vars(self).items()])

6
dev_opsgpt/connector/shcema/__init__.py
View File

@ -1,6 +0,0 @@
from .memory import Memory
__all__ = [
"Memory"
]

25
dev_opsgpt/connector/utils.py
View File

@ -1,5 +1,30 @@
import re
def parse_section(text, section_name):
# Define a pattern to extract the named section along with its content
section_pattern = rf'#### {section_name}\n(.*?)(?=####|$)'
# Find the specific section content
section_content = re.search(section_pattern, text, re.DOTALL)
if section_content:
# If the section is found, extract the content
content = section_content.group(1)
# Define a pattern to find segments that follow the format **xx:**
segments_pattern = r'\*\*([^*]+):\*\*'
# Use findall method to extract all matches in the section content
segments = re.findall(segments_pattern, content)
return segments
else:
# If the section is not found, return an empty list
return []
def prompt_cost(model_type: str, num_prompt_tokens: float, num_completion_tokens: float):
input_cost_map = {
"gpt-3.5-turbo": 0.0015,

7
dev_opsgpt/db_handler/__init__.py Normal file
View File

@ -0,0 +1,7 @@
# encoding: utf-8
'''
@author: 温进
@file: __init__.py.py
@time: 2023/11/16 下午3:15
@desc:
'''

7
dev_opsgpt/db_handler/graph_db_handler/__init__.py Normal file
View File

@ -0,0 +1,7 @@
# encoding: utf-8
'''
@author: 温进
@file: __init__.py.py
@time: 2023/11/20 下午3:07
@desc:
'''

263
dev_opsgpt/db_handler/graph_db_handler/nebula_handler.py Normal file
View File

@ -0,0 +1,263 @@
# encoding: utf-8
'''
@author: 温进
@file: nebula_handler.py
@time: 2023/11/16 下午3:15
@desc:
'''
import time
from loguru import logger
from nebula3.gclient.net import ConnectionPool
from nebula3.Config import Config
class NebulaHandler:
def __init__(self, host: str, port: int, username: str, password: str = '', space_name: str = ''):
'''
init nebula connection_pool
@param host: host
@param port: port
@param username: username
@param password: password
'''
config = Config()
self.connection_pool = ConnectionPool()
self.connection_pool.init([(host, port)], config)
self.username = username
self.password = password
self.space_name = space_name
def execute_cypher(self, cypher: str, space_name: str = ''):
'''
@param space_name: space_name, if provided, will execute use space_name first
@param cypher:
@return:
'''
with self.connection_pool.session_context(self.username, self.password) as session:
if space_name:
cypher = f'USE {space_name};{cypher}'
logger.debug(cypher)
resp = session.execute(cypher)
return resp
def close_connection(self):
self.connection_pool.close()
def create_space(self, space_name: str, vid_type: str, comment: str = ''):
'''
create space
@param space_name: cannot startwith number
@return:
'''
cypher = f'CREATE SPACE IF NOT EXISTS {space_name} (vid_type={vid_type}) comment="{comment}";'
resp = self.execute_cypher(cypher)
return resp
def show_space(self):
cypher = 'SHOW SPACES'
resp = self.execute_cypher(cypher)
return resp
def drop_space(self, space_name):
cypher = f'DROP SPACE {space_name}'
return self.execute_cypher(cypher)
def create_tag(self, tag_name: str, prop_dict: dict = {}):
'''
创建 tag
@param tag_name: tag 名称
@param prop_dict: 属性字典 {'prop 名字': 'prop 类型'}
@return:
'''
cypher = f'CREATE TAG IF NOT EXISTS {tag_name}'
cypher += '('
for k, v in prop_dict.items():
cypher += f'{k} {v},'
cypher = cypher.rstrip(',')
cypher += ')'
cypher += ';'
res = self.execute_cypher(cypher, self.space_name)
return res
def show_tags(self):
'''
查看 tag
@return:
'''
cypher = 'SHOW TAGS'
resp = self.execute_cypher(cypher, self.space_name)
return resp
def insert_vertex(self, tag_name: str, value_dict: dict):
'''
insert vertex
@param tag_name:
@param value_dict: {'properties_name': [], values: {'vid':[]}} order should be the same in properties_name and values
@return:
'''
cypher = f'INSERT VERTEX {tag_name} ('
properties_name = value_dict['properties_name']
for property_name in properties_name:
cypher += f'{property_name},'
cypher = cypher.rstrip(',')
cypher += ') VALUES '
for vid, properties in value_dict['values'].items():
cypher += f'"{vid}":('
for property in properties:
if type(property) == str:
cypher += f'"{property}",'
else:
cypher += f'{property}'
cypher = cypher.rstrip(',')
cypher += '),'
cypher = cypher.rstrip(',')
cypher += ';'
res = self.execute_cypher(cypher, self.space_name)
return res
def create_edge_type(self, edge_type_name: str, prop_dict: dict = {}):
'''
创建 tag
@param edge_type_name: tag 名称
@param prop_dict: 属性字典 {'prop 名字': 'prop 类型'}
@return:
'''
cypher = f'CREATE EDGE IF NOT EXISTS {edge_type_name}'
cypher += '('
for k, v in prop_dict.items():
cypher += f'{k} {v},'
cypher = cypher.rstrip(',')
cypher += ')'
cypher += ';'
res = self.execute_cypher(cypher, self.space_name)
return res
def show_edge_type(self):
'''
查看 tag
@return:
'''
cypher = 'SHOW EDGES'
resp = self.execute_cypher(cypher, self.space_name)
return resp
def drop_edge_type(self, edge_type_name: str):
cypher = f'DROP EDGE {edge_type_name}'
return self.execute_cypher(cypher, self.space_name)
def insert_edge(self, edge_type_name: str, value_dict: dict):
'''
insert edge
@param edge_type_name:
@param value_dict: value_dict: {'properties_name': [], values: {(src_vid, dst_vid):[]}} order should be the
same in properties_name and values
@return:
'''
cypher = f'INSERT EDGE {edge_type_name} ('
properties_name = value_dict['properties_name']
for property_name in properties_name:
cypher += f'{property_name},'
cypher = cypher.rstrip(',')
cypher += ') VALUES '
for (src_vid, dst_vid), properties in value_dict['values'].items():
cypher += f'"{src_vid}"->"{dst_vid}":('
for property in properties:
if type(property) == str:
cypher += f'"{property}",'
else:
cypher += f'{property}'
cypher = cypher.rstrip(',')
cypher += '),'
cypher = cypher.rstrip(',')
cypher += ';'
res = self.execute_cypher(cypher, self.space_name)
return res
def set_space_name(self, space_name):
self.space_name = space_name
def add_host(self, host: str, port: str):
'''
add host
@return:
'''
cypher = f'ADD HOSTS {host}:{port}'
res = self.execute_cypher(cypher)
return res
def get_stat(self):
'''
@return:
'''
submit_cypher = 'SUBMIT JOB STATS;'
self.execute_cypher(cypher=submit_cypher, space_name=self.space_name)
time.sleep(2)
stats_cypher = 'SHOW STATS;'
stats_res = self.execute_cypher(cypher=stats_cypher, space_name=self.space_name)
res = {'vertices': -1, 'edges': -1}
stats_res_dict = self.result_to_dict(stats_res)
for idx in range(len(stats_res_dict['Type'])):
t = stats_res_dict['Type'][idx].as_string()
name = stats_res_dict['Name'][idx].as_string()
count = stats_res_dict['Count'][idx].as_int()
if t == 'Space' and name in res:
res[name] = count
return res
def get_vertices(self, tag_name: str = '', limit: int = 10000):
'''
get all vertices
@return:
'''
if tag_name:
cypher = f'''MATCH (v:{tag_name}) RETURN v LIMIT {limit};'''
else:
cypher = f'MATCH (v) RETURN v LIMIT {limit};'
res = self.execute_cypher(cypher, self.space_name)
return self.result_to_dict(res)
def result_to_dict(self, result) -> dict:
"""
build list for each column, and transform to dataframe
"""
logger.info(result.error_msg())
assert result.is_succeeded()
columns = result.keys()
d = {}
for col_num in range(result.col_size()):
col_name = columns[col_num]
col_list = result.column_values(col_name)
d[col_name] = [x for x in col_list]
return d

7
dev_opsgpt/db_handler/vector_db_handler/__init__.py Normal file
View File

@ -0,0 +1,7 @@
# encoding: utf-8
'''
@author: 温进
@file: __init__.py.py
@time: 2023/11/20 下午3:08
@desc:
'''

140
dev_opsgpt/db_handler/vector_db_handler/chroma_handler.py Normal file
View File

@ -0,0 +1,140 @@
# encoding: utf-8
'''
@author: 温进
@file: chroma_handler.py
@time: 2023/11/21 下午12:21
@desc:
'''
from loguru import logger
import chromadb
class ChromaHandler:
def __init__(self, path: str, collection_name: str = ''):
'''
init client
@param path: path of data
@collection_name: name of collection
'''
self.client = chromadb.PersistentClient(path)
self.client.heartbeat()
if collection_name:
self.collection = self.client.get_or_create_collection(name=collection_name)
def create_collection(self, collection_name: str):
'''
create collection, if exists, will override
@return:
'''
try:
collection = self.client.create_collection(name=collection_name)
except Exception as e:
return {'result_code': -1, 'msg': f'fail, error={e}'}
return {'result_code': 0, 'msg': 'success'}
def delete_collection(self, collection_name: str):
'''
@param collection_name:
@return:
'''
try:
self.client.delete_collection(name=collection_name)
except Exception as e:
return {'result_code': -1, 'msg': f'fail, error={e}'}
return {'result_code': 0, 'msg': 'success'}
def set_collection(self, collection_name: str):
'''
@param collection_name:
@return:
'''
try:
self.collection = self.client.get_collection(collection_name)
except Exception as e:
return {'result_code': -1, 'msg': f'fail, error={e}'}
return {'result_code': 0, 'msg': 'success'}
def add_data(self, ids: list, documents: list = None, embeddings: list = None, metadatas: list = None):
'''
add data to chroma
@param documents: list of doc string
@param embeddings: list of vector
@param metadatas: list of metadata
@param ids: list of id
@return:
'''
try:
self.collection.add(
ids=ids,
embeddings=embeddings,
metadatas=metadatas,
documents=documents
)
except Exception as e:
return {'result_code': -1, 'msg': f'fail, error={e}'}
return {'result_code': 0, 'msg': 'success'}
def query(self, query_embeddings=None, query_texts=None, n_results=10, where=None, where_document=None,
include=["metadatas", "documents", "distances"]):
'''
@param query_embeddings:
@param query_texts:
@param n_results:
@param where:
@param where_document:
@param include:
@return:
'''
try:
query_result = self.collection.query(query_embeddings=query_embeddings, query_texts=query_texts,
n_results=n_results, where=where, where_document=where_document,
include=include)
return {'result_code': 0, 'msg': 'success', 'result': query_result}
except Exception as e:
return {'result_code': -1, 'msg': f'fail, error={e}'}
def get(self, ids=None, where=None, limit=None, offset=None, where_document=None, include=["metadatas", "documents"]):
'''
get by condition
@param ids:
@param where:
@param limit:
@param offset:
@param where_document:
@param include:
@return:
'''
try:
query_result = self.collection.get(ids=ids, where=where, where_document=where_document,
limit=limit,
offset=offset, include=include)
return {'result_code': 0, 'msg': 'success', 'result': query_result}
except Exception as e:
return {'result_code': -1, 'msg': f'fail, error={e}'}
def peek(self, limit: int=10):
'''
peek
@param limit:
@return:
'''
try:
query_result = self.collection.peek(limit)
return {'result_code': 0, 'msg': 'success', 'result': query_result}
except Exception as e:
return {'result_code': -1, 'msg': f'fail, error={e}'}
def count(self):
'''
count
@return:
'''
try:
query_result = self.collection.count()
return {'result_code': 0, 'msg': 'success', 'result': query_result}
except Exception as e:
return {'result_code': -1, 'msg': f'fail, error={e}'}

39
dev_opsgpt/embeddings/get_embedding.py Normal file
View File

@ -0,0 +1,39 @@
# encoding: utf-8
'''
@author: 温进
@file: get_embedding.py
@time: 2023/11/22 上午11:30
@desc:
'''
from loguru import logger
from configs.model_config import EMBEDDING_MODEL
from dev_opsgpt.embeddings.openai_embedding import OpenAIEmbedding
from dev_opsgpt.embeddings.huggingface_embedding import HFEmbedding
def get_embedding(engine: str, text_list: list):
'''
get embedding
@param engine: openai / hf
@param text_list:
@return:
'''
emb_res = {}
if engine == 'openai':
oae = OpenAIEmbedding()
emb_res = oae.get_emb(text_list)
elif engine == 'model':
hfe = HFEmbedding(EMBEDDING_MODEL)
emb_res = hfe.get_emb(text_list)
return emb_res
if __name__ == '__main__':
engine = 'model'
text_list = ['这段代码是一个OkHttp拦截器，用于在请求头中添加授权令牌。它继承自`com.theokanning.openai.client.AuthenticationInterceptor`类，并且被标记为`@Deprecated`，意味着它已经过时了。\n\n这个拦截器的作用是在每个请求的头部添加一个名为"Authorization"的字段，值为传入的授权令牌。这样，当请求被发送到服务器时，服务器可以使用这个令牌来验证请求的合法性。\n\n这段代码的构造函数接受一个令牌作为参数，并将其传递给父类的构造函数。这个令牌应该是一个有效的授权令牌，用于访问受保护的资源。', '这段代码定义了一个接口`OpenAiApi`，并使用`@Deprecated`注解将其标记为已过时。它还扩展了`com.theokanning.openai.client.OpenAiApi`接口。\n\n`@Deprecated`注解表示该接口已经过时，不推荐使用。开发者应该使用`com.theokanning.openai.client.OpenAiApi`接口代替。\n\n注释中提到这个接口只是为了保持向后兼容性。这意味着它可能是为了与旧版本的代码兼容而保留的，但不推荐在新代码中使用。', '这段代码是一个OkHttp的拦截器，用于在请求头中添加授权令牌（authorization token）。\n\n在这个拦截器中，首先获取到传入的授权令牌（token），然后在每个请求的构建过程中，使用`newBuilder()`方法创建一个新的请求构建器，并在该构建器中添加一个名为"Authorization"的请求头，值为"Bearer " + token。最后，使用该构建器构建一个新的请求，并通过`chain.proceed(request)`方法继续处理该请求。\n\n这样，当使用OkHttp发送请求时，该拦截器会自动在请求头中添加授权令牌，以实现身份验证的功能。', '这段代码是一个Java接口，用于定义与OpenAI API进行通信的方法。它包含了各种不同类型的请求和响应方法，用于与OpenAI API的不同端点进行交互。\n\n接口中的方法包括：\n- `listModels()`：获取可用的模型列表。\n- `getModel(String modelId)`：获取指定模型的详细信息。\n- `createCompletion(CompletionRequest request)`：创建文本生成的请求。\n- `createChatCompletion(ChatCompletionRequest request)`：创建聊天式文本生成的请求。\n- `createEdit(EditRequest request)`：创建文本编辑的请求。\n- `createEmbeddings(EmbeddingRequest request)`：创建文本嵌入的请求。\n- `listFiles()`：获取已上传文件的列表。\n- `uploadFile(RequestBody purpose, MultipartBody.Part file)`：上传文件。\n- `deleteFile(String fileId)`：删除文件。\n- `retrieveFile(String fileId)`：获取文件的详细信息。\n- `retrieveFileContent(String fileId)`：获取文件的内容。\n- `createFineTuningJob(FineTuningJobRequest request)`：创建Fine-Tuning任务。\n- `listFineTuningJobs()`：获取Fine-Tuning任务的列表。\n- `retrieveFineTuningJob(String fineTuningJobId)`：获取指定Fine-Tuning任务的详细信息。\n- `cancelFineTuningJob(String fineTuningJobId)`：取消Fine-Tuning任务。\n- `listFineTuningJobEvents(String fineTuningJobId)`：获取Fine-Tuning任务的事件列表。\n- `createFineTuneCompletion(CompletionRequest request)`：创建Fine-Tuning模型的文本生成请求。\n- `createImage(CreateImageRequest request)`：创建图像生成的请求。\n- `createImageEdit(RequestBody requestBody)`：创建图像编辑的请求。\n- `createImageVariation(RequestBody requestBody)`：创建图像变体的请求。\n- `createTranscription(RequestBody requestBody)`：创建音频转录的请求。\n- `createTranslation(RequestBody requestBody)`：创建音频翻译的请求。\n- `createModeration(ModerationRequest request)`：创建内容审核的请求。\n- `getEngines()`：获取可用的引擎列表。\n- `getEngine(String engineId)`：获取指定引擎的详细信息。\n- `subscription()`：获取账户订阅信息。\n- `billingUsage(LocalDate starDate, LocalDate endDate)`：获取账户消费信息。\n\n这些方法使用不同的HTTP请求类型（GET、POST、DELETE）和路径来与OpenAI API进行交互，并返回相应的响应数据。']
res = get_embedding(engine, text_list)
logger.debug(res)

49
dev_opsgpt/embeddings/huggingface_embedding.py Normal file
View File

@ -0,0 +1,49 @@
# encoding: utf-8
'''
@author: 温进
@file: huggingface_embedding.py
@time: 2023/11/30 上午11:41
@desc:
'''
from loguru import logger
from configs.model_config import EMBEDDING_DEVICE
from dev_opsgpt.embeddings.utils import load_embeddings
class HFEmbedding:
_instance = {}
def __new__(cls, *args, **kwargs):
instance_key = f'{args},{kwargs}'
if cls._instance.get(instance_key, None):
return cls._instance[instance_key]
else:
cls._instance[instance_key] = super().__new__(cls)
return cls._instance[instance_key]
def __init__(self, model_name):
self.model = load_embeddings(model=model_name, device=EMBEDDING_DEVICE)
logger.debug('load success')
def get_emb(self, text_list):
'''
get embedding
@param text_list:
@return:
'''
logger.info('st')
emb_res = self.model.embed_documents(text_list)
logger.info('ed')
res = {
text_list[idx]: emb_res[idx] for idx in range(len(text_list))
}
return res
if __name__ == '__main__':
model_name = 'text2vec-base'
hfe = HFEmbedding(model_name)
text_list = ['这段代码是一个OkHttp拦截器，用于在请求头中添加授权令牌。它继承自`com.theokanning.openai.client.AuthenticationInterceptor`类，并且被标记为`@Deprecated`，意味着它已经过时了。\n\n这个拦截器的作用是在每个请求的头部添加一个名为"Authorization"的字段，值为传入的授权令牌。这样，当请求被发送到服务器时，服务器可以使用这个令牌来验证请求的合法性。\n\n这段代码的构造函数接受一个令牌作为参数，并将其传递给父类的构造函数。这个令牌应该是一个有效的授权令牌，用于访问受保护的资源。', '这段代码定义了一个接口`OpenAiApi`，并使用`@Deprecated`注解将其标记为已过时。它还扩展了`com.theokanning.openai.client.OpenAiApi`接口。\n\n`@Deprecated`注解表示该接口已经过时，不推荐使用。开发者应该使用`com.theokanning.openai.client.OpenAiApi`接口代替。\n\n注释中提到这个接口只是为了保持向后兼容性。这意味着它可能是为了与旧版本的代码兼容而保留的，但不推荐在新代码中使用。', '这段代码是一个OkHttp的拦截器，用于在请求头中添加授权令牌（authorization token）。\n\n在这个拦截器中，首先获取到传入的授权令牌（token），然后在每个请求的构建过程中，使用`newBuilder()`方法创建一个新的请求构建器，并在该构建器中添加一个名为"Authorization"的请求头，值为"Bearer " + token。最后，使用该构建器构建一个新的请求，并通过`chain.proceed(request)`方法继续处理该请求。\n\n这样，当使用OkHttp发送请求时，该拦截器会自动在请求头中添加授权令牌，以实现身份验证的功能。', '这段代码是一个Java接口，用于定义与OpenAI API进行通信的方法。它包含了各种不同类型的请求和响应方法，用于与OpenAI API的不同端点进行交互。\n\n接口中的方法包括：\n- `listModels()`：获取可用的模型列表。\n- `getModel(String modelId)`：获取指定模型的详细信息。\n- `createCompletion(CompletionRequest request)`：创建文本生成的请求。\n- `createChatCompletion(ChatCompletionRequest request)`：创建聊天式文本生成的请求。\n- `createEdit(EditRequest request)`：创建文本编辑的请求。\n- `createEmbeddings(EmbeddingRequest request)`：创建文本嵌入的请求。\n- `listFiles()`：获取已上传文件的列表。\n- `uploadFile(RequestBody purpose, MultipartBody.Part file)`：上传文件。\n- `deleteFile(String fileId)`：删除文件。\n- `retrieveFile(String fileId)`：获取文件的详细信息。\n- `retrieveFileContent(String fileId)`：获取文件的内容。\n- `createFineTuningJob(FineTuningJobRequest request)`：创建Fine-Tuning任务。\n- `listFineTuningJobs()`：获取Fine-Tuning任务的列表。\n- `retrieveFineTuningJob(String fineTuningJobId)`：获取指定Fine-Tuning任务的详细信息。\n- `cancelFineTuningJob(String fineTuningJobId)`：取消Fine-Tuning任务。\n- `listFineTuningJobEvents(String fineTuningJobId)`：获取Fine-Tuning任务的事件列表。\n- `createFineTuneCompletion(CompletionRequest request)`：创建Fine-Tuning模型的文本生成请求。\n- `createImage(CreateImageRequest request)`：创建图像生成的请求。\n- `createImageEdit(RequestBody requestBody)`：创建图像编辑的请求。\n- `createImageVariation(RequestBody requestBody)`：创建图像变体的请求。\n- `createTranscription(RequestBody requestBody)`：创建音频转录的请求。\n- `createTranslation(RequestBody requestBody)`：创建音频翻译的请求。\n- `createModeration(ModerationRequest request)`：创建内容审核的请求。\n- `getEngines()`：获取可用的引擎列表。\n- `getEngine(String engineId)`：获取指定引擎的详细信息。\n- `subscription()`：获取账户订阅信息。\n- `billingUsage(LocalDate starDate, LocalDate endDate)`：获取账户消费信息。\n\n这些方法使用不同的HTTP请求类型（GET、POST、DELETE）和路径来与OpenAI API进行交互，并返回相应的响应数据。']
hfe.get_emb(text_list)

50
dev_opsgpt/embeddings/openai_embedding.py Normal file
View File

@ -0,0 +1,50 @@
# encoding: utf-8
'''
@author: 温进
@file: openai_embedding.py
@time: 2023/11/22 上午10:45
@desc:
'''
import openai
import base64
import json
import os
from loguru import logger
from configs.model_config import OPENAI_API_BASE
class OpenAIEmbedding:
def __init__(self):
pass
def get_emb(self, text_list):
openai.api_key = os.environ["OPENAI_API_KEY"]
openai.api_base = os.environ["API_BASE_URL"]
# change , to ，to avoid bug
modified_text_list = [i.replace(',', '，') for i in text_list]
emb_all_result = openai.Embedding.create(
model="text-embedding-ada-002",
input=modified_text_list
)
res = {}
# logger.debug(emb_all_result)
logger.debug(f'len of result={len(emb_all_result["data"])}')
for emb_result in emb_all_result['data']:
index = emb_result['index']
# logger.debug(index)
text = text_list[index]
emb = emb_result['embedding']
res[text] = emb
return res
if __name__ == '__main__':
oae = OpenAIEmbedding()
res = oae.get_emb(text_list=['这段代码是一个OkHttp拦截器，用于在请求头中添加授权令牌。它继承自`com.theokanning.openai.client.AuthenticationInterceptor`类，并且被标记为`@Deprecated`，意味着它已经过时了。\n\n这个拦截器的作用是在每个请求的头部添加一个名为"Authorization"的字段，值为传入的授权令牌。这样，当请求被发送到服务器时，服务器可以使用这个令牌来验证请求的合法性。\n\n这段代码的构造函数接受一个令牌作为参数，并将其传递给父类的构造函数。这个令牌应该是一个有效的授权令牌，用于访问受保护的资源。', '这段代码定义了一个接口`OpenAiApi`，并使用`@Deprecated`注解将其标记为已过时。它还扩展了`com.theokanning.openai.client.OpenAiApi`接口。\n\n`@Deprecated`注解表示该接口已经过时，不推荐使用。开发者应该使用`com.theokanning.openai.client.OpenAiApi`接口代替。\n\n注释中提到这个接口只是为了保持向后兼容性。这意味着它可能是为了与旧版本的代码兼容而保留的，但不推荐在新代码中使用。', '这段代码是一个OkHttp的拦截器，用于在请求头中添加授权令牌（authorization token）。\n\n在这个拦截器中，首先获取到传入的授权令牌（token），然后在每个请求的构建过程中，使用`newBuilder()`方法创建一个新的请求构建器，并在该构建器中添加一个名为"Authorization"的请求头，值为"Bearer " + token。最后，使用该构建器构建一个新的请求，并通过`chain.proceed(request)`方法继续处理该请求。\n\n这样，当使用OkHttp发送请求时，该拦截器会自动在请求头中添加授权令牌，以实现身份验证的功能。', '这段代码是一个Java接口，用于定义与OpenAI API进行通信的方法。它包含了各种不同类型的请求和响应方法，用于与OpenAI API的不同端点进行交互。\n\n接口中的方法包括：\n- `listModels()`：获取可用的模型列表。\n- `getModel(String modelId)`：获取指定模型的详细信息。\n- `createCompletion(CompletionRequest request)`：创建文本生成的请求。\n- `createChatCompletion(ChatCompletionRequest request)`：创建聊天式文本生成的请求。\n- `createEdit(EditRequest request)`：创建文本编辑的请求。\n- `createEmbeddings(EmbeddingRequest request)`：创建文本嵌入的请求。\n- `listFiles()`：获取已上传文件的列表。\n- `uploadFile(RequestBody purpose, MultipartBody.Part file)`：上传文件。\n- `deleteFile(String fileId)`：删除文件。\n- `retrieveFile(String fileId)`：获取文件的详细信息。\n- `retrieveFileContent(String fileId)`：获取文件的内容。\n- `createFineTuningJob(FineTuningJobRequest request)`：创建Fine-Tuning任务。\n- `listFineTuningJobs()`：获取Fine-Tuning任务的列表。\n- `retrieveFineTuningJob(String fineTuningJobId)`：获取指定Fine-Tuning任务的详细信息。\n- `cancelFineTuningJob(String fineTuningJobId)`：取消Fine-Tuning任务。\n- `listFineTuningJobEvents(String fineTuningJobId)`：获取Fine-Tuning任务的事件列表。\n- `createFineTuneCompletion(CompletionRequest request)`：创建Fine-Tuning模型的文本生成请求。\n- `createImage(CreateImageRequest request)`：创建图像生成的请求。\n- `createImageEdit(RequestBody requestBody)`：创建图像编辑的请求。\n- `createImageVariation(RequestBody requestBody)`：创建图像变体的请求。\n- `createTranscription(RequestBody requestBody)`：创建音频转录的请求。\n- `createTranslation(RequestBody requestBody)`：创建音频翻译的请求。\n- `createModeration(ModerationRequest request)`：创建内容审核的请求。\n- `getEngines()`：获取可用的引擎列表。\n- `getEngine(String engineId)`：获取指定引擎的详细信息。\n- `subscription()`：获取账户订阅信息。\n- `billingUsage(LocalDate starDate, LocalDate endDate)`：获取账户消费信息。\n\n这些方法使用不同的HTTP请求类型（GET、POST、DELETE）和路径来与OpenAI API进行交互，并返回相应的响应数据。'])
# res = oae.get_emb(text_list=['''test1"test2test3''', '''test4test5test6'''])
print(res)

4
dev_opsgpt/embeddings/utils.py
View File

@ -9,4 +9,6 @@ def load_embeddings(model: str, device: str):
logger.info("load embedding model: {}, {}".format(model, embedding_model_dict[model]))
embeddings = HuggingFaceEmbeddings(model_name=embedding_model_dict[model],
model_kwargs={'device': device})
return embeddings
return embeddings

15
dev_opsgpt/orm/commands/code_base_cds.py
View File

@ -11,12 +11,14 @@ from dev_opsgpt.orm.schemas.base_schema import CodeBaseSchema
@with_session
def add_cb_to_db(session, code_name, code_path, code_graph_node_num, code_file_num):
def add_cb_to_db(session, code_name, code_path, code_graph_node_num, code_file_num, do_interpret):
do_interpret = 'YES' if do_interpret else 'NO'
# 增：创建知识库实例
cb = session.query(CodeBaseSchema).filter_by(code_name=code_name).first()
if not cb:
cb = CodeBaseSchema(code_name=code_name, code_path=code_path, code_graph_node_num=code_graph_node_num,
code_file_num=code_file_num)
code_file_num=code_file_num, do_interpret=do_interpret)
session.add(cb)
else:
cb.code_path = code_path
@ -47,10 +49,10 @@ def cb_exists(session, code_name):
def load_cb_from_db(session, code_name):
cb = session.query(CodeBaseSchema).filter_by(code_name=code_name).first()
if cb:
code_name, code_path, code_graph_node_num = cb.code_name, cb.code_path, cb.code_graph_node_num
code_name, code_path, code_graph_node_num, do_interpret = cb.code_name, cb.code_path, cb.code_graph_node_num, cb.do_interpret
else:
code_name, code_path, code_graph_node_num = None, None, None
return code_name, code_path, code_graph_node_num
code_name, code_path, code_graph_node_num = None, None, None, None
return code_name, code_path, code_graph_node_num, do_interpret
@with_session
@ -71,7 +73,8 @@ def get_cb_detail(session, code_name: str) -> dict:
"code_name": cb.code_name,
"code_path": cb.code_path,
"code_graph_node_num": cb.code_graph_node_num,
'code_file_num': cb.code_file_num
'code_file_num': cb.code_file_num,
'do_interpret': cb.do_interpret
}
else:
return {

4
dev_opsgpt/orm/schemas/base_schema.py
View File

@ -58,6 +58,7 @@ class CodeBaseSchema(Base):
code_path = Column(String, comment='代码本地路径')
code_graph_node_num = Column(String, comment='代码图谱节点数')
code_file_num = Column(String, comment='代码解析文件数')
do_interpret = Column(String, comment='是否代码解读，Y or N')
create_time = Column(DateTime, default=func.now(), comment='创建时间')
def __repr__(self):
@ -65,5 +66,6 @@ class CodeBaseSchema(Base):
code_name='{self.code_name}',
code_path='{self.code_path}',
code_graph_node_num='{self.code_graph_node_num}',
code_file_num='{self.code_file_num}'
code_file_num='{self.code_file_num}',
do_interpret='{self.do_interpret}',
create_time='{self.create_time}')>"""

10
dev_opsgpt/sandbox/pycodebox.py
View File

@ -92,11 +92,11 @@ class PyCodeBox(BaseBox):
"metadata": {},
"content": {
"code": code_text,
"silent": True,
"store_history": True,
"silent": False, # True，则内核会执行代码，但不会发送执行结果（如输出）
"store_history": True, # True，则执行的代码会被记录在交互式环境的历史记录中
"user_expressions": {},
"allow_stdin": False,
"stop_on_error": True,
"allow_stdin": False, # True，允许代码执行时接受用户输入
"stop_on_error": True, # True，当执行中遇到错误时，后续代码将不会继续执行。
},
"channel": "shell",
"buffers": [],
@ -163,7 +163,7 @@ class PyCodeBox(BaseBox):
return CodeBoxResponse(
code_exe_type="text",
code_text=code_text,
code_exe_response=result or "Code run successfully (no output)，可能没有打印需要确认的变量",
code_exe_response=result or "Code run successfully (no output)",
code_exe_status=200,
do_code_exe=self.do_code_exe
)

37
dev_opsgpt/service/cb_api.py
View File

@ -23,7 +23,9 @@ from configs.model_config import (
CB_ROOT_PATH
)
from dev_opsgpt.codebase_handler.codebase_handler import CodeBaseHandler
# from dev_opsgpt.codebase_handler.codebase_handler import CodeBaseHandler
from dev_opsgpt.codechat.codebase_handler.codebase_handler import CodeBaseHandler
from loguru import logger
@ -33,10 +35,12 @@ async def list_cbs():
return ListResponse(data=list_cbs_from_db())
async def create_cb(cb_name: str = Body(..., examples=["samples"]),
code_path: str = Body(..., examples=["samples"])
async def create_cb(zip_file,
cb_name: str = Body(..., examples=["samples"]),
code_path: str = Body(..., examples=["samples"]),
do_interpret: bool = Body(..., examples=["samples"])
) -> BaseResponse:
logger.info('cb_name={}, zip_path={}'.format(cb_name, code_path))
logger.info('cb_name={}, zip_path={}, do_interpret={}'.format(cb_name, code_path, do_interpret))
# Create selected knowledge base
if not validate_kb_name(cb_name):
@ -50,17 +54,16 @@ async def create_cb(cb_name: str = Body(..., examples=["samples"]),
try:
logger.info('start build code base')
cbh = CodeBaseHandler(cb_name, code_path, cb_root_path=CB_ROOT_PATH)
cbh.import_code(do_save=True)
code_graph_node_num = len(cbh.nh)
code_file_num = len(cbh.lcdh)
cbh = CodeBaseHandler(cb_name, code_path)
vertices_num, edge_num, file_num = cbh.import_code(zip_file=zip_file, do_interpret=do_interpret)
logger.info('build code base done')
# create cb to table
add_cb_to_db(cb_name, cbh.code_path, code_graph_node_num, code_file_num)
add_cb_to_db(cb_name, cbh.code_path, vertices_num, file_num, do_interpret)
logger.info('add cb to mysql table success')
except Exception as e:
print(e)
logger.exception(e)
return BaseResponse(code=500, msg=f"创建代码知识库出错： {e}")
return BaseResponse(code=200, msg=f"已新增代码知识库 {cb_name}")
@ -81,6 +84,11 @@ async def delete_cb(cb_name: str = Body(..., examples=["samples"])) -> BaseRespo
# delete local file
shutil.rmtree(CB_ROOT_PATH + os.sep + cb_name)
# delete from codebase
cbh = CodeBaseHandler(cb_name)
cbh.delete_codebase(codebase_name=cb_name)
except Exception as e:
print(e)
return BaseResponse(code=500, msg=f"删除代码知识库出错： {e}")
@ -91,24 +99,25 @@ async def delete_cb(cb_name: str = Body(..., examples=["samples"])) -> BaseRespo
def search_code(cb_name: str = Body(..., examples=["sofaboot"]),
query: str = Body(..., examples=['你好']),
code_limit: int = Body(..., examples=['1']),
search_type: str = Body(..., examples=['你好']),
history_node_list: list = Body(...)) -> dict:
logger.info('cb_name={}'.format(cb_name))
logger.info('query={}'.format(query))
logger.info('code_limit={}'.format(code_limit))
logger.info('search_type={}'.format(search_type))
logger.info('history_node_list={}'.format(history_node_list))
try:
# load codebase
cbh = CodeBaseHandler(code_name=cb_name, cb_root_path=CB_ROOT_PATH)
cbh.import_code(do_load=True)
cbh = CodeBaseHandler(codebase_name=cb_name)
# search code
related_code, related_node = cbh.search_code(query, code_limit=code_limit, history_node_list=history_node_list)
context, related_vertices = cbh.search_code(query, search_type=search_type, limit=code_limit)
res = {
'related_code': related_code,
'related_node': related_node
'context': context,
'related_vertices': related_vertices
}
return res
except Exception as e:

7
dev_opsgpt/service/sdfile_api.py
View File

@ -12,7 +12,7 @@ from fastapi.middleware.cors import CORSMiddleware
from fastapi.responses import StreamingResponse, FileResponse
from fastapi import File, UploadFile
from dev_opsgpt.utils.server_utils import BaseResponse, ListResponse
from dev_opsgpt.utils.server_utils import BaseResponse, ListResponse, DataResponse
from configs.server_config import OPEN_CROSS_DOMAIN, SDFILE_API_SERVER
from configs.model_config import (
JUPYTER_WORK_PATH
@ -35,7 +35,8 @@ async def sd_upload_file(file: UploadFile = File(...), work_dir: str = JUPYTER_W
async def sd_download_file(filename: str, save_filename: str = "filename_to_download.ext", work_dir: str = JUPYTER_WORK_PATH):
# 从服务器下载文件
logger.debug(f"{os.path.join(work_dir, filename)}")
return {"data": FileResponse(os.path.join(work_dir, filename), filename=save_filename)}
return {"data": os.path.join(work_dir, filename), "filename": save_filename}
# return {"data": FileResponse(os.path.join(work_dir, filename), filename=save_filename)}
async def sd_list_files(work_dir: str = JUPYTER_WORK_PATH):
@ -78,7 +79,7 @@ def create_app():
app.get("/sdfiles/download",
tags=["files upload and download"],
response_model=BaseResponse,
response_model=DataResponse,
summary="从沙盒下载文件"
)(sd_download_file)

37
dev_opsgpt/tools/__init__.py
View File

@ -1,3 +1,5 @@
import importlib
from .base_tool import toLangchainTools, get_tool_schema, BaseToolModel
from .weather import WeatherInfo, DistrictInfo
from .multiplier import Multiplier
@ -7,27 +9,22 @@ from .metrics_query import MetricsQuery
from .duckduckgo_search import DDGSTool
from .docs_retrieval import DocRetrieval
from .cb_query_tool import CodeRetrieval
from .ocr_tool import BaiduOcrTool
from .stock_tool import StockInfo, StockName
TOOL_SETS = [
"WeatherInfo", "WorldTimeGetTimezoneByArea", "Multiplier", "DistrictInfo", "KSigmaDetector", "MetricsQuery", "DDGSTool",
"DocRetrieval", "CodeRetrieval"
]
TOOL_DICT = {
"WeatherInfo": WeatherInfo,
"WorldTimeGetTimezoneByArea": WorldTimeGetTimezoneByArea,
"Multiplier": Multiplier,
"DistrictInfo": DistrictInfo,
"KSigmaDetector": KSigmaDetector,
"MetricsQuery": MetricsQuery,
"DDGSTool": DDGSTool,
"DocRetrieval": DocRetrieval,
"CodeRetrieval": CodeRetrieval
}
__all__ = [
"WeatherInfo", "WorldTimeGetTimezoneByArea", "Multiplier", "DistrictInfo", "KSigmaDetector", "MetricsQuery", "DDGSTool",
"DocRetrieval", "CodeRetrieval",
"toLangchainTools", "get_tool_schema", "tool_sets", "BaseToolModel"
IMPORT_TOOL = [
WeatherInfo, DistrictInfo, Multiplier, WorldTimeGetTimezoneByArea,
KSigmaDetector, MetricsQuery, DDGSTool, DocRetrieval, CodeRetrieval,
BaiduOcrTool, StockInfo, StockName
]
TOOL_SETS = [tool.__name__ for tool in IMPORT_TOOL]
TOOL_DICT = {tool.__name__: tool for tool in IMPORT_TOOL}
__all__ = [
"toLangchainTools", "get_tool_schema", "tool_sets", "BaseToolModel"
] + TOOL_SETS

27
dev_opsgpt/tools/cb_query_tool.py
View File

@ -35,13 +35,26 @@ class CodeRetrieval(BaseToolModel):
code: str = Field(..., description="检索代码")
@classmethod
def run(cls, code_base_name, query, code_limit=CODE_SEARCH_TOP_K, history_node_list=[]):
def run(cls, code_base_name, query, code_limit=CODE_SEARCH_TOP_K, history_node_list=[], search_type="tag"):
"""excute your tool!"""
codes = search_code(code_base_name, query, code_limit, history_node_list=history_node_list)
return_codes = []
related_code = codes['related_code']
related_nodes = codes['related_node']
search_type = {
'基于 cypher': 'cypher',
'基于标签': 'tag',
'基于描述': 'description',
'tag': 'tag',
'description': 'description',
'cypher': 'cypher'
}.get(search_type, 'tag')
# default
codes = search_code(code_base_name, query, code_limit, search_type=search_type, history_node_list=history_node_list)
return_codes = []
context = codes['context']
related_nodes = codes['related_vertices']
logger.debug(f"{code_base_name}, {query}, {code_limit}, {search_type}")
logger.debug(f"context: {context}, related_nodes: {related_nodes}")
return_codes.append({'index': 0, 'code': context, "related_nodes": related_nodes})
for idx, code in enumerate(related_code):
return_codes.append({'index': idx, 'code': code, "related_nodes": related_nodes})
return return_codes

5
dev_opsgpt/tools/multiplier.py
View File

@ -32,7 +32,4 @@ class Multiplier(BaseToolModel):
@staticmethod
def run(a, b):
return a * b
def multi_run(a, b):
return a * b
return a * b

96
dev_opsgpt/tools/ocr_tool.py Normal file
View File

@ -0,0 +1,96 @@
from pydantic import BaseModel, Field
from typing import List, Dict
import requests
import base64
import urllib
import os
from loguru import logger
from .base_tool import BaseToolModel
from configs.model_config import JUPYTER_WORK_PATH
class BaiduOcrTool(BaseToolModel):
"""
Tips:
百度ocr tool
example：
API_KEY = ""
SECRET_KEY = ""
image_path = ''
ocr_result = BaiduOcrTool.run(API_KEY=API_KEY , SECRET_KEY=SECRET_KEY, image_path=image_path)
"""
name: str = "Baidu_orc_tool"
description: str = """ 百度OCR手写字符识别调用器。 输入一张图片位置，返回图片中的文本"""
class ToolInputArgs(BaseModel):
"""Input for Multiplier."""
image_name : str = Field(..., description="待提取文本信息的图片名称")
class ToolOutputArgs(BaseModel):
"""Output for Multiplier."""
ocr_result: str = Field(..., description="OCR分析提取的自然语言文本")
@classmethod
def ocr_baidu_main(cls, API_KEY, SECRET_KEY, image_path):
'''
根据图片地址，返回OCR识别结果
OCR的结果不仅包含了文字，也包含了文字的位置。但可以根据简单的提取方法，只将文字提前取出来
下面是ocr的返回结果
'{"words_result":[{"location":{"top":17,"left":33,"width":227,"height":24},"words":"手写识别测试图片样例:"},
{"location":{"top":91,"left":190,"width":713,"height":70},"words":"每一个人的生命中,都应该有一次,"},
{"location":{"top":177,"left":87,"width":831,"height":65},"words":"为了某个人而忘了自己,不求有结果."},
{"location":{"top":263,"left":80,"width":842,"height":76},"words":"不求同行,不求曾经拥有,甚至不求"}],
"words_result_num":4,"log_id":1722502064951792680}'
'''
url = "https://aip.baidubce.com/rest/2.0/ocr/v1/handwriting?access_token=" + BaiduOcrTool.get_access_token(API_KEY, SECRET_KEY)
# image 可以通过 get_file_content_as_base64("C:\fakepath\ocr_input_example.png",True) 方法获取
image = BaiduOcrTool.get_file_content_as_base64(image_path, True)
payload = 'image=' + image + '&detect_direction=false&probability=false'
headers = {
'Content-Type': 'application/x-www-form-urlencoded',
'Accept': 'application/json'
}
response = requests.request("POST", url, headers=headers, data=payload)
s = ""
try:
for word_result in response.json()["words_result"]:
s += "\n" + word_result["words"]
except Exception as e:
logger.exception(e)
s = "无法识别图片内容"
return s
@classmethod
def get_file_content_as_base64(cls, image_path, urlencoded=False):
"""
获取文件base64编码
:param path: 文件路径
:param urlencoded: 是否对结果进行urlencoded
:return: base64编码信息
"""
with open(image_path, "rb") as f:
content = base64.b64encode(f.read()).decode("utf8")
if urlencoded:
content = urllib.parse.quote_plus(content)
return content
@classmethod
def get_access_token(cls, API_KEY, SECRET_KEY):
"""
使用 AK，SK 生成鉴权签名（Access Token）
:return: access_token，或是None(如果错误)
"""
url = "https://aip.baidubce.com/oauth/2.0/token"
params = {"grant_type": "client_credentials", "client_id": API_KEY, "client_secret": SECRET_KEY}
return str(requests.post(url, params=params).json().get("access_token"))
@classmethod
def run(cls, image_name, image_path=JUPYTER_WORK_PATH, API_KEY=os.environ.get("BAIDU_OCR_API_KEY"), SECRET_KEY=os.environ.get("BAIDU_OCR_SECRET_KEY")):
image_file = os.path.join(image_path, image_name)
return cls.ocr_baidu_main(API_KEY, SECRET_KEY, image_file)

189
dev_opsgpt/tools/stock_tool.py Normal file
View File

@ -0,0 +1,189 @@
import json
import os
import re
from pydantic import BaseModel, Field
from typing import List, Dict, Optional
import requests
import numpy as np
from loguru import logger
from .base_tool import BaseToolModel
from dev_opsgpt.utils.common_utils import read_json_file
cur_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
stock_infos = read_json_file(os.path.join(cur_dir, "../sources/tool_datas/stock.json"))
stock_dict = {i["mc"]: i["jys"]+i["dm"] for i in stock_infos}
class StockName(BaseToolModel):
"""
Tips
"""
name: str = "StockName"
description: str = "通过股票名称查询股票代码"
class ToolInputArgs(BaseModel):
"""Input for StockName"""
stock_name: int = Field(..., description="股票名称")
class ToolOutputArgs(BaseModel):
"""Output for StockName"""
stock_code: str = Field(..., description="股票代码")
@staticmethod
def run(stock_name: str):
return stock_dict.get(stock_name, "no stock_code")
class StockInfo(BaseToolModel):
"""
用于查询股票市场数据的StockInfo工具。
"""
name: str = "StockInfo"
description: str = "根据提供的股票代码、日期范围和数据频率提供股票市场数据。"
class ToolInputArgs(BaseModel):
"""StockInfo的输入参数。"""
code: str = Field(..., description="要查询的股票代码，格式为'marketcode'")
end_date: Optional[str] = Field(default="", description="数据查询的结束日期。留空则为当前日期。如果没有提供结束日期，就留空")
count: int = Field(default=10, description="返回数据点的数量。")
frequency: str = Field(default='1d', description="数据点的频率，例如，'1d'表示每日，'1w'表示每周，'1M'表示每月，'1m'表示每分钟等。")
class ToolOutputArgs(BaseModel):
"""StockInfo的输出参数。"""
data: dict = Field(default=None, description="查询到的股票市场数据。")
@staticmethod
def run(code: str, count: int, frequency: str, end_date: Optional[str]="") -> "ToolOutputArgs":
"""执行股票数据查询工具。"""
# 该方法封装了调用底层股票数据API的逻辑，并将结果格式化为pandas DataFrame。
try:
df = get_price(code, end_date=end_date, count=count, frequency=frequency)
# 将DataFrame转换为输出的字典格式
data = df.reset_index().to_dict(orient='list') # 将dataframe转换为字典列表
return StockInfo.ToolOutputArgs(data=data)
except Exception as e:
logger.exception("获取股票数据时发生错误。")
return e
#-*- coding:utf-8 -*- --------------Ashare 股票行情数据双核心版( https://github.com/mpquant/Ashare )
import json,requests,datetime
import pandas as pd #
#腾讯日线
def get_price_day_tx(code, end_date='', count=10, frequency='1d'): #日线获取
unit='week' if frequency in '1w' else 'month' if frequency in '1M' else 'day' #判断日线，周线，月线
if end_date: end_date=end_date.strftime('%Y-%m-%d') if isinstance(end_date,datetime.date) else end_date.split(' ')[0]
end_date='' if end_date==datetime.datetime.now().strftime('%Y-%m-%d') else end_date #如果日期今天就变成空
URL=f'http://web.ifzq.gtimg.cn/appstock/app/fqkline/get?param={code},{unit},,{end_date},{count},qfq'
st= json.loads(requests.get(URL).content); ms='qfq'+unit; stk=st['data'][code]
buf=stk[ms] if ms in stk else stk[unit] #指数返回不是qfqday,是day
df=pd.DataFrame(buf,columns=['time','open','close','high','low','volume'],dtype='float')
df.time=pd.to_datetime(df.time); df.set_index(['time'], inplace=True); df.index.name='' #处理索引
return df
#腾讯分钟线
def get_price_min_tx(code, end_date=None, count=10, frequency='1d'): #分钟线获取
ts=int(frequency[:-1]) if frequency[:-1].isdigit() else 1 #解析K线周期数
if end_date: end_date=end_date.strftime('%Y-%m-%d') if isinstance(end_date,datetime.date) else end_date.split(' ')[0]
URL=f'http://ifzq.gtimg.cn/appstock/app/kline/mkline?param={code},m{ts},,{count}'
st= json.loads(requests.get(URL).content); buf=st[ 'data'][code]['m'+str(ts)]
df=pd.DataFrame(buf,columns=['time','open','close','high','low','volume','n1','n2'])
df=df[['time','open','close','high','low','volume']]
df[['open','close','high','low','volume']]=df[['open','close','high','low','volume']].astype('float')
df.time=pd.to_datetime(df.time); df.set_index(['time'], inplace=True); df.index.name='' #处理索引
df['close'][-1]=float(st['data'][code]['qt'][code][3]) #最新基金数据是3位的
return df
#sina新浪全周期获取函数，分钟线 5m,15m,30m,60m 日线1d=240m 周线1w=1200m 1月=7200m
def get_price_sina(code, end_date='', count=10, frequency='60m'): #新浪全周期获取函数
frequency=frequency.replace('1d','240m').replace('1w','1200m').replace('1M','7200m'); mcount=count
ts=int(frequency[:-1]) if frequency[:-1].isdigit() else 1 #解析K线周期数
if (end_date!='') & (frequency in ['240m','1200m','7200m']):
end_date=pd.to_datetime(end_date) if not isinstance(end_date,datetime.date) else end_date #转换成datetime
unit=4 if frequency=='1200m' else 29 if frequency=='7200m' else 1 #4,29多几个数据不影响速度
count=count+(datetime.datetime.now()-end_date).days//unit #结束时间到今天有多少天自然日(肯定 >交易日)
#print(code,end_date,count)
URL=f'http://money.finance.sina.com.cn/quotes_service/api/json_v2.php/CN_MarketData.getKLineData?symbol={code}&scale={ts}&ma=5&datalen={count}'
dstr= json.loads(requests.get(URL).content);
#df=pd.DataFrame(dstr,columns=['day','open','high','low','close','volume'],dtype='float')
df= pd.DataFrame(dstr,columns=['day','open','high','low','close','volume'])
df['open'] = df['open'].astype(float); df['high'] = df['high'].astype(float); #转换数据类型
df['low'] = df['low'].astype(float); df['close'] = df['close'].astype(float); df['volume'] = df['volume'].astype(float)
df.day=pd.to_datetime(df.day);
df.set_index(['day'], inplace=True);
df.index.name='' #处理索引
if (end_date!='') & (frequency in ['240m','1200m','7200m']):
return df[df.index<=end_date][-mcount:] #日线带结束时间先返回
return df
def get_price(code, end_date='',count=10, frequency='1d', fields=[]): #对外暴露只有唯一函数，这样对用户才是最友好的
xcode= code.replace('.XSHG','').replace('.XSHE','') #证券代码编码兼容处理
xcode='sh'+xcode if ('XSHG' in code) else 'sz'+xcode if ('XSHE' in code) else code
if frequency in ['1d','1w','1M']: #1d日线 1w周线 1M月线
try:
return get_price_sina( xcode, end_date=end_date,count=count,frequency=frequency) #主力
except:
return get_price_day_tx(xcode,end_date=end_date,count=count,frequency=frequency) #备用
if frequency in ['1m','5m','15m','30m','60m']: #分钟线 ,1m只有腾讯接口 5分钟5m 60分钟60m
if frequency in '1m':
return get_price_min_tx(xcode,end_date=end_date,count=count,frequency=frequency)
try:
return get_price_sina(xcode,end_date=end_date,count=count,frequency=frequency) #主力
except:
return get_price_min_tx(xcode,end_date=end_date,count=count,frequency=frequency) #备用
if __name__ == "__main__":
tool = StockInfo()
output = tool.run(code='sh600519', end_date='', count=10, frequency='15m')
print(output.json(indent=2))

4
dev_opsgpt/utils/common_utils.py
View File

@ -73,14 +73,14 @@ def save_to_json_file(data, filename):
def file_normalize(file: Union[str, Path, bytes], filename=None):
logger.debug(f"{file}")
# logger.debug(f"{file}")
if isinstance(file, bytes): # raw bytes
file = BytesIO(file)
elif hasattr(file, "read"): # a file io like object
filename = filename or file.name
else: # a local path
file = Path(file).absolute().open("rb")
logger.debug(file)
# logger.debug(file)
filename = filename or file.name
return file, filename

Some files were not shown because too many files have changed in this diff Show More