Merge pull request #6 from codefuse-ai/ma_cb_commit

add features code answer and multi-agents by markdown

.gitignore

@@ -5,6 +5,9 @@ embedding_models
 jupyter_work
 model_config.py
 server_config.py
+internal_start.py
 code_base
 .DS_Store
 .idea
+data
+tests

Dockerfile

@@ -12,6 +12,8 @@ RUN apt-get install -y iputils-ping telnetd net-tools vim tcpdump
 # RUN service inetutils-inetd start
 # service inetutils-inetd status
 
+RUN wget https://oss-cdn.nebula-graph.com.cn/package/3.6.0/nebula-graph-3.6.0.ubuntu1804.amd64.deb
+RUN dpkg -i nebula-graph-3.6.0.ubuntu1804.amd64.deb
 
 RUN pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple
 RUN pip install -r /home/user/docker_requirements.txt

README.md

@@ -10,10 +10,11 @@
     <br><br>
 </p>
 
-本项目是一个开源的 AI 智能助手，专为软件开发的全生命周期而设计，涵盖设计、编码、测试、部署和运维等阶段。通过知识检索、代码检索，工具使用和沙箱执行，Codefuse-ChatBot 能解答您开发过程中的各种专业问题、问答操作周边独立分散平台。
+本项目是一个开源的 AI 智能助手，专为软件开发的全生命周期而设计，涵盖设计、编码、测试、部署和运维等阶段。通过知识检索、代码检索，工具使用和沙箱执行，Codefuse-ChatBot不仅能回答您在开发过程中遇到的专业问题，还能通过对话界面协调多个独立分散的平台。
 
 
 ## 🔔 更新
+- [2023.12.01] Multi-Agent和代码库检索功能开放
 - [2023.11.15] 增加基于本地代码库的问答增强模式
 - [2023.09.15] 本地/隔离环境的沙盒功能开放，基于爬虫实现指定url知识检索
 
@@ -29,13 +30,12 @@
 
 💡 本项目旨在通过检索增强生成（Retrieval Augmented Generation，RAG）、工具学习（Tool Learning）和沙盒环境来构建软件开发全生命周期的AI智能助手，涵盖设计、编码、测试、部署和运维等阶段。 逐渐从各处资料查询、独立分散平台操作的传统开发运维模式转变到大模型问答的智能化开发运维模式，改变人们的开发运维习惯。
 
-- 📚 知识库管理：DevOps专业高质量知识库 + 企业级知识库自助构建 + 对话实现快速检索开源/私有技术文档
-- ⌨️ 代码知识库管理：支持本地代码库导入和代码结构解析 + 对话实现快速检索本地代码
-- 🐳 隔离沙盒环境：实现代码的快速编译执行测试
-- 🔄 React范式：支撑代码的自我迭代、自动执行
-- 🛠️ Prompt管理：实现各种开发、运维任务的prompt管理
-- 🔌 丰富的领域插件：执行各种定制开发任务
-- 🚀 对话驱动：需求设计、系分设计、代码生成、开发测试、部署运维自动化
+本项目核心差异技术、功能点：
+- **🧠 智能调度核心：** 构建了体系链路完善的调度核心，支持多模式一键配置，简化操作流程。
+- **💻 代码整库分析：** 实现了仓库级的代码深入理解，以及项目文件级的代码编写与生成，提升了开发效率。
+- **📄 文档分析增强：** 融合了文档知识库与知识图谱，通过检索和推理增强，为文档分析提供了更深层次的支持。
+- **🔧 垂类专属知识：** 为DevOps领域定制的专属知识库，支持垂类知识库的自助一键构建，便捷实用。
+- **🤖 垂类模型兼容：** 针对DevOps领域的小型模型，保证了与DevOps相关平台的兼容性，促进了技术生态的整合。
 
 🌍 依托于开源的 LLM 与 Embedding 模型，本项目可实现基于开源模型的离线私有部署。此外，本项目也支持 OpenAI API 的调用。
 
@@ -57,26 +57,26 @@
 
 ## 🧭 技术路线
 <div align=center>
-  <img src="sources/docs_imgs/devops-chatbot-module.png" alt="图片" width="600" height="503">
+  <img src="sources/docs_imgs/devops-chatbot-module-v2.png" alt="图片" width="600" height="503">
 </div>
 
-- 🕷️ **Web Crawl**：实现定期网络文档爬取，确保数据的及时性，并依赖于开源社区的持续补充。
-- 🗂️ **DocLoader & TextSplitter**：对从多种来源爬取的数据进行数据清洗、去重和分类，并支持私有文档的导入。
-- 🗄️ **Vector Database**：结合Text Embedding模型对文档进行Embedding并在Milvus中存储。
-- 🔌 **Connector**：作为调度中心，负责LLM与Vector Database之间的交互调度，基于Langchain技术实现。
-- 📝 **Prompt Control**：从开发和运维角度设计，为不同问题分类并为Prompt添加背景，确保答案的可控性和完整性。
-- 💬 **LLM**：默认使用GPT-3.5-turbo，并为私有部署和其他涉及隐私的场景提供专有模型选择。
-- 🔤 **Text Embedding**：默认采用OpenAI的Text Embedding模型，支持私有部署和其他隐私相关场景，并提供专有模型选择。
-- 🚧 **SandBox**：对于生成的输出，如代码，为帮助用户判断其真实性，提供了一个交互验证环境（基于FaaS），并支持用户进行调整。
-
+- 🧠 **Multi-Agent Schedule Core:** 多智能体调度核心，简易配置即可打造交互式智能体。
+- 🕷️ **Multi Source Web Crawl:** 多源网络爬虫，提供对指定 URL 的爬取功能，以搜集所需信息。
+- 🗂️ **Data Processor:** 数据处理器，轻松完成文档载入、数据清洗，及文本切分，整合不同来源的数据。
+- 🔤 **Text Embedding & Index:**：文本嵌入索引，用户可以轻松上传文件进行文档检索，优化文档分析过程。
+- 🗄️ **Vector Database & Graph Database:** 向量与图数据库，提供灵活强大的数据管理解决方案。
+- 📝 **Prompt Control & Management:**：Prompt 控制与管理，精确定义智能体的上下文环境。
+- 🚧 **SandBox:**：沙盒环境，安全地执行代码编译和动作。
+- 💬 **LLM:**：智能体大脑，支持多种开源模型和 LLM 接口。
+- 🛠️ **API Management:：** API 管理工具，实现对开源组件和运维平台的快速集成。
 
 具体实现明细见：[技术路线明细](sources/readme_docs/roadmap.md)
 
 
 
-## 模型接入
+## 🌐 模型接入
 
-有需要接入的model，可以提issue
+如果您需要集成特定的模型，请通过提交issue来告知我们您的需求。
 
 |      model_name    | model_size | gpu_memory | quantize | HFhub | ModelScope |
 | ------------------ | ---------- | ---------- | -------- | ----- | ---------- |
@@ -193,4 +193,4 @@ python start.py
 
 ## 🤗 致谢
 
-本项目基于[langchain-chatchat](https://github.com/chatchat-space/Langchain-Chatchat)和[codebox-api](https://github.com/shroominic/codebox-api)，在此深深感谢他们的开源贡献！
+本项目基于[langchain-chatchat](https://github.com/chatchat-space/Langchain-Chatchat)和[codebox-api](https://github.com/shroominic/codebox-api)，在此深深感谢他们的开源贡献！

README_en.md

@@ -9,16 +9,18 @@
     </a>
     <br><br>
 </p>
-This project is an open-source AI intelligent assistant, specifically designed for the entire lifecycle of software development, covering design, coding, testing, deployment, and operations. Through knowledge retrieval, tool utilization, and sandbox execution, Codefuse-ChatBot can answer various professional questions during your development process and perform question-answering operations on standalone, disparate platforms.
+This project is an open-source AI intelligent assistant, specifically designed for the entire lifecycle of software development, covering design, coding, testing, deployment, and operations. Through knowledge retrieval, tool utilization, and sandbox execution, Codefuse-ChatBot can not only answer professional questions you encounter during the development process but also coordinate multiple independent, dispersed platforms through a conversational interface.
 
 
 ## 🔔 Updates
-- [2023.09.15] Sandbox features for local/isolated environments are now available, implementing specified URL knowledge retrieval based on web crawling.
+- [2023.12.01] Release of Multi-Agent and codebase retrieval functionalities.
+- [2023.11.15] Addition of Q&A enhancement mode based on the local codebase.
+- [2023.09.15] Launch of sandbox functionality for local/isolated environments, enabling knowledge retrieval from specified URLs using web crawlers.
 
 ## 📜 Contents
 - [🤝 Introduction](#-introduction)
 - [🧭 Technical Route](#-technical-route)
-- [🌐 模型接入](#-模型接入)
+- [🌐 Model Integration](#-model-integration)
 - [🚀 Quick Start](#-quick-start)
 - [🤗 Acknowledgements](#-acknowledgements)
 
@@ -26,11 +28,11 @@ This project is an open-source AI intelligent assistant, specifically designed f
 
 💡 The aim of this project is to construct an AI intelligent assistant for the entire lifecycle of software development, covering design, coding, testing, deployment, and operations, through Retrieval Augmented Generation (RAG), Tool Learning, and sandbox environments. It transitions gradually from the traditional development and operations mode of querying information from various sources and operating on standalone, disparate platforms to an intelligent development and operations mode based on large-model Q&A, changing people's development and operations habits.
 
-- 📚 Knowledge Base Management: Professional high-quality Codefuse knowledge base + enterprise-level knowledge base self-construction + dialogue-based fast retrieval of open-source/private technical documents.
-- 🐳 Isolated Sandbox Environment: Enables quick compilation, execution, and testing of code.
-- 🔄 React Paradigm: Supports code self-iteration and automatic execution.
-- 🛠️ Prompt Management: Manages prompts for various development and operations tasks.
-- 🚀 Conversation Driven: Automates requirement design, system analysis design, code generation, development testing, deployment, and operations.
+- **🧠 Intelligent Scheduling Core:** Constructed a well-integrated scheduling core system that supports multi-mode one-click configuration, simplifying the operational process.
+- **💻 Comprehensive Code Repository Analysis:** Achieved in-depth understanding at the repository level and coding and generation at the project file level, enhancing development efficiency.
+- **📄 Enhanced Document Analysis:** Integrated document knowledge bases with knowledge graphs, providing deeper support for document analysis through enhanced retrieval and reasoning.
+- **🔧 Industry-Specific Knowledge:** Tailored a specialized knowledge base for the DevOps domain, supporting the self-service one-click construction of industry-specific knowledge bases for convenience and practicality.
+- **🤖 Compatible Models for Specific Verticals:** Designed small models specifically for the DevOps field, ensuring compatibility with related DevOps platforms and promoting the integration of the technological ecosystem.
 
 🌍 Relying on open-source LLM and Embedding models, this project can achieve offline private deployments based on open-source models. Additionally, this project also supports the use of the OpenAI API.
 
@@ -47,24 +49,25 @@ This project is an open-source AI intelligent assistant, specifically designed f
 
 ## 🧭 Technical Route
 <div align=center>
-  <img src="sources/docs_imgs/devops-chatbot-module.png" alt="Image" width="600" height="503">
+  <img src="sources/docs_imgs/devops-chatbot-module-v2.png" alt="Image" width="600" height="503">
 </div>
 
-- 🕷️ **Web Crawl**: Implements periodic web document crawling to ensure data timeliness and relies on continuous supplementation from the open-source community.
-- 🗂️ **DocLoader & TextSplitter**: Cleans, deduplicates, and categorizes data crawled from various sources and supports the import of private documents.
-- 🗄️ **Vector Database**: Integrates Text Embedding models to embed documents and store them in Milvus.
-- 🔌 **Connector**: Acts as the scheduling center, responsible for coordinating interactions between LLM and Vector Database, implemented based on Langchain technology.
-- 📝 **Prompt Control**: Designs from development and operations perspectives, categorizes different problems, and adds backgrounds to prompts to ensure the controllability and completeness of answers.
-- 💬 **LLM**: Uses GPT-3.5-turbo by default and provides proprietary model options for private deployments and other privacy-related scenarios.
-- 🔤 **Text Embedding**: Uses OpenAI's Text Embedding model by default, supports private deployments and other privacy-related scenarios, and provides proprietary model options.
-- 🚧 **SandBox**: For generated outputs, like code, to help users judge their authenticity, an interactive verification environment is provided (based on FaaS), allowing user adjustments.
+- 🧠 **Multi-Agent Schedule Core:** Easily configurable to create interactive intelligent agents.
+- 🕷️ **Multi Source Web Crawl:** Offers the capability to crawl specified URLs for collecting the required information.
+- 🗂️ **Data Processor:** Effortlessly handles document loading, data cleansing, and text segmentation, integrating data from different sources.
+- 🔤 **Text Embedding & Index:**：Users can easily upload files for document retrieval, optimizing the document analysis process.
+- 🗄️ **Vector Database & Graph Database:** Provides flexible and powerful data management solutions.
+- 📝 **Prompt Control & Management:**：Precisely defines the contextual environment for intelligent agents.
+- 🚧 **SandBox:**：Safely executes code compilation and actions.
+- 💬 **LLM:**：Supports various open-source models and LLM interfaces.
+- 🛠️ **API Management:：** Enables rapid integration of open-source components and operational platforms.
 
 For implementation details, see: [Technical Route Details](sources/readme_docs/roadmap.md)
 
 
-## 模型接入
+## 🌐 Model Integration
 
-有需要接入的model，可以提issue
+If you need to integrate a specific model, please inform us of your requirements by submitting an issue.
 
 |      model_name    | model_size | gpu_memory | quantize | HFhub | ModelScope |
 | ------------------ | ---------- | ---------- | -------- | ----- | ---------- |

configs/__init__.py

@@ -1,4 +1,4 @@
 from .model_config import *
 from .server_config import *
 
-VERSION = "v0.0.1"
+VERSION = "v0.1.0"

configs/model_config.py.example

@@ -1,6 +1,10 @@
 import os
+import sys
 import logging
 import torch
+import openai
+import base64
+from .utils import is_running_in_docker
 # 日志格式
 LOG_FORMAT = "%(asctime)s - %(filename)s[line:%(lineno)d] - %(levelname)s: %(message)s"
 logger = logging.getLogger()
@@ -8,8 +12,11 @@ logger.setLevel(logging.INFO)
 logging.basicConfig(format=LOG_FORMAT)
 
 # os.environ["OPENAI_PROXY"] = "socks5h://127.0.0.1:13659"
+os.environ["API_BASE_URL"] = "http://openai.com/v1/chat/completions"
 os.environ["OPENAI_API_KEY"] = ""
 os.environ["DUCKDUCKGO_PROXY"] = "socks5://127.0.0.1:13659"
+os.environ["BAIDU_OCR_API_KEY"] = ""
+os.environ["BAIDU_OCR_SECRET_KEY"] = ""
 
 import platform
 system_name = platform.system()
@@ -38,6 +45,7 @@ LOCAL_MODEL_DIR = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(_
 embedding_model_dict = {k: f"/home/user/chatbot/embedding_models/{v}" if is_running_in_docker() else f"{LOCAL_MODEL_DIR}/{v}" for k, v in embedding_model_dict.items()}
 
 # 选用的 Embedding 名称
+EMBEDDING_ENGINE = 'openai'
 EMBEDDING_MODEL = "text2vec-base"
 
 # Embedding 模型运行设备
@@ -99,6 +107,20 @@ llm_model_dict = {
 }
 
 
+LOCAL_LLM_MODEL_DIR = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), "llm_models")
+llm_model_dict_c = {}
+for k, v in llm_model_dict.items():
+    v_c = {}
+    for kk, vv in v.items():
+        if k=="local_model_path":
+            v_c[kk] = f"/home/user/chatbot/llm_models/{vv}" if is_running_in_docker() else f"{LOCAL_LLM_MODEL_DIR}/{vv}" 
+        else:
+            v_c[kk] = vv
+    llm_model_dict_c[k] = v_c
+
+llm_model_dict = llm_model_dict_c
+
+
 # LLM 名称
 LLM_MODEL = "gpt-3.5-turbo"
 USE_FASTCHAT = "gpt" not in LLM_MODEL # 判断是否进行fastchat
@@ -129,7 +151,10 @@ JUPYTER_WORK_PATH = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath
 # WEB_CRAWL存储路径
 WEB_CRAWL_PATH = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), "sources/docs")
 
-for _path in [LOG_PATH, SOURCE_PATH, KB_ROOT_PATH, NLTK_DATA_PATH, JUPYTER_WORK_PATH, WEB_CRAWL_PATH]:
+# NEBULA_DATA存储路径
+NELUBA_PATH = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), "data/neluba_data")
+
+for _path in [LOG_PATH, SOURCE_PATH, KB_ROOT_PATH, NLTK_DATA_PATH, JUPYTER_WORK_PATH, WEB_CRAWL_PATH, NELUBA_PATH]:
     if not os.path.exists(_path):
         os.mkdir(_path)
 
@@ -194,6 +219,11 @@ CODE_PROMPT_TEMPLATE = """【指令】根据已知信息来回答问题。
 
 【问题】{question}"""
 
+# 代码解释模版
+CODE_INTERPERT_TEMPLATE = '''{code}
+
+解释一下这段代码'''
+
 # API 是否开启跨域，默认为False，如果需要开启，请设置为True
 # is open cross domain
 OPEN_CROSS_DOMAIN = False

configs/server_config.py.example

@@ -45,6 +45,21 @@ FSCHAT_OPENAI_API = {
     "docker_port": 8888,  # model_config.llm_model_dict中模型配置的api_base_url需要与这里一致。
 }
 
+# nebula conf
+NEBULA_HOST = DEFAULT_BIND_HOST
+NEBULA_PORT = 9669
+NEBULA_STORAGED_PORT = 9779
+NEBULA_USER = 'root'
+NEBULA_PASSWORD = ''
+NEBULA_GRAPH_SERVER = {
+    "host": DEFAULT_BIND_HOST,
+    "port": NEBULA_PORT,
+    "docker_port": NEBULA_PORT
+}
+
+# chroma conf
+CHROMA_PERSISTENT_PATH = '/home/user/chatbot/data/chroma_data'
+
 # sandbox api server
 SANDBOX_CONTRAINER_NAME = "devopsgpt_sandbox"
 SANDBOX_IMAGE_NAME = "devopsgpt:py39"

dev_opsgpt/chat/__init__.py

@@ -2,12 +2,10 @@ from .base_chat import Chat
 from .knowledge_chat import KnowledgeChat
 from .llm_chat import LLMChat
 from .search_chat import SearchChat
-from .tool_chat import ToolChat
-from .data_chat import DataChat
 from .code_chat import CodeChat
 from .agent_chat import AgentChat
 
 
 __all__ = [
-    "Chat", "KnowledgeChat", "LLMChat", "SearchChat", "ToolChat", "DataChat", "CodeChat", "AgentChat"
+    "Chat", "KnowledgeChat", "LLMChat", "SearchChat", "CodeChat", "AgentChat"
 ]

dev_opsgpt/chat/agent_chat.py

@@ -1,10 +1,11 @@
 from fastapi import Body, Request
 from fastapi.responses import StreamingResponse
-from typing import List
+from typing import List, Union, Dict
 from loguru import logger
 import importlib
 import copy
 import json
+from pathlib import Path
 
 from configs.model_config import (
     llm_model_dict, LLM_MODEL, PROMPT_TEMPLATE, 
@@ -18,12 +19,12 @@ from dev_opsgpt.tools import (
 from dev_opsgpt.connector.phase import BasePhase
 from dev_opsgpt.connector.agents import BaseAgent, ReactAgent
 from dev_opsgpt.connector.chains import BaseChain
-from dev_opsgpt.connector.connector_schema import (
+from dev_opsgpt.connector.schema import (
     Message,
     load_phase_configs, load_chain_configs, load_role_configs
     )
-from dev_opsgpt.connector.shcema import Memory
-
+from dev_opsgpt.connector.schema import Memory
+from dev_opsgpt.utils.common_utils import file_normalize
 from dev_opsgpt.chat.utils import History, wrap_done
 from dev_opsgpt.connector.configs import PHASE_CONFIGS, AGETN_CONFIGS, CHAIN_CONFIGS
 
@@ -41,6 +42,7 @@ class AgentChat:
             ) -> None:
         self.top_k = top_k
         self.stream = stream
+        self.chatPhase_dict: Dict[str, BasePhase] = {}
 
     def chat(
             self, 
@@ -67,7 +69,8 @@ class AgentChat:
             custom_chain_configs: dict = Body({}, description="自定义chain配置"),
             custom_role_configs: dict = Body({}, description="自定义role配置"),
             history_node_list: List = Body([], description="代码历史相关节点"),
-            isDetaild: bool = Body([], description="是否输出完整的agent相关内容"),
+            isDetailed: bool = Body(False, description="是否输出完整的agent相关内容"),
+            upload_file: Union[str, Path, bytes] = "",
             **kargs
             ) -> Message:
         
@@ -83,11 +86,21 @@ class AgentChat:
 
         # choose tools
         tools = toLangchainTools([TOOL_DICT[i] for i in choose_tools if i in TOOL_DICT])
+        logger.debug(f"upload_file: {upload_file}")
+
+        if upload_file:
+            upload_file_name = upload_file if upload_file and isinstance(upload_file, str) else upload_file.name
+            for _filename_idx in range(len(upload_file_name), 0, -1):
+                if upload_file_name[:_filename_idx] in query:
+                    query = query.replace(upload_file_name[:_filename_idx], upload_file_name)
+                    break
+
         input_message = Message(
             role_content=query,
             role_type="human",
             role_name="user",
             input_query=query,
+            origin_query=query,
             phase_name=phase_name,
             chain_name=chain_name,
             do_search=do_search,
@@ -101,7 +114,7 @@ class AgentChat:
             tools=tools
         )
         # history memory mangemant
-        history = Memory([
+        history = Memory(messages=[
             Message(role_name=i["role"], role_type=i["role"], role_content=i["content"]) 
             for i in history
             ])
@@ -128,14 +141,18 @@ class AgentChat:
         #         "figures": output_message.figures
         #         }
 
-        def chat_iterator(message: Message, local_memory: Memory, isDetaild=False):
+        def chat_iterator(message: Message, local_memory: Memory, isDetailed=False):
+            step_content = local_memory.to_str_messages(content_key='step_content', filter_roles=["user"])
+            final_content = message.role_content
             result = {
                 "answer": "",
                 "db_docs": [str(doc) for doc in message.db_docs],
                 "search_docs": [str(doc) for doc in message.search_docs],
                 "code_docs":  [str(doc) for doc in message.code_docs],
                 "related_nodes": [doc.get_related_node() for idx, doc in enumerate(message.code_docs) if idx==0],
-                "figures": message.figures
+                "figures": message.figures,
+                "step_content": step_content,
+                "final_content": final_content,
                 }
             
 
@@ -146,8 +163,8 @@ class AgentChat:
                         related_nodes.append(node)
             result["related_nodes"] = related_nodes
             
-            # logger.debug(f"{result['figures'].keys()}")
-            message_str = local_memory.to_str_messages(content_key='step_content') if isDetaild else message.role_content
+            # logger.debug(f"{result['figures'].keys()}, isDetailed: {isDetailed}")
+            message_str = step_content
             if self.stream:
                 for token in message_str:
                     result["answer"] = token
@@ -157,7 +174,139 @@ class AgentChat:
                     result["answer"] += token
                 yield json.dumps(result, ensure_ascii=False)
             
-        return StreamingResponse(chat_iterator(output_message, local_memory, isDetaild), media_type="text/event-stream")
+        return StreamingResponse(chat_iterator(output_message, local_memory, isDetailed), media_type="text/event-stream")
+
+
+    def achat(
+            self, 
+            query: str = Body(..., description="用户输入", examples=["hello"]),
+            phase_name: str = Body(..., description="执行场景名称", examples=["chatPhase"]),
+            chain_name: str = Body(..., description="执行链的名称", examples=["chatChain"]),
+            history: List[History] = Body(
+                [], description="历史对话",
+                examples=[[{"role": "user", "content": "我们来玩成语接龙，我先来，生龙活虎"}]]
+                ),
+            doc_engine_name: str = Body(..., description="知识库名称", examples=["samples"]),
+            search_engine_name: str = Body(..., description="搜索引擎名称", examples=["duckduckgo"]),
+            code_engine_name: str = Body(..., description="代码引擎名称", examples=["samples"]),
+            cb_search_type: str = Body(..., description="代码查询模式", examples=["tag"]),
+            top_k: int = Body(VECTOR_SEARCH_TOP_K, description="匹配向量数"),
+            score_threshold: float = Body(SCORE_THRESHOLD, description="知识库匹配相关度阈值，取值范围在0-1之间，SCORE越小，相关度越高，取到1相当于不筛选，建议设置在0.5左右", ge=0, le=1),
+            stream: bool = Body(False, description="流式输出"),
+            local_doc_url: bool = Body(False, description="知识文件返回本地路径(true)或URL(false)"),
+            choose_tools: List[str] = Body([], description="选择tool的集合"),
+            do_search: bool = Body(False, description="是否进行搜索"),
+            do_doc_retrieval: bool = Body(False, description="是否进行知识库检索"),
+            do_code_retrieval: bool = Body(False, description="是否执行代码检索"),
+            do_tool_retrieval: bool = Body(False, description="是否执行工具检索"),
+            custom_phase_configs: dict = Body({}, description="自定义phase配置"),
+            custom_chain_configs: dict = Body({}, description="自定义chain配置"),
+            custom_role_configs: dict = Body({}, description="自定义role配置"),
+            history_node_list: List = Body([], description="代码历史相关节点"),
+            isDetailed: bool = Body(False, description="是否输出完整的agent相关内容"),
+            upload_file: Union[str, Path, bytes] = "",
+            **kargs
+            ) -> Message:
+        
+        # update configs
+        phase_configs, chain_configs, agent_configs = self.update_configs(
+            custom_phase_configs, custom_chain_configs, custom_role_configs)
+        # choose tools
+        tools = toLangchainTools([TOOL_DICT[i] for i in choose_tools if i in TOOL_DICT])
+        logger.debug(f"upload_file: {upload_file}")
+
+        if upload_file:
+            upload_file_name = upload_file if upload_file and isinstance(upload_file, str) else upload_file.name
+            for _filename_idx in range(len(upload_file_name), 0, -1):
+                if upload_file_name[:_filename_idx] in query:
+                    query = query.replace(upload_file_name[:_filename_idx], upload_file_name)
+                    break
+
+        input_message = Message(
+            role_content=query,
+            role_type="human",
+            role_name="user",
+            input_query=query,
+            origin_query=query,
+            phase_name=phase_name,
+            chain_name=chain_name,
+            do_search=do_search,
+            do_doc_retrieval=do_doc_retrieval,
+            do_code_retrieval=do_code_retrieval,
+            do_tool_retrieval=do_tool_retrieval,
+            doc_engine_name=doc_engine_name, 
+            search_engine_name=search_engine_name,
+            code_engine_name=code_engine_name,
+            cb_search_type=cb_search_type,
+            score_threshold=score_threshold, top_k=top_k,
+            history_node_list=history_node_list,
+            tools=tools
+        )
+        # history memory mangemant
+        history = Memory(messages=[
+            Message(role_name=i["role"], role_type=i["role"], role_content=i["content"]) 
+            for i in history
+            ])
+        # start to execute
+        if  phase_configs[input_message.phase_name]["phase_type"] not in self.chatPhase_dict:
+            phase_class = getattr(PHASE_MODULE, phase_configs[input_message.phase_name]["phase_type"])
+            phase = phase_class(input_message.phase_name,
+                task = input_message.task,
+                phase_config = phase_configs,
+                chain_config = chain_configs,
+                role_config = agent_configs,
+                do_summary=phase_configs[input_message.phase_name]["do_summary"],
+                do_code_retrieval=input_message.do_code_retrieval,
+                do_doc_retrieval=input_message.do_doc_retrieval,
+                do_search=input_message.do_search,
+                )
+            self.chatPhase_dict[phase_configs[input_message.phase_name]["phase_type"]] = phase
+        else:
+            phase = self.chatPhase_dict[phase_configs[input_message.phase_name]["phase_type"]]
+        
+        def chat_iterator(message: Message, local_memory: Memory, isDetailed=False):
+            step_content = local_memory.to_str_messages(content_key='step_content', filter_roles=["user"])
+            step_content = "\n\n".join([f"{v}" for parsed_output in local_memory.get_parserd_output_list() for k, v in parsed_output.items() if k not in ["Action Status"]])
+            final_content = message.role_content
+            result = {
+                "answer": "",
+                "db_docs": [str(doc) for doc in message.db_docs],
+                "search_docs": [str(doc) for doc in message.search_docs],
+                "code_docs":  [str(doc) for doc in message.code_docs],
+                "related_nodes": [doc.get_related_node() for idx, doc in enumerate(message.code_docs) if idx==0],
+                "figures": message.figures,
+                "step_content": step_content,
+                "final_content": final_content,
+                }
+            
+
+            related_nodes, has_nodes = [], [ ]
+            for nodes in result["related_nodes"]:
+                for node in nodes:
+                    if node not in has_nodes:
+                        related_nodes.append(node)
+            result["related_nodes"] = related_nodes
+            
+            # logger.debug(f"{result['figures'].keys()}, isDetailed: {isDetailed}")
+            message_str = step_content
+            if self.stream:
+                for token in message_str:
+                    result["answer"] = token
+                    yield json.dumps(result, ensure_ascii=False)
+            else:
+                for token in message_str:
+                    result["answer"] += token
+                yield json.dumps(result, ensure_ascii=False)
+            
+
+        for output_message, local_memory in phase.astep(input_message, history):
+
+            # logger.debug(f"output_message: {output_message.role_content}")
+            # output_message = Message(**output_message)
+            # local_memory = Memory(**local_memory)
+            for result in chat_iterator(output_message, local_memory, isDetailed):
+                yield result
+
 
     def _chat(self, ):
         pass

dev_opsgpt/chat/code_chat.py

@@ -53,23 +53,23 @@ class CodeChat(Chat):
 
     def _process(self, query: str, history: List[History], model):
         '''process'''
+
         codes_res = search_code(query=query, cb_name=self.engine_name, code_limit=self.code_limit,
+                                search_type=self.cb_search_type,
                                 history_node_list=self.history_node_list)
 
-        codes = codes_res['related_code']
-        nodes = codes_res['related_node']
+        context = codes_res['context']
+        related_vertices = codes_res['related_vertices']
 
         # update node names
-        node_names = [node[0] for node in nodes]
-        self.history_node_list.extend(node_names)
-        self.history_node_list = list(set(self.history_node_list))
+        # node_names = [node[0] for node in nodes]
+        # self.history_node_list.extend(node_names)
+        # self.history_node_list = list(set(self.history_node_list))
 
-        context = "\n".join(codes)
         source_nodes = []
 
-        for inum, node_info in enumerate(nodes[0:5]):
-            node_name, node_type, node_score = node_info[0], node_info[1], node_info[2]
-            source_nodes.append(f'{inum + 1}. 节点名为 {node_name}, 节点类型为 `{node_type}`, 节点得分为 `{node_score}`')
+        for inum, node_name in enumerate(related_vertices[0:5]):
+            source_nodes.append(f'{inum + 1}. 节点名: `{node_name}`')
 
         logger.info('history={}'.format(history))
         logger.info('message={}'.format([i.to_msg_tuple() for i in history] + [("human", CODE_PROMPT_TEMPLATE)]))
@@ -90,6 +90,7 @@ class CodeChat(Chat):
                 ),
             engine_name: str = Body(..., description="知识库名称", examples=["samples"]),
             code_limit: int = Body(1, examples=['1']),
+            cb_search_type: str = Body('', examples=['1']),
             stream: bool = Body(False, description="流式输出"),
             local_doc_url: bool = Body(False, description="知识文件返回本地路径(true)或URL(false)"),
             request: Request = None,
@@ -100,6 +101,7 @@ class CodeChat(Chat):
         self.stream = stream if isinstance(stream, bool) else stream.default
         self.local_doc_url = local_doc_url if isinstance(local_doc_url, bool) else local_doc_url.default
         self.request = request
+        self.cb_search_type = cb_search_type
         return self._chat(query, history, **kargs)
 
     def _chat(self, query: str, history: List[History], **kargs):

dev_opsgpt/chat/data_chat.py

@@ -1,229 +0,0 @@
-import asyncio
-from typing import List
-
-from langchain import LLMChain
-from langchain.callbacks import AsyncIteratorCallbackHandler
-from langchain.prompts.chat import ChatPromptTemplate
-from langchain.agents import AgentType, initialize_agent
-
-from dev_opsgpt.tools import (
-    WeatherInfo, WorldTimeGetTimezoneByArea, Multiplier,
-    toLangchainTools, get_tool_schema
-    )
-from .utils import History, wrap_done
-from .base_chat import Chat
-from loguru import logger
-import json, re
-
-from dev_opsgpt.sandbox import PyCodeBox, CodeBoxResponse
-from configs.server_config import SANDBOX_SERVER
-
-def get_tool_agent(tools, llm):
-    return initialize_agent(
-        tools,
-        llm,
-        agent=AgentType.STRUCTURED_CHAT_ZERO_SHOT_REACT_DESCRIPTION,
-        verbose=True,
-    )
-
-PROMPT_TEMPLATE = """
-`角色`
-你是一个数据分析师，借鉴下述步骤，逐步完成数据分析任务的拆解和代码编写，尽可能帮助和准确地回答用户的问题。
-
-数据文件的存放路径为 `./`
-
-`数据分析流程`
-- 判断文件是否存在，并读取文件数据
-- 输出数据的基本信息，包括但不限于字段、文本、数据类型等
-- 输出数据的详细统计信息
-- 判断是否需要画图分析，选择合适的字段进行画图
-- 判断数据是否需要进行清洗
-- 判断数据或图片是否需要保存
-...
-- 结合数据统计分析结果和画图结果，进行总结和分析这份数据的价值
-
-`要求`
-- 每轮选择一个数据分析流程，需要综合考虑上轮和后续的可能影响
-- 数据分析流程只提供参考，不要拘泥于它的具体流程，要有自己的思考
-- 使用JSON blob来指定一个计划，通过提供task_status关键字（任务状态）、plan关键字（数据分析计划）和code关键字（可执行代码）。
-
-合法的 "task_status" 值: "finished" 表明当前用户问题已被准确回答 或者 "continued" 表明用户问题仍需要进一步分析
- 
-`$JSON_BLOB如下所示`
-```
-{{
-  "task_status": $TASK_STATUS,
-  "plan": $PLAN,
-  "code": ```python\n$CODE```
-}}
-```
-
-`跟随如下示例`
-问题: 输入待回答的问题
-行动：$JSON_BLOB
-
-... (重复 行动 N 次，每次只生成一个行动)
-
-行动:
-```
-{{
-  "task_status": "finished",
-  "plan": 我已经可以回答用户问题了，最后回答用户的内容
-}}
-
-```
-
-`数据分析，开始`
-
-问题：{query}
-"""
-
-
-PROMPT_TEMPLATE_2 = """
-`角色`
-你是一个数据分析师，借鉴下述步骤，逐步完成数据分析任务的拆解和代码编写，尽可能帮助和准确地回答用户的问题。
-
-数据文件的存放路径为 `./`
-
-`数据分析流程`
-- 判断文件是否存在，并读取文件数据
-- 输出数据的基本信息，包括但不限于字段、文本、数据类型等
-- 输出数据的详细统计信息
-- 判断数据是否需要进行清洗
-- 判断是否需要画图分析，选择合适的字段进行画图
-- 判断清洗后数据或图片是否需要保存
-...
-- 结合数据统计分析结果和画图结果，进行总结和分析这份数据的价值
-
-`要求`
-- 每轮选择一个数据分析流程，需要综合考虑上轮和后续的可能影响
-- 数据分析流程只提供参考，不要拘泥于它的具体流程，要有自己的思考
-- 使用JSON blob来指定一个计划，通过提供task_status关键字（任务状态）、plan关键字（数据分析计划）和code关键字（可执行代码）。
-
-合法的 "task_status" 值: "finished" 表明当前用户问题已被准确回答 或者 "continued" 表明用户问题仍需要进一步分析
- 
-`$JSON_BLOB如下所示`
-```
-{{
-  "task_status": $TASK_STATUS,
-  "plan": $PLAN,
-  "code": ```python\n$CODE```
-}}
-```
-
-`跟随如下示例`
-问题: 输入待回答的问题
-行动：$JSON_BLOB
-
-... (重复 行动 N 次，每次只生成一个行动)
-
-行动:
-```
-{{
-  "task_status": "finished",
-  "plan": 我已经可以回答用户问题了，最后回答用户的内容
-}}
-
-`数据分析，开始`
-
-问题：上传了一份employee_data.csv文件，请对它进行数据分析
-
-问题：{query}
-{history}
-
-"""
-
-class DataChat(Chat):
-
-    def __init__(
-            self,
-            engine_name: str = "",
-            top_k: int = 1,
-            stream: bool = False,
-            ) -> None:
-        super().__init__(engine_name, top_k, stream)
-        self.tool_prompt = """结合上下文信息，{tools} {input}"""
-        self.codebox = PyCodeBox(
-            remote_url=SANDBOX_SERVER["url"],
-            remote_ip=SANDBOX_SERVER["host"], # "http://localhost",
-            remote_port=SANDBOX_SERVER["port"],
-            token="mytoken",
-            do_code_exe=True,
-            do_remote=SANDBOX_SERVER["do_remote"]
-            )
-
-    def create_task(self, query: str, history: List[History], model):
-        '''构建 llm 生成任务'''
-        logger.debug("content:{}".format([i.to_msg_tuple() for i in history] + [("human", PROMPT_TEMPLATE)]))
-        chat_prompt = ChatPromptTemplate.from_messages(
-            [i.to_msg_tuple() for i in history] + [("human", PROMPT_TEMPLATE)]
-        )
-        pattern = re.compile(r"```(?:json)?\n(.*?)\n", re.DOTALL)
-        internal_history = []
-        retry_nums = 2
-        while retry_nums >= 0:
-            if len(internal_history) == 0:
-                chat_prompt = ChatPromptTemplate.from_messages(
-                    [i.to_msg_tuple() for i in history] + [("human", PROMPT_TEMPLATE)]
-                )
-            else:
-                chat_prompt = ChatPromptTemplate.from_messages(
-                    [i.to_msg_tuple() for i in history] + [("human", PROMPT_TEMPLATE_2)]
-                )
-                
-            chain = LLMChain(prompt=chat_prompt, llm=model)
-            content = chain({"query": query, "history": "\n".join(internal_history)})["text"]
-
-            # content = pattern.search(content)
-            # logger.info(f"content: {content}")
-            # content = json.loads(content.group(1).strip(), strict=False)
-
-            internal_history.append(f"{content}")
-            refer_info = "\n".join(internal_history)
-            logger.info(f"refer_info: {refer_info}")
-            try:
-                content = content.split("行动:")[-1].split("行动：")[-1]
-                content = json.loads(content)
-            except:
-                content = content.split("行动:")[-1].split("行动：")[-1]
-                content = eval(content)
-
-            if "finished" == content["task_status"]:
-                break
-            elif "code" in content:
-            # elif "```code" in content or "```python" in content:
-                # code_text = self.codebox.decode_code_from_text(content)
-                code_text = content["code"]
-                codebox_res = self.codebox.chat("```"+code_text+"```", do_code_exe=True)
-
-                if codebox_res is not None and codebox_res.code_exe_status != 200:
-                    logger.warning(f"{codebox_res.code_exe_response}")
-                    internal_history.append(f"观察： 根据这个报错信息 {codebox_res.code_exe_response}，进行代码修复")
-
-                if codebox_res is not None and codebox_res.code_exe_status == 200:
-                    if codebox_res.code_exe_type == "image/png":
-                        base_text = f"```\n{code_text}\n```\n\n"
-                        img_html = "<img src='data:image/png;base64,{}' class='img-fluid'>".format(
-                            codebox_res.code_exe_response
-                        )
-                        internal_history.append(f"观察： {img_html}")
-                        # logger.info('```\n'+code_text+'\n```'+"\n\n"+'```\n'+codebox_res.code_exe_response+'\n```')
-                    else:
-                        internal_history.append(f"观察： {codebox_res.code_exe_response}")
-                        # logger.info('```\n'+code_text+'\n```'+"\n\n"+'```\n'+codebox_res.code_exe_response+'\n```')
-            else:
-                internal_history.append(f"观察：下一步应该怎么做？")
-            retry_nums -= 1
-
-
-        return {"answer": "", "docs": ""}, {"text": "\n".join(internal_history)}
-
-    def create_atask(self, query, history, model, callback: AsyncIteratorCallbackHandler):
-        chat_prompt = ChatPromptTemplate.from_messages(
-            [i.to_msg_tuple() for i in history] + [("human", PROMPT_TEMPLATE)]
-        )
-        chain = LLMChain(prompt=chat_prompt, llm=model)
-        task = asyncio.create_task(wrap_done(
-            chain.acall({"input": query}), callback.done
-        ))
-        return task, {"answer": "", "docs": ""}

dev_opsgpt/chat/tool_chat.py

@@ -1,84 +0,0 @@
-import asyncio
-from typing import List
-
-from langchain import LLMChain
-from langchain.callbacks import AsyncIteratorCallbackHandler
-from langchain.prompts.chat import ChatPromptTemplate
-from langchain.agents import AgentType, initialize_agent
-import langchain
-from langchain.schema import (
-            AgentAction
-        )
-
-
-# langchain.debug = True
-
-from dev_opsgpt.tools import (
-    TOOL_SETS, TOOL_DICT,
-    toLangchainTools, get_tool_schema
-    )
-from .utils import History, wrap_done
-from .base_chat import Chat
-from loguru import logger
-
-
-def get_tool_agent(tools, llm):
-    return initialize_agent(
-        tools,
-        llm,
-        agent=AgentType.STRUCTURED_CHAT_ZERO_SHOT_REACT_DESCRIPTION,
-        verbose=True,
-        return_intermediate_steps=True
-    )
-
-
-class ToolChat(Chat):
-
-    def __init__(
-            self,
-            engine_name: str = "",
-            top_k: int = 1,
-            stream: bool = False,
-            ) -> None:
-        super().__init__(engine_name, top_k, stream)
-        self.tool_prompt = """结合上下文信息，{tools} {input}"""
-        self.tools = toLangchainTools([TOOL_DICT[i] for i in TOOL_SETS if i in TOOL_DICT])
-
-    def create_task(self, query: str, history: List[History], model, **kargs):
-        '''构建 llm 生成任务'''
-        logger.debug("content:{}".format([i.to_msg_tuple() for i in history] + [("human", "{query}")]))
-        # chat_prompt = ChatPromptTemplate.from_messages(
-        #     [i.to_msg_tuple() for i in history] + [("human", "{query}")]
-        # )
-        tools = kargs.get("tool_sets", [])
-        tools = toLangchainTools([TOOL_DICT[i] for i in tools if i in TOOL_DICT])
-        agent = get_tool_agent(tools if tools else self.tools, model)
-        content = agent(query)
-
-        logger.debug(f"content: {content}")
-
-        s = ""
-        if isinstance(content, str):
-            s = content
-        else:
-            for i in content["intermediate_steps"]:
-                for j in i:
-                    if isinstance(j, AgentAction):
-                        s += j.log + "\n"
-                    else:
-                        s += "Observation: " + str(j) + "\n"
-
-            s += "final answer:" + content["output"]
-        # chain = LLMChain(prompt=chat_prompt, llm=model)
-        # content = chain({"tools": tools, "input": query})
-        return {"answer": "", "docs": ""}, {"text": s}
-
-    def create_atask(self, query, history, model, callback: AsyncIteratorCallbackHandler):
-        chat_prompt = ChatPromptTemplate.from_messages(
-            [i.to_msg_tuple() for i in history] + [("human", self.tool_prompt)]
-        )
-        chain = LLMChain(prompt=chat_prompt, llm=model)
-        task = asyncio.create_task(wrap_done(
-            chain.acall({"input": query}), callback.done
-        ))
-        return task, {"answer": "", "docs": ""}

dev_opsgpt/codebase_handler/codebase_handler.py

@@ -1,139 +0,0 @@
-# encoding: utf-8
-'''
-@author: 温进
-@file: codebase_handler.py
-@time: 2023/10/23 下午5:05
-@desc:
-'''
-
-from loguru import logger
-import time
-import os
-
-from dev_opsgpt.codebase_handler.parser.java_paraser.java_crawler import JavaCrawler
-from dev_opsgpt.codebase_handler.parser.java_paraser.java_preprocess import JavaPreprocessor
-from dev_opsgpt.codebase_handler.parser.java_paraser.java_dedup import JavaDedup
-from dev_opsgpt.codebase_handler.parser.java_paraser.java_parser import JavaParser
-from dev_opsgpt.codebase_handler.tagger.tagger import Tagger
-from dev_opsgpt.codebase_handler.tagger.tuple_generation import node_edge_update
-
-from dev_opsgpt.codebase_handler.networkx_handler.networkx_handler import NetworkxHandler
-from dev_opsgpt.codebase_handler.codedb_handler.local_codedb_handler import LocalCodeDBHandler
-
-
-class CodeBaseHandler():
-    def __init__(self, code_name: str, code_path: str = '', cb_root_path: str = '', history_node_list: list = []):
-        self.nh = None
-        self.lcdh = None
-        self.code_name = code_name
-        self.code_path = code_path
-
-        self.codebase_path = cb_root_path + os.sep + code_name
-        self.graph_path = self.codebase_path + os.sep + 'graph.pk'
-        self.codedb_path = self.codebase_path + os.sep + 'codedb.pk'
-
-        self.tagger = Tagger()
-        self.history_node_list = history_node_list
-
-    def import_code(self, do_save: bool=False, do_load: bool=False) -> bool:
-        '''
-        import code to codeBase
-        @param code_path:
-        @param do_save:
-        @param do_load:
-        @return: True as success; False as failure
-        '''
-        if do_load:
-            logger.info('start load from codebase_path')
-            load_graph_path = self.graph_path
-            load_codedb_path = self.codedb_path
-
-            st = time.time()
-            self.nh = NetworkxHandler(graph_path=load_graph_path)
-            logger.info('generate graph success, rt={}'.format(time.time() - st))
-
-            st = time.time()
-            self.lcdh = LocalCodeDBHandler(db_path=load_codedb_path)
-            logger.info('generate codedb success, rt={}'.format(time.time() - st))
-        else:
-            logger.info('start load from code_path')
-            st = time.time()
-            java_code_dict = JavaCrawler.local_java_file_crawler(self.code_path)
-            logger.info('crawl success, rt={}'.format(time.time() - st))
-
-            jp = JavaPreprocessor()
-            java_code_dict = jp.preprocess(java_code_dict)
-
-            jd = JavaDedup()
-            java_code_dict = jd.dedup(java_code_dict)
-
-            st = time.time()
-            j_parser = JavaParser()
-            parse_res = j_parser.parse(java_code_dict)
-            logger.info('parse success, rt={}'.format(time.time() - st))
-
-            st = time.time()
-            tagged_code = self.tagger.generate_tag(parse_res)
-            node_list, edge_list = node_edge_update(parse_res.values())
-            logger.info('get node and edge success, rt={}'.format(time.time() - st))
-
-            st = time.time()
-            self.nh = NetworkxHandler(node_list=node_list, edge_list=edge_list)
-            logger.info('generate graph success, rt={}'.format(time.time() - st))
-
-            st = time.time()
-            self.lcdh = LocalCodeDBHandler(tagged_code)
-            logger.info('CodeDB load success, rt={}'.format(time.time() - st))
-
-            if do_save:
-                save_graph_path = self.graph_path
-                save_codedb_path = self.codedb_path
-                self.nh.save_graph(save_graph_path)
-                self.lcdh.save_db(save_codedb_path)
-
-    def search_code(self, query: str, code_limit: int, history_node_list: list = []):
-        '''
-        search code related to query
-        @param self:
-        @param query:
-        @return:
-        '''
-        # get query tag
-        query_tag_list = self.tagger.generate_tag_query(query)
-
-        related_node_score_list = self.nh.search_node_with_score(query_tag_list=query_tag_list,
-                                                                 history_node_list=history_node_list)
-
-        score_dict = {
-            i[0]: i[1]
-            for i in related_node_score_list
-        }
-        related_node = [i[0] for i in related_node_score_list]
-        related_score = [i[1] for i in related_node_score_list]
-
-        related_code, code_related_node = self.lcdh.search_by_multi_tag(related_node, lim=code_limit)
-
-        related_node = [
-            (node, self.nh.get_node_type(node), score_dict[node])
-            for node in code_related_node
-        ]
-
-        related_node.sort(key=lambda x: x[2], reverse=True)
-
-        logger.info('related_node={}'.format(related_node))
-        logger.info('related_code={}'.format(related_code))
-        logger.info('num of code={}'.format(len(related_code)))
-        return related_code, related_node
-
-    def refresh_history(self):
-        self.history_node_list = []
-
-
-
-
-
-
-
-
-
-

dev_opsgpt/codebase_handler/codedb_handler/local_codedb_handler.py

@@ -1,55 +0,0 @@
-# encoding: utf-8
-'''
-@author: 温进
-@file: local_codedb_handler.py
-@time: 2023/10/23 下午5:05
-@desc:
-'''
-import pickle
-
-
-class LocalCodeDBHandler:
-    def __init__(self, tagged_code: dict = {}, db_path: str = ''):
-        if db_path:
-            with open(db_path, 'rb') as f:
-                self.data = pickle.load(f)
-        else:
-            self.data = {}
-            for code, tag in tagged_code.items():
-                self.data[code] = str(tag)
-
-    def search_by_single_tag(self, tag, lim):
-        res = list()
-        for k, v in self.data.items():
-            if tag in v and k not in res:
-                res.append(k)
-
-                if len(res) > lim:
-                    break
-        return res
-
-    def search_by_multi_tag(self, tag_list, lim=3):
-        res = list()
-        res_related_node = []
-        for tag in tag_list:
-            single_tag_res = self.search_by_single_tag(tag, lim)
-            for code in single_tag_res:
-                if code not in res:
-                    res.append(code)
-                    res_related_node.append(tag)
-            if len(res) >= lim:
-                break
-
-        # reverse order so that most relevant one is close to the query
-        res = res[0:lim]
-        res.reverse()
-
-        return res, res_related_node
-
-    def save_db(self, save_path):
-        with open(save_path, 'wb') as f:
-            pickle.dump(self.data, f)
-
-    def __len__(self):
-        return len(self.data)
-

dev_opsgpt/codebase_handler/networkx_handler/networkx_handler.py

@@ -1,129 +0,0 @@
-# encoding: utf-8
-'''
-@author: 温进
-@file: networkx_handler.py
-@time: 2023/10/23 下午5:02
-@desc:
-'''
-
-import networkx as nx
-from loguru import logger
-import matplotlib.pyplot as plt
-import pickle
-from collections import defaultdict
-import json
-
-QUERY_SCORE = 10
-HISTORY_SCORE = 5
-RATIO = 0.5
-
-
-class NetworkxHandler:
-    def __init__(self, graph_path: str = '', node_list: list = [], edge_list: list = []):
-        if graph_path:
-            self.graph_path = graph_path
-            with open(graph_path, 'r') as f:
-                self.G = nx.node_link_graph(json.load(f))
-        else:
-            self.G = nx.DiGraph()
-            self.populate_graph(node_list, edge_list)
-        logger.debug(
-            'number of nodes={}, number of edges={}'.format(self.G.number_of_nodes(), self.G.number_of_edges()))
-
-        self.query_score = QUERY_SCORE
-        self.history_score = HISTORY_SCORE
-        self.ratio = RATIO
-
-    def populate_graph(self, node_list, edge_list):
-        '''
-        populate graph with node_list and edge_list
-        '''
-        self.G.add_nodes_from(node_list)
-        for edge in edge_list:
-            self.G.add_edge(edge[0], edge[-1], relation=edge[1])
-
-    def draw_graph(self, save_path: str):
-        '''
-        draw and save to save_path
-        '''
-        sub = plt.subplot(111)
-        nx.draw(self.G, with_labels=True)
-
-        plt.savefig(save_path)
-
-    def search_node(self, query_tag_list: list, history_node_list: list = []):
-        '''
-        search node by tag_list, search from history_tag neighbors first
-        > query_tag_list: tag from query
-        > history_node_list
-        '''
-        node_list = set()
-
-        # search from history_tag_list first, then all nodes
-        for tag in query_tag_list:
-            add = False
-            for history_node in history_node_list:
-                connect_node_list: list = self.G.adj[history_node]
-                connect_node_list.insert(0, history_node)
-                for connect_node in connect_node_list:
-                    node_name_lim = len(connect_node) if '_' not in connect_node else connect_node.index('_')
-                    node_name = connect_node[0:node_name_lim]
-                    if tag.lower() in node_name.lower():
-                        node_list.add(connect_node)
-                        add = True
-            if not add:
-                for node in self.G.nodes():
-                    if tag.lower() in node.lower():
-                        node_list.add(node)
-        return node_list
-
-    def search_node_with_score(self, query_tag_list: list, history_node_list: list = []):
-        '''
-        search node by tag_list, search from history_tag neighbors first
-        > query_tag_list: tag from query
-        > history_node_list
-        '''
-        logger.info('query_tag_list={}, history_node_list={}'.format(query_tag_list, history_node_list))
-        node_dict = defaultdict(lambda: 0)
-
-        # loop over query_tag_list and add node:
-        for tag in query_tag_list:
-            for node in self.G.nodes:
-                if tag.lower() in node.lower():
-                    node_dict[node] += self.query_score
-
-        # loop over history_node and add node score
-        for node in history_node_list:
-            node_dict[node] += self.history_score
-
-        logger.info('temp_res={}'.format(node_dict))
-
-        # adj score broadcast
-        for node in node_dict:
-            adj_node_list = self.G.adj[node]
-            for adj_node in adj_node_list:
-                node_dict[node] += node_dict.get(adj_node, 0) * self.ratio
-
-        # sort
-        node_list = [(node, node_score) for node, node_score in node_dict.items()]
-        node_list.sort(key=lambda x: x[1], reverse=True)
-        return node_list
-
-    def save_graph(self, save_path: str):
-        to_save = nx.node_link_data(self.G)
-        with open(save_path, 'w') as f:
-            json.dump(to_save, f)
-
-    def __len__(self):
-        return self.G.number_of_nodes()
-
-    def get_node_type(self, node_name):
-        node_type = self.G.nodes[node_name]['type']
-        return node_type
-
-    def refresh_graph(self, ):
-        with open(self.graph_path, 'r') as f:
-            self.G = nx.node_link_graph(json.load(f))
-
-
-

dev_opsgpt/codebase_handler/parser/java_paraser/__init__.py

@@ -1,7 +0,0 @@
-# encoding: utf-8
-'''
-@author: 温进
-@file: __init__.py.py
-@time: 2023/10/23 下午5:01
-@desc:
-'''

dev_opsgpt/codebase_handler/parser/java_paraser/java_crawler.py

@@ -1,32 +0,0 @@
-# encoding: utf-8
-'''
-@author: 温进
-@file: java_crawler.py
-@time: 2023/10/23 下午5:02
-@desc:
-'''
-
-import os
-import glob
-from loguru import logger
-
-
-class JavaCrawler:
-    @staticmethod
-    def local_java_file_crawler(path: str):
-        '''
-        read local java file in path
-        > path: path to crawl, must be absolute path like A/B/C
-        < dict of java code string
-        '''
-        java_file_list = glob.glob('{path}{sep}**{sep}*.java'.format(path=path, sep=os.path.sep), recursive=True)
-        java_code_dict = {}
-
-        logger.debug('number of file={}'.format(len(java_file_list)))
-        # logger.debug(java_file_list)
-
-        for java_file in java_file_list:
-            with open(java_file) as f:
-                java_code = ''.join(f.readlines())
-                java_code_dict[java_file] = java_code
-        return java_code_dict

dev_opsgpt/codebase_handler/parser/java_paraser/java_dedup.py

@@ -1,15 +0,0 @@
-# encoding: utf-8
-'''
-@author: 温进
-@file: java_dedup.py
-@time: 2023/10/23 下午5:02
-@desc:
-'''
-
-
-class JavaDedup:
-    def __init__(self):
-        pass
-
-    def dedup(self, java_code_dict):
-        return java_code_dict

dev_opsgpt/codebase_handler/parser/java_paraser/java_preprocess.py

@@ -1,14 +0,0 @@
-# encoding: utf-8
-'''
-@author: 温进
-@file: java_preprocess.py
-@time: 2023/10/23 下午5:04
-@desc:
-'''
-
-class JavaPreprocessor:
-    def __init__(self):
-        pass
-
-    def preprocess(self, java_code_dict):
-        return java_code_dict

dev_opsgpt/codebase_handler/tagger/__init__.py

@@ -1,7 +0,0 @@
-# encoding: utf-8
-'''
-@author: 温进
-@file: __init__.py.py
-@time: 2023/10/23 下午5:00
-@desc:
-'''

dev_opsgpt/codebase_handler/tagger/tagger.py

@@ -1,48 +0,0 @@
-# encoding: utf-8
-'''
-@author: 温进
-@file: tagger.py
-@time: 2023/10/23 下午5:01
-@desc:
-'''
-import re
-from loguru import logger
-
-
-class Tagger:
-    def __init__(self):
-        pass
-
-    def generate_tag(self, parse_res_dict: dict):
-        '''
-        generate tag from parse_res
-        '''
-        res = {}
-        for java_code, parse_res in parse_res_dict.items():
-            tag = {}
-            tag['pac_name'] = parse_res.get('pac_name')
-            tag['class_name'] = set(parse_res.get('class_name_list'))
-            tag['func_name'] = set()
-
-            for _, func_name_list in parse_res.get('func_name_dict', {}).items():
-                tag['func_name'].update(func_name_list)
-
-            res[java_code] = tag
-        return res
-
-    def generate_tag_query(self, query):
-        '''
-        generate tag from query
-        '''
-        # simple extract english
-        tag_list = re.findall(r'[a-zA-Z\_\.]+', query)
-        tag_list = list(set(tag_list))
-        return tag_list
-
-
-if __name__ == '__main__':
-    tagger = Tagger()
-    logger.debug(tagger.generate_tag_query('com.CheckHolder 有哪些函数'))
-
-
-

dev_opsgpt/codebase_handler/tagger/tuple_generation.py

@@ -1,51 +0,0 @@
-# encoding: utf-8
-'''
-@author: 温进
-@file: tuple_generation.py
-@time: 2023/10/23 下午5:01
-@desc:
-'''
-
-
-def node_edge_update(parse_res_list: list, node_list: list = list(), edge_list: list = list()):
-    '''
-    generate node and edge by parse_res
-    < node: list of string node
-    < edge: (node_st, relation, node_ed)
-    '''
-    node_dict = {i: j for i, j in node_list}
-
-    for single_parse_res in parse_res_list:
-        pac_name = single_parse_res['pac_name']
-
-        node_dict[pac_name] = {'type': 'package'}
-
-        # class_name
-        for class_name in single_parse_res['class_name_list']:
-            node_dict[class_name] = {'type': 'class'}
-            edge_list.append((pac_name, 'contain', class_name))
-            edge_list.append((class_name, 'inside', pac_name))
-
-        # func_name
-        for class_name, func_name_list in single_parse_res['func_name_dict'].items():
-            node_list.append(class_name)
-            for func_name in func_name_list:
-                node_dict[func_name] = {'type': 'func'}
-                edge_list.append((class_name, 'contain', func_name))
-                edge_list.append((func_name, 'inside', class_name))
-
-        # depend
-        for depend_pac_name in single_parse_res['import_pac_name_list']:
-            if depend_pac_name.endswith('*'):
-                depend_pac_name = depend_pac_name[0:-2]
-
-            if depend_pac_name in node_dict:
-                continue
-            else:
-                node_dict[depend_pac_name] = {'type': 'unknown'}
-            edge_list.append((pac_name, 'depend', depend_pac_name))
-            edge_list.append((depend_pac_name, 'beDepended', pac_name))
-
-    node_list = [(i, j) for i, j in node_dict.items()]
-
-    return node_list, edge_list

dev_opsgpt/codechat/__init__.py

@@ -2,6 +2,6 @@
 '''
 @author: 温进
 @file: __init__.py.py
-@time: 2023/10/23 下午5:04
+@time: 2023/11/21 下午2:01
 @desc:
 '''

dev_opsgpt/codechat/code_analyzer/__init__.py

@@ -2,6 +2,6 @@
 '''
 @author: 温进
 @file: __init__.py.py
-@time: 2023/10/23 下午5:00
+@time: 2023/11/21 下午2:27
 @desc:
 '''

dev_opsgpt/codechat/code_analyzer/code_analyzer.py

@@ -0,0 +1,219 @@
+# encoding: utf-8
+'''
+@author: 温进
+@file: code_analyzer.py
+@time: 2023/11/21 下午2:27
+@desc:
+'''
+import time
+from loguru import logger
+
+from dev_opsgpt.codechat.code_analyzer.code_static_analysis import CodeStaticAnalysis
+from dev_opsgpt.codechat.code_analyzer.code_intepreter import CodeIntepreter
+from dev_opsgpt.codechat.code_analyzer.code_preprocess import CodePreprocessor
+from dev_opsgpt.codechat.code_analyzer.code_dedup import CodeDedup
+
+
+class CodeAnalyzer:
+    def __init__(self, language: str):
+        self.code_preprocessor = CodePreprocessor()
+        self.code_debup = CodeDedup()
+        self.code_interperter = CodeIntepreter()
+        self.code_static_analyzer = CodeStaticAnalysis(language=language)
+
+    def analyze(self, code_dict: dict, do_interpret: bool = True):
+        '''
+        analyze code
+        @param code_dict: {fp: code_text}
+        @param do_interpret: Whether to get analysis result
+        @return:
+        '''
+        # preprocess and dedup
+        st = time.time()
+        code_dict = self.code_preprocessor.preprocess(code_dict)
+        code_dict = self.code_debup.dedup(code_dict)
+        logger.debug('preprocess and dedup rt={}'.format(time.time() - st))
+
+        # static analysis
+        st = time.time()
+        static_analysis_res = self.code_static_analyzer.analyze(code_dict)
+        logger.debug('static analysis rt={}'.format(time.time() - st))
+
+        # interpretation
+        if do_interpret:
+            logger.info('start interpret code')
+            st = time.time()
+            code_list = list(code_dict.values())
+            interpretation = self.code_interperter.get_intepretation_batch(code_list)
+            logger.debug('interpret rt={}'.format(time.time() - st))
+        else:
+            interpretation = {i: '' for i in code_dict.values()}
+
+        return static_analysis_res, interpretation
+
+
+if __name__ == '__main__':
+    engine = 'openai'
+    language = 'java'
+    code_dict = {'1': '''package com.theokanning.openai.client;
+import com.theokanning.openai.DeleteResult;
+import com.theokanning.openai.OpenAiResponse;
+import com.theokanning.openai.audio.TranscriptionResult;
+import com.theokanning.openai.audio.TranslationResult;
+import com.theokanning.openai.billing.BillingUsage;
+import com.theokanning.openai.billing.Subscription;
+import com.theokanning.openai.completion.CompletionRequest;
+import com.theokanning.openai.completion.CompletionResult;
+import com.theokanning.openai.completion.chat.ChatCompletionRequest;
+import com.theokanning.openai.completion.chat.ChatCompletionResult;
+import com.theokanning.openai.edit.EditRequest;
+import com.theokanning.openai.edit.EditResult;
+import com.theokanning.openai.embedding.EmbeddingRequest;
+import com.theokanning.openai.embedding.EmbeddingResult;
+import com.theokanning.openai.engine.Engine;
+import com.theokanning.openai.file.File;
+import com.theokanning.openai.fine_tuning.FineTuningEvent;
+import com.theokanning.openai.fine_tuning.FineTuningJob;
+import com.theokanning.openai.fine_tuning.FineTuningJobRequest;
+import com.theokanning.openai.finetune.FineTuneEvent;
+import com.theokanning.openai.finetune.FineTuneRequest;
+import com.theokanning.openai.finetune.FineTuneResult;
+import com.theokanning.openai.image.CreateImageRequest;
+import com.theokanning.openai.image.ImageResult;
+import com.theokanning.openai.model.Model;
+import com.theokanning.openai.moderation.ModerationRequest;
+import com.theokanning.openai.moderation.ModerationResult;
+import io.reactivex.Single;
+import okhttp3.MultipartBody;
+import okhttp3.RequestBody;
+import okhttp3.ResponseBody;
+import retrofit2.Call;
+import retrofit2.http.*;
+import java.time.LocalDate;
+public interface OpenAiApi {
+    @GET("v1/models")
+    Single<OpenAiResponse<Model>> listModels();
+    @GET("/v1/models/{model_id}")
+    Single<Model> getModel(@Path("model_id") String modelId);
+    @POST("/v1/completions")
+    Single<CompletionResult> createCompletion(@Body CompletionRequest request);
+    @Streaming
+    @POST("/v1/completions")
+    Call<ResponseBody> createCompletionStream(@Body CompletionRequest request);
+    @POST("/v1/chat/completions")
+    Single<ChatCompletionResult> createChatCompletion(@Body ChatCompletionRequest request);
+    @Streaming
+    @POST("/v1/chat/completions")
+    Call<ResponseBody> createChatCompletionStream(@Body ChatCompletionRequest request);
+    @Deprecated
+    @POST("/v1/engines/{engine_id}/completions")
+    Single<CompletionResult> createCompletion(@Path("engine_id") String engineId, @Body CompletionRequest request);
+    @POST("/v1/edits")
+    Single<EditResult> createEdit(@Body EditRequest request);
+    @Deprecated
+    @POST("/v1/engines/{engine_id}/edits")
+    Single<EditResult> createEdit(@Path("engine_id") String engineId, @Body EditRequest request);
+    @POST("/v1/embeddings")
+    Single<EmbeddingResult> createEmbeddings(@Body EmbeddingRequest request);
+    @Deprecated
+    @POST("/v1/engines/{engine_id}/embeddings")
+    Single<EmbeddingResult> createEmbeddings(@Path("engine_id") String engineId, @Body EmbeddingRequest request);
+    @GET("/v1/files")
+    Single<OpenAiResponse<File>> listFiles();
+    @Multipart
+    @POST("/v1/files")
+    Single<File> uploadFile(@Part("purpose") RequestBody purpose, @Part MultipartBody.Part file);
+    @DELETE("/v1/files/{file_id}")
+    Single<DeleteResult> deleteFile(@Path("file_id") String fileId);
+    @GET("/v1/files/{file_id}")
+    Single<File> retrieveFile(@Path("file_id") String fileId);
+    @Streaming
+    @GET("/v1/files/{file_id}/content")
+    Single<ResponseBody> retrieveFileContent(@Path("file_id") String fileId);
+    @POST("/v1/fine_tuning/jobs")
+    Single<FineTuningJob> createFineTuningJob(@Body FineTuningJobRequest request);
+    @GET("/v1/fine_tuning/jobs")
+    Single<OpenAiResponse<FineTuningJob>> listFineTuningJobs();
+    @GET("/v1/fine_tuning/jobs/{fine_tuning_job_id}")
+    Single<FineTuningJob> retrieveFineTuningJob(@Path("fine_tuning_job_id") String fineTuningJobId);
+    @POST("/v1/fine_tuning/jobs/{fine_tuning_job_id}/cancel")
+    Single<FineTuningJob> cancelFineTuningJob(@Path("fine_tuning_job_id") String fineTuningJobId);
+    @GET("/v1/fine_tuning/jobs/{fine_tuning_job_id}/events")
+    Single<OpenAiResponse<FineTuningEvent>> listFineTuningJobEvents(@Path("fine_tuning_job_id") String fineTuningJobId);
+    @Deprecated
+    @POST("/v1/fine-tunes")
+    Single<FineTuneResult> createFineTune(@Body FineTuneRequest request);
+    @POST("/v1/completions")
+    Single<CompletionResult> createFineTuneCompletion(@Body CompletionRequest request);
+    @Deprecated
+    @GET("/v1/fine-tunes")
+    Single<OpenAiResponse<FineTuneResult>> listFineTunes();
+    @Deprecated
+    @GET("/v1/fine-tunes/{fine_tune_id}")
+    Single<FineTuneResult> retrieveFineTune(@Path("fine_tune_id") String fineTuneId);
+    @Deprecated
+    @POST("/v1/fine-tunes/{fine_tune_id}/cancel")
+    Single<FineTuneResult> cancelFineTune(@Path("fine_tune_id") String fineTuneId);
+    @Deprecated
+    @GET("/v1/fine-tunes/{fine_tune_id}/events")
+    Single<OpenAiResponse<FineTuneEvent>> listFineTuneEvents(@Path("fine_tune_id") String fineTuneId);
+    @DELETE("/v1/models/{fine_tune_id}")
+    Single<DeleteResult> deleteFineTune(@Path("fine_tune_id") String fineTuneId);
+    @POST("/v1/images/generations")
+    Single<ImageResult> createImage(@Body CreateImageRequest request);
+    @POST("/v1/images/edits")
+    Single<ImageResult> createImageEdit(@Body RequestBody requestBody);
+    @POST("/v1/images/variations")
+    Single<ImageResult> createImageVariation(@Body RequestBody requestBody);
+    @POST("/v1/audio/transcriptions")
+    Single<TranscriptionResult> createTranscription(@Body RequestBody requestBody);
+    @POST("/v1/audio/translations")
+    Single<TranslationResult> createTranslation(@Body RequestBody requestBody);
+    @POST("/v1/moderations")
+    Single<ModerationResult> createModeration(@Body ModerationRequest request);
+    @Deprecated
+    @GET("v1/engines")
+    Single<OpenAiResponse<Engine>> getEngines();
+    @Deprecated
+    @GET("/v1/engines/{engine_id}")
+    Single<Engine> getEngine(@Path("engine_id") String engineId);
+    /**
+     * Account information inquiry: It contains total amount (in US dollars) and other information.
+     *
+     * @return
+     */
+    @Deprecated
+    @GET("v1/dashboard/billing/subscription")
+    Single<Subscription> subscription();
+    /**
+     * Account call interface consumption amount inquiry.
+     * totalUsage = Total amount used by the account (in US cents).
+     *
+     * @param starDate
+     * @param endDate
+     * @return Consumption amount information.
+     */
+    @Deprecated
+    @GET("v1/dashboard/billing/usage")
+    Single<BillingUsage> billingUsage(@Query("start_date") LocalDate starDate, @Query("end_date") LocalDate endDate);
+}''', '2': '''
+package com.theokanning.openai;
+
+/**
+ * OkHttp Interceptor that adds an authorization token header
+ * 
+ * @deprecated Use {@link com.theokanning.openai.client.AuthenticationInterceptor}
+ */
+@Deprecated
+public class AuthenticationInterceptor extends com.theokanning.openai.client.AuthenticationInterceptor {
+
+    AuthenticationInterceptor(String token) {
+        super(token);
+    }
+
+}
+'''}
+
+    ca = CodeAnalyzer(engine, language)
+    res = ca.analyze(code_dict)
+    logger.debug(res)

dev_opsgpt/codechat/code_analyzer/code_dedup.py

@@ -0,0 +1,31 @@
+# encoding: utf-8
+'''
+@author: 温进
+@file: code_dedup.py
+@time: 2023/11/21 下午2:27
+@desc:
+'''
+# encoding: utf-8
+'''
+@author: 温进
+@file: java_dedup.py
+@time: 2023/10/23 下午5:02
+@desc:
+'''
+
+
+class CodeDedup:
+    def __init__(self):
+        pass
+
+    def dedup(self, code_dict):
+        code_dict = self.exact_dedup(code_dict)
+        return code_dict
+
+    def exact_dedup(self, code_dict):
+        res = {}
+        for fp, code_text in code_dict.items():
+            if code_text not in res.values():
+                res[fp] = code_text
+
+        return res

dev_opsgpt/codechat/code_analyzer/code_intepreter.py

@@ -0,0 +1,229 @@
+# encoding: utf-8
+'''
+@author: 温进
+@file: code_intepreter.py
+@time: 2023/11/22 上午11:57
+@desc:
+'''
+from loguru import logger
+from langchain.schema import (
+    HumanMessage,
+)
+
+from configs.model_config import CODE_INTERPERT_TEMPLATE
+
+from dev_opsgpt.llm_models.openai_model import getChatModel
+
+
+class CodeIntepreter:
+    def __init__(self):
+        pass
+
+    def get_intepretation(self, code_list):
+        '''
+        get intepretion of code
+        @param code_list:
+        @return:
+        '''
+        chat_model = getChatModel()
+
+        res = {}
+        for code in code_list:
+            message = CODE_INTERPERT_TEMPLATE.format(code=code)
+            message = [HumanMessage(content=message)]
+            chat_res = chat_model.predict_messages(message)
+            content = chat_res.content
+            res[code] = content
+        return res
+
+    def get_intepretation_batch(self, code_list):
+        '''
+        get intepretion of code
+        @param code_list:
+        @return:
+        '''
+        chat_model = getChatModel()
+
+        res = {}
+        messages = []
+        for code in code_list:
+            message = CODE_INTERPERT_TEMPLATE.format(code=code)
+            messages.append(message)
+
+        chat_ress = chat_model.batch(messages)
+        for chat_res, code in zip(chat_ress, code_list):
+            res[code] = chat_res.content
+        return res
+
+
+
+
+if __name__ == '__main__':
+    engine = 'openai'
+    code_list = ['''package com.theokanning.openai.client;
+import com.theokanning.openai.DeleteResult;
+import com.theokanning.openai.OpenAiResponse;
+import com.theokanning.openai.audio.TranscriptionResult;
+import com.theokanning.openai.audio.TranslationResult;
+import com.theokanning.openai.billing.BillingUsage;
+import com.theokanning.openai.billing.Subscription;
+import com.theokanning.openai.completion.CompletionRequest;
+import com.theokanning.openai.completion.CompletionResult;
+import com.theokanning.openai.completion.chat.ChatCompletionRequest;
+import com.theokanning.openai.completion.chat.ChatCompletionResult;
+import com.theokanning.openai.edit.EditRequest;
+import com.theokanning.openai.edit.EditResult;
+import com.theokanning.openai.embedding.EmbeddingRequest;
+import com.theokanning.openai.embedding.EmbeddingResult;
+import com.theokanning.openai.engine.Engine;
+import com.theokanning.openai.file.File;
+import com.theokanning.openai.fine_tuning.FineTuningEvent;
+import com.theokanning.openai.fine_tuning.FineTuningJob;
+import com.theokanning.openai.fine_tuning.FineTuningJobRequest;
+import com.theokanning.openai.finetune.FineTuneEvent;
+import com.theokanning.openai.finetune.FineTuneRequest;
+import com.theokanning.openai.finetune.FineTuneResult;
+import com.theokanning.openai.image.CreateImageRequest;
+import com.theokanning.openai.image.ImageResult;
+import com.theokanning.openai.model.Model;
+import com.theokanning.openai.moderation.ModerationRequest;
+import com.theokanning.openai.moderation.ModerationResult;
+import io.reactivex.Single;
+import okhttp3.MultipartBody;
+import okhttp3.RequestBody;
+import okhttp3.ResponseBody;
+import retrofit2.Call;
+import retrofit2.http.*;
+import java.time.LocalDate;
+public interface OpenAiApi {
+    @GET("v1/models")
+    Single<OpenAiResponse<Model>> listModels();
+    @GET("/v1/models/{model_id}")
+    Single<Model> getModel(@Path("model_id") String modelId);
+    @POST("/v1/completions")
+    Single<CompletionResult> createCompletion(@Body CompletionRequest request);
+    @Streaming
+    @POST("/v1/completions")
+    Call<ResponseBody> createCompletionStream(@Body CompletionRequest request);
+    @POST("/v1/chat/completions")
+    Single<ChatCompletionResult> createChatCompletion(@Body ChatCompletionRequest request);
+    @Streaming
+    @POST("/v1/chat/completions")
+    Call<ResponseBody> createChatCompletionStream(@Body ChatCompletionRequest request);
+    @Deprecated
+    @POST("/v1/engines/{engine_id}/completions")
+    Single<CompletionResult> createCompletion(@Path("engine_id") String engineId, @Body CompletionRequest request);
+    @POST("/v1/edits")
+    Single<EditResult> createEdit(@Body EditRequest request);
+    @Deprecated
+    @POST("/v1/engines/{engine_id}/edits")
+    Single<EditResult> createEdit(@Path("engine_id") String engineId, @Body EditRequest request);
+    @POST("/v1/embeddings")
+    Single<EmbeddingResult> createEmbeddings(@Body EmbeddingRequest request);
+    @Deprecated
+    @POST("/v1/engines/{engine_id}/embeddings")
+    Single<EmbeddingResult> createEmbeddings(@Path("engine_id") String engineId, @Body EmbeddingRequest request);
+    @GET("/v1/files")
+    Single<OpenAiResponse<File>> listFiles();
+    @Multipart
+    @POST("/v1/files")
+    Single<File> uploadFile(@Part("purpose") RequestBody purpose, @Part MultipartBody.Part file);
+    @DELETE("/v1/files/{file_id}")
+    Single<DeleteResult> deleteFile(@Path("file_id") String fileId);
+    @GET("/v1/files/{file_id}")
+    Single<File> retrieveFile(@Path("file_id") String fileId);
+    @Streaming
+    @GET("/v1/files/{file_id}/content")
+    Single<ResponseBody> retrieveFileContent(@Path("file_id") String fileId);
+    @POST("/v1/fine_tuning/jobs")
+    Single<FineTuningJob> createFineTuningJob(@Body FineTuningJobRequest request);
+    @GET("/v1/fine_tuning/jobs")
+    Single<OpenAiResponse<FineTuningJob>> listFineTuningJobs();
+    @GET("/v1/fine_tuning/jobs/{fine_tuning_job_id}")
+    Single<FineTuningJob> retrieveFineTuningJob(@Path("fine_tuning_job_id") String fineTuningJobId);
+    @POST("/v1/fine_tuning/jobs/{fine_tuning_job_id}/cancel")
+    Single<FineTuningJob> cancelFineTuningJob(@Path("fine_tuning_job_id") String fineTuningJobId);
+    @GET("/v1/fine_tuning/jobs/{fine_tuning_job_id}/events")
+    Single<OpenAiResponse<FineTuningEvent>> listFineTuningJobEvents(@Path("fine_tuning_job_id") String fineTuningJobId);
+    @Deprecated
+    @POST("/v1/fine-tunes")
+    Single<FineTuneResult> createFineTune(@Body FineTuneRequest request);
+    @POST("/v1/completions")
+    Single<CompletionResult> createFineTuneCompletion(@Body CompletionRequest request);
+    @Deprecated
+    @GET("/v1/fine-tunes")
+    Single<OpenAiResponse<FineTuneResult>> listFineTunes();
+    @Deprecated
+    @GET("/v1/fine-tunes/{fine_tune_id}")
+    Single<FineTuneResult> retrieveFineTune(@Path("fine_tune_id") String fineTuneId);
+    @Deprecated
+    @POST("/v1/fine-tunes/{fine_tune_id}/cancel")
+    Single<FineTuneResult> cancelFineTune(@Path("fine_tune_id") String fineTuneId);
+    @Deprecated
+    @GET("/v1/fine-tunes/{fine_tune_id}/events")
+    Single<OpenAiResponse<FineTuneEvent>> listFineTuneEvents(@Path("fine_tune_id") String fineTuneId);
+    @DELETE("/v1/models/{fine_tune_id}")
+    Single<DeleteResult> deleteFineTune(@Path("fine_tune_id") String fineTuneId);
+    @POST("/v1/images/generations")
+    Single<ImageResult> createImage(@Body CreateImageRequest request);
+    @POST("/v1/images/edits")
+    Single<ImageResult> createImageEdit(@Body RequestBody requestBody);
+    @POST("/v1/images/variations")
+    Single<ImageResult> createImageVariation(@Body RequestBody requestBody);
+    @POST("/v1/audio/transcriptions")
+    Single<TranscriptionResult> createTranscription(@Body RequestBody requestBody);
+    @POST("/v1/audio/translations")
+    Single<TranslationResult> createTranslation(@Body RequestBody requestBody);
+    @POST("/v1/moderations")
+    Single<ModerationResult> createModeration(@Body ModerationRequest request);
+    @Deprecated
+    @GET("v1/engines")
+    Single<OpenAiResponse<Engine>> getEngines();
+    @Deprecated
+    @GET("/v1/engines/{engine_id}")
+    Single<Engine> getEngine(@Path("engine_id") String engineId);
+    /**
+     * Account information inquiry: It contains total amount (in US dollars) and other information.
+     *
+     * @return
+     */
+    @Deprecated
+    @GET("v1/dashboard/billing/subscription")
+    Single<Subscription> subscription();
+    /**
+     * Account call interface consumption amount inquiry.
+     * totalUsage = Total amount used by the account (in US cents).
+     *
+     * @param starDate
+     * @param endDate
+     * @return Consumption amount information.
+     */
+    @Deprecated
+    @GET("v1/dashboard/billing/usage")
+    Single<BillingUsage> billingUsage(@Query("start_date") LocalDate starDate, @Query("end_date") LocalDate endDate);
+}''', '''
+package com.theokanning.openai;
+
+/**
+ * OkHttp Interceptor that adds an authorization token header
+ * 
+ * @deprecated Use {@link com.theokanning.openai.client.AuthenticationInterceptor}
+ */
+@Deprecated
+public class AuthenticationInterceptor extends com.theokanning.openai.client.AuthenticationInterceptor {
+
+    AuthenticationInterceptor(String token) {
+        super(token);
+    }
+
+}
+''']
+
+    ci = CodeIntepreter(engine)
+    res = ci.get_intepretation_batch(code_list)
+    logger.debug(res)
+
+
+
+
+

dev_opsgpt/codechat/code_analyzer/code_preprocess.py

@@ -0,0 +1,14 @@
+# encoding: utf-8
+'''
+@author: 温进
+@file: code_preprocess.py
+@time: 2023/11/21 下午2:28
+@desc:
+'''
+
+class CodePreprocessor:
+    def __init__(self):
+        pass
+
+    def preprocess(self, code_dict):
+        return code_dict

dev_opsgpt/codechat/code_analyzer/code_static_analysis.py

@@ -0,0 +1,26 @@
+# encoding: utf-8
+'''
+@author: 温进
+@file: code_static_analysis.py
+@time: 2023/11/21 下午2:28
+@desc:
+'''
+from dev_opsgpt.codechat.code_analyzer.language_static_analysis import *
+
+class CodeStaticAnalysis:
+    def __init__(self, language):
+        self.language = language
+
+    def analyze(self, code_dict):
+        '''
+        analyze code
+        @param code_list:
+        @return:
+        '''
+        if self.language == 'java':
+            analyzer = JavaStaticAnalysis()
+        else:
+            raise ValueError('language should be one of [java]')
+
+        analyze_res = analyzer.analyze(code_dict)
+        return analyze_res

dev_opsgpt/codechat/code_analyzer/language_static_analysis/__init__.py

@@ -0,0 +1,14 @@
+# encoding: utf-8
+'''
+@author: 温进
+@file: __init__.py.py
+@time: 2023/11/21 下午4:24
+@desc:
+'''
+
+from .java_static_analysis import JavaStaticAnalysis
+
+
+__all__ = [
+    'JavaStaticAnalysis'
+    ]

dev_opsgpt/codechat/code_analyzer/language_static_analysis/java_static_analysis.py

@@ -1,26 +1,24 @@
 # encoding: utf-8
 '''
 @author: 温进
-@file: java_parser.py
-@time: 2023/10/23 下午5:03
+@file: java_static_analysis.py
+@time: 2023/11/21 下午4:25
 @desc:
 '''
-import json
-import javalang
-import glob
 import os
 from loguru import logger
+import javalang
 
 
-class JavaParser:
+class JavaStaticAnalysis:
     def __init__(self):
         pass
 
-    def parse(self, java_code_list):
+    def analyze(self, java_code_dict):
         '''
         parse java code and extract entity
         '''
-        tree_dict = self.preparse(java_code_list)
+        tree_dict = self.preparse(java_code_dict)
         res = self.multi_java_code_parse(tree_dict)
 
         return res
@@ -38,15 +36,15 @@ class JavaParser:
                 continue
 
             if tree.package is not None:
-                tree_dict[java_code] = tree
+                tree_dict[fp] = {'code': java_code, 'tree': tree}
         logger.info('success parse {} files'.format(len(tree_dict)))
         return tree_dict
 
-    def single_java_code_parse(self, tree):
+    def single_java_code_parse(self, tree, fp):
         '''
         parse single code file
         > tree: javalang parse result
-        < {pac_name: '', class_name_list: [], func_name_list: [], import_pac_name_list: []]}
+        < {pac_name: '', class_name_list: [], func_name_dict: {}, import_pac_name_list: []]}
         '''
         import_pac_name_list = []
 
@@ -57,25 +55,26 @@ class JavaParser:
             import_pac_name = import_pac.path
             import_pac_name_list.append(import_pac_name)
 
-        pac_name = tree.package.name
+        fp_last = fp.split(os.path.sep)[-1]
+        pac_name = tree.package.name + '#' + fp_last
         class_name_list = []
         func_name_dict = {}
 
         for node in tree.types:
             if type(node) in (javalang.tree.ClassDeclaration, javalang.tree.InterfaceDeclaration):
-                class_name = pac_name + '.' + node.name
+                class_name = pac_name + '#' + node.name
                 class_name_list.append(class_name)
 
                 for node_inner in node.body:
                     if type(node_inner) is javalang.tree.MethodDeclaration:
-                        func_name = class_name + '.' + node_inner.name
+                        func_name = class_name + '#' + node_inner.name
 
                         # add params name to func_name
                         params_list = node_inner.parameters
 
                         for params in params_list:
                             params_name = params.type.name
-                            func_name = func_name + '_' + params_name
+                            func_name = func_name + '-' + params_name
 
                         if class_name not in func_name_dict:
                             func_name_dict[class_name] = []
@@ -97,11 +96,21 @@ class JavaParser:
         < parse_result_dict
         '''
         res_dict = {}
-        for java_code, tree in tree_dict.items():
+        for fp, value in tree_dict.items():
+            java_code = value['code']
+            tree = value['tree']
             try:
-                res_dict[java_code] = self.single_java_code_parse(tree)
+                res_dict[java_code] = self.single_java_code_parse(tree, fp)
             except Exception as e:
                 logger.debug(java_code)
                 raise ImportError
 
-        return res_dict
+        return res_dict
+
+
+
+
+
+
+
+

dev_opsgpt/codechat/code_crawler/__init__.py

@@ -0,0 +1,15 @@
+# encoding: utf-8
+'''
+@author: 温进
+@file: __init__.py.py
+@time: 2023/11/21 下午2:02
+@desc:
+'''
+from .zip_crawler import ZipCrawler
+from .dir_crawler import DirCrawler
+
+
+__all__ = [
+    'ZipCrawler',
+    'DirCrawler'
+    ]

dev_opsgpt/codechat/code_crawler/dir_crawler.py

@@ -0,0 +1,39 @@
+# encoding: utf-8
+'''
+@author: 温进
+@file: dir_crawler.py
+@time: 2023/11/22 下午2:54
+@desc:
+'''
+from loguru import logger
+import os
+import glob
+
+
+class DirCrawler:
+    @staticmethod
+    def crawl(path: str, suffix: str):
+        '''
+        read local java file in path
+        > path: path to crawl, must be absolute path like A/B/C
+        < dict of java code string
+        '''
+        java_file_list = glob.glob('{path}{sep}**{sep}*.{suffix}'.format(path=path, sep=os.path.sep, suffix=suffix),
+                                   recursive=True)
+        java_code_dict = {}
+
+        logger.info(path)
+        logger.info('number of file={}'.format(len(java_file_list)))
+        logger.info(java_file_list)
+
+        for java_file in java_file_list:
+            with open(java_file) as f:
+                java_code = ''.join(f.readlines())
+                java_code_dict[java_file] = java_code
+        return java_code_dict
+
+
+if __name__ == '__main__':
+    path = '/Users/bingxu/Desktop/工作/大模型/chatbot/test_code_repo/middleware-alipay-starters-parent'
+    suffix = 'java'
+    DirCrawler.crawl(path, suffix)

dev_opsgpt/codechat/code_crawler/zip_crawler.py

@@ -0,0 +1,31 @@
+# encoding: utf-8
+'''
+@author: 温进
+@file: zip_crawler.py
+@time: 2023/11/21 下午2:02
+@desc:
+'''
+from loguru import logger
+
+import zipfile
+from dev_opsgpt.codechat.code_crawler.dir_crawler import DirCrawler
+
+
+class ZipCrawler:
+    @staticmethod
+    def crawl(zip_file, output_path, suffix):
+        '''
+        unzip to output_path
+        @param zip_file:
+        @param output_path:
+        @return:
+        '''
+        logger.info(f'output_path={output_path}')
+        print(f'output_path={output_path}')
+        with zipfile.ZipFile(zip_file, 'r') as z:
+            z.extractall(output_path)
+
+        code_dict = DirCrawler.crawl(output_path, suffix)
+        return code_dict
+
+

dev_opsgpt/codechat/code_search/__init__.py

@@ -2,6 +2,6 @@
 '''
 @author: 温进
 @file: __init__.py.py
-@time: 2023/10/23 下午4:57
+@time: 2023/11/21 下午2:35
 @desc:
 '''

dev_opsgpt/codechat/code_search/code_search.py

@@ -0,0 +1,179 @@
+# encoding: utf-8
+'''
+@author: 温进
+@file: code_search.py
+@time: 2023/11/21 下午2:35
+@desc:
+'''
+import time
+from loguru import logger
+from collections import defaultdict
+
+from dev_opsgpt.db_handler.graph_db_handler.nebula_handler import NebulaHandler
+from dev_opsgpt.db_handler.vector_db_handler.chroma_handler import ChromaHandler
+
+from dev_opsgpt.codechat.code_search.cypher_generator import CypherGenerator
+from dev_opsgpt.codechat.code_search.tagger import Tagger
+from dev_opsgpt.embeddings.get_embedding import get_embedding
+
+# search_by_tag
+VERTEX_SCORE = 10
+HISTORY_VERTEX_SCORE = 5
+VERTEX_MERGE_RATIO = 0.5
+
+# search_by_description
+MAX_DISTANCE = 0.5
+
+
+class CodeSearch:
+    def __init__(self, nh: NebulaHandler, ch: ChromaHandler, limit: int = 3):
+        '''
+        init
+        @param nh: NebulaHandler
+        @param ch: ChromaHandler
+        @param limit: limit of result
+        '''
+        self.nh = nh
+        self.ch = ch
+        self.limit = limit
+
+    def search_by_tag(self, query: str):
+        '''
+        search_code_res by tag
+        @param query: str
+        @return:
+        '''
+        tagger = Tagger()
+        tag_list = tagger.generate_tag_query(query)
+        logger.info(f'query tag={tag_list}')
+
+        # get all verticex
+        vertex_list = self.nh.get_vertices().get('v', [])
+        vertex_vid_list = [i.as_node().get_id().as_string() for i in vertex_list]
+        logger.debug(vertex_vid_list)
+
+        # update score
+        vertex_score_dict = defaultdict(lambda: 0)
+        for vid in vertex_vid_list:
+            for tag in tag_list:
+                if tag in vid:
+                    vertex_score_dict[vid] += VERTEX_SCORE
+
+        # merge depend adj score
+        vertex_score_dict_final = {}
+        for vertex in vertex_score_dict:
+            cypher = f'''MATCH (v1)-[e]-(v2) where id(v1) == "{vertex}" RETURN v2'''
+            cypher_res = self.nh.execute_cypher(cypher, self.nh.space_name)
+            cypher_res_dict = self.nh.result_to_dict(cypher_res)
+
+            adj_vertex_list = [i.as_node().get_id().as_string() for i in cypher_res_dict.get('v2', [])]
+
+            score = vertex_score_dict.get(vertex, 0)
+            for adj_vertex in adj_vertex_list:
+                score += vertex_score_dict.get(adj_vertex, 0) * VERTEX_MERGE_RATIO
+
+            if score > 0:
+                vertex_score_dict_final[vertex] = score
+
+        # get most prominent package tag
+        package_score_dict = defaultdict(lambda: 0)
+        for vertex, score in vertex_score_dict.items():
+            package = '#'.join(vertex.split('#')[0:2])
+            package_score_dict[package] += score
+
+        # get respective code
+        res = []
+        package_score_tuple = list(package_score_dict.items())
+        package_score_tuple.sort(key=lambda x: x[1], reverse=True)
+
+        ids = [i[0] for i in package_score_tuple]
+        chroma_res = self.ch.get(ids=ids, include=['metadatas'])
+        for vertex, score in package_score_tuple:
+            index = chroma_res['result']['ids'].index(vertex)
+            code_text = chroma_res['result']['metadatas'][index]['code_text']
+            res.append({
+                "vertex": vertex,
+                "code_text": code_text}
+            )
+            if len(res) >= self.limit:
+                break
+        return res
+
+    def search_by_desciption(self, query: str, engine: str):
+        '''
+        search by perform sim search
+        @param query:
+        @return:
+        '''
+        query = query.replace(',', '，')
+        query_emb = get_embedding(engine=engine, text_list=[query])
+        query_emb = query_emb[query]
+
+        query_embeddings = [query_emb]
+        query_result = self.ch.query(query_embeddings=query_embeddings, n_results=self.limit,
+                                     include=['metadatas', 'distances'])
+        logger.debug(query_result)
+
+        res = []
+        for idx, distance in enumerate(query_result['result']['distances'][0]):
+            if distance < MAX_DISTANCE:
+                vertex = query_result['result']['ids'][0][idx]
+                code_text = query_result['result']['metadatas'][0][idx]['code_text']
+                res.append({
+                    "vertex": vertex,
+                    "code_text": code_text
+                })
+
+        return res
+
+    def search_by_cypher(self, query: str):
+        '''
+        search by generating cypher
+        @param query:
+        @param engine:
+        @return:
+        '''
+        cg = CypherGenerator()
+        cypher = cg.get_cypher(query)
+
+        if not cypher:
+            return None
+
+        cypher_res = self.nh.execute_cypher(cypher, self.nh.space_name)
+        logger.info(f'cypher execution result={cypher_res}')
+        if not cypher_res.is_succeeded():
+            return {
+            'cypher': '',
+            'cypher_res': ''
+        }
+
+        res = {
+            'cypher': cypher,
+            'cypher_res': cypher_res
+        }
+
+        return res
+
+
+if __name__ == '__main__':
+    from configs.server_config import NEBULA_HOST, NEBULA_PORT, NEBULA_USER, NEBULA_PASSWORD, NEBULA_STORAGED_PORT
+    from configs.server_config import CHROMA_PERSISTENT_PATH
+
+    codebase_name = 'testing'
+
+    nh = NebulaHandler(host=NEBULA_HOST, port=NEBULA_PORT, username=NEBULA_USER,
+                       password=NEBULA_PASSWORD, space_name=codebase_name)
+    nh.add_host(NEBULA_HOST, NEBULA_STORAGED_PORT)
+    time.sleep(0.5)
+
+    ch = ChromaHandler(path=CHROMA_PERSISTENT_PATH, collection_name=codebase_name)
+
+    cs = CodeSearch(nh, ch)
+    # res = cs.search_by_tag(tag_list=['createFineTuneCompletion', 'OpenAiApi'])
+    # logger.debug(res)
+
+    # res = cs.search_by_cypher('代码中一共有多少个类', 'openai')
+    # logger.debug(res)
+
+    res = cs.search_by_desciption('使用不同的HTTP请求类型（GET、POST、DELETE等）来执行不同的操作', 'openai')
+    logger.debug(res)

dev_opsgpt/codechat/code_search/cypher_generator.py

@@ -0,0 +1,63 @@
+# encoding: utf-8
+'''
+@author: 温进
+@file: cypher_generator.py
+@time: 2023/11/24 上午10:17
+@desc:
+'''
+from loguru import logger
+
+from dev_opsgpt.llm_models.openai_model import getChatModel
+from dev_opsgpt.utils.postprocess import replace_lt_gt
+from langchain.schema import (
+    HumanMessage,
+)
+from langchain.chains.graph_qa.prompts import NGQL_GENERATION_PROMPT
+
+
+schema = '''
+Node properties: [{'tag': 'package', 'properties': []}, {'tag': 'class', 'properties': []}, {'tag': 'method', 'properties': []}]
+Edge properties: [{'edge': 'contain', 'properties': []}, {'edge': 'depend', 'properties': []}]
+Relationships: ['(:package)-[:contain]->(:class)', '(:class)-[:contain]->(:method)', '(:package)-[:contain]->(:package)']
+'''
+
+
+class CypherGenerator:
+    def __init__(self):
+        self.model = getChatModel()
+
+    def get_cypher(self, query: str):
+        '''
+        get cypher from query
+        @param query:
+        @return:
+        '''
+        content = NGQL_GENERATION_PROMPT.format(schema=schema, question=query)
+
+        ans = ''
+        message = [HumanMessage(content=content)]
+        chat_res = self.model.predict_messages(message)
+        ans = chat_res.content
+
+        ans = replace_lt_gt(ans)
+
+        ans = self.post_process(ans)
+        return ans
+
+    def post_process(self, cypher_res: str):
+        '''
+        判断是否为正确的 cypher
+        @param cypher_res:
+        @return:
+        '''
+        if '(' not in cypher_res or ')' not in cypher_res:
+            return ''
+
+        return cypher_res
+
+
+if __name__ == '__main__':
+    query = '代码中一共有多少个类'
+    cg = CypherGenerator(engine='openai')
+
+    cg.get_cypher(query)

dev_opsgpt/codechat/code_search/tagger.py

@@ -0,0 +1,23 @@
+# encoding: utf-8
+'''
+@author: 温进
+@file: tagger.py
+@time: 2023/11/24 下午1:32
+@desc:
+'''
+import re
+from loguru import logger
+
+
+class Tagger:
+    def __init__(self):
+        pass
+
+    def generate_tag_query(self, query):
+        '''
+        generate tag from query
+        '''
+        # simple extract english
+        tag_list = re.findall(r'[a-zA-Z\_\.]+', query)
+        tag_list = list(set(tag_list))
+        return tag_list

dev_opsgpt/codechat/codebase_handler/__init__.py

@@ -2,6 +2,6 @@
 '''
 @author: 温进
 @file: __init__.py.py
-@time: 2023/10/23 下午5:00
+@time: 2023/11/21 下午2:07
 @desc:
 '''

dev_opsgpt/codechat/codebase_handler/codebase_handler.py

@@ -0,0 +1,169 @@
+# encoding: utf-8
+'''
+@author: 温进
+@file: codebase_handler.py
+@time: 2023/11/21 下午2:25
+@desc:
+'''
+import time
+from loguru import logger
+
+from configs.server_config import NEBULA_HOST, NEBULA_PORT, NEBULA_USER, NEBULA_PASSWORD, NEBULA_STORAGED_PORT
+from configs.server_config import CHROMA_PERSISTENT_PATH
+
+from configs.model_config import EMBEDDING_ENGINE
+
+from dev_opsgpt.db_handler.graph_db_handler.nebula_handler import NebulaHandler
+from dev_opsgpt.db_handler.vector_db_handler.chroma_handler import ChromaHandler
+from dev_opsgpt.codechat.code_crawler.zip_crawler import *
+from dev_opsgpt.codechat.code_analyzer.code_analyzer import CodeAnalyzer
+from dev_opsgpt.codechat.codebase_handler.code_importer import CodeImporter
+from dev_opsgpt.codechat.code_search.code_search import CodeSearch
+
+
+class CodeBaseHandler:
+    def __init__(self, codebase_name: str, code_path: str = '',
+                 language: str = 'java', crawl_type: str = 'ZIP'):
+        self.codebase_name = codebase_name
+        self.code_path = code_path
+        self.language = language
+        self.crawl_type = crawl_type
+
+        self.nh = NebulaHandler(host=NEBULA_HOST, port=NEBULA_PORT, username=NEBULA_USER,
+                                password=NEBULA_PASSWORD, space_name=codebase_name)
+        self.nh.add_host(NEBULA_HOST, NEBULA_STORAGED_PORT)
+        time.sleep(1)
+
+        self.ch = ChromaHandler(path=CHROMA_PERSISTENT_PATH, collection_name=codebase_name)
+
+    def import_code(self, zip_file='', do_interpret=True):
+        '''
+        analyze code and save it to codekg and codedb
+        @return:
+        '''
+        # init graph to init tag and edge
+        code_importer = CodeImporter(engine=EMBEDDING_ENGINE, codebase_name=self.codebase_name,
+                                     nh=self.nh, ch=self.ch)
+        code_importer.init_graph()
+        time.sleep(5)
+
+        # crawl code
+        st0 = time.time()
+        logger.info('start crawl')
+        code_dict = self.crawl_code(zip_file)
+        logger.debug('crawl done, rt={}'.format(time.time() - st0))
+
+        # analyze code
+        logger.info('start analyze')
+        st1 = time.time()
+        code_analyzer = CodeAnalyzer(language=self.language)
+        static_analysis_res, interpretation = code_analyzer.analyze(code_dict, do_interpret=do_interpret)
+        logger.debug('analyze done, rt={}'.format(time.time() - st1))
+
+        # add info to nebula and chroma
+        st2 = time.time()
+        code_importer.import_code(static_analysis_res, interpretation, do_interpret=do_interpret)
+        logger.debug('update codebase done, rt={}'.format(time.time() - st2))
+
+        # get KG info
+        stat = self.nh.get_stat()
+        vertices_num, edges_num = stat['vertices'], stat['edges']
+
+        # get chroma info
+        file_num = self.ch.count()['result']
+
+        return vertices_num, edges_num, file_num
+
+    def delete_codebase(self, codebase_name: str):
+        '''
+        delete codebase
+        @param codebase_name: name of codebase
+        @return:
+        '''
+        self.nh.drop_space(space_name=codebase_name)
+        self.ch.delete_collection(collection_name=codebase_name)
+
+    def crawl_code(self, zip_file=''):
+        '''
+        @return:
+        '''
+        if self.language == 'java':
+            suffix = 'java'
+
+        logger.info(f'crawl_type={self.crawl_type}')
+
+        code_dict = {}
+        if self.crawl_type.lower() == 'zip':
+            code_dict = ZipCrawler.crawl(zip_file, output_path=self.code_path, suffix=suffix)
+        elif self.crawl_type.lower() == 'dir':
+            code_dict = DirCrawler.crawl(self.code_path, suffix)
+
+        return code_dict
+
+    def search_code(self, query: str, search_type: str, limit: int = 3):
+        '''
+        search code from codebase
+        @param limit:
+        @param engine:
+        @param query: query from user
+        @param search_type: ['cypher', 'graph', 'vector']
+        @return: 
+        '''
+        assert search_type in ['cypher', 'tag', 'description']
+
+        code_search = CodeSearch(nh=self.nh, ch=self.ch, limit=limit)
+
+        if search_type == 'cypher':
+            search_res = code_search.search_by_cypher(query=query)
+        elif search_type == 'tag':
+            search_res = code_search.search_by_tag(query=query)
+        elif search_type == 'description':
+            search_res = code_search.search_by_desciption(query=query, engine=EMBEDDING_ENGINE)
+
+        context, related_vertice = self.format_search_res(search_res, search_type)
+        return context, related_vertice
+
+    def format_search_res(self, search_res: str, search_type: str):
+        '''
+        format search_res
+        @param search_res:
+        @param search_type:
+        @return:
+        '''
+        CYPHER_QA_PROMPT = '''
+        执行的 Cypher 是: {cypher}
+        Cypher 的结果是: {result}
+        '''
+
+        if search_type == 'cypher':
+            context = CYPHER_QA_PROMPT.format(cypher=search_res['cypher'], result=search_res['cypher_res'])
+            related_vertice = []
+        elif search_type == 'tag':
+            context = ''
+            related_vertice = []
+            for code in search_res:
+                context = context + code['code_text'] + '\n'
+                related_vertice.append(code['vertex'])
+        elif search_type == 'description':
+            context = ''
+            related_vertice = []
+            for code in search_res:
+                context = context + code['code_text'] + '\n'
+                related_vertice.append(code['vertex'])
+
+        return context, related_vertice
+
+
+if __name__ == '__main__':
+    codebase_name = 'testing'
+    code_path = '/Users/bingxu/Desktop/工作/大模型/chatbot/test_code_repo/client'
+    cbh = CodeBaseHandler(codebase_name, code_path, crawl_type='dir')
+
+    # query = '使用不同的HTTP请求类型（GET、POST、DELETE等）来执行不同的操作'
+    # query = '代码中一共有多少个类'
+
+    query = 'intercept 函数作用是什么'
+    search_type = 'graph'
+    limit = 2
+    res = cbh.search_code(query, search_type, limit)
+    logger.debug(res)

dev_opsgpt/connector/agents/__init__.py

@@ -1,6 +1,9 @@
 from .base_agent import BaseAgent
 from .react_agent import ReactAgent
+from .check_agent import CheckAgent
+from .executor_agent import ExecutorAgent
+from .selector_agent import SelectorAgent
 
 __all__ = [
-    "BaseAgent", "ReactAgent"
+    "BaseAgent", "ReactAgent", "CheckAgent", "ExecutorAgent", "SelectorAgent"
 ]

dev_opsgpt/connector/agents/base_agent.py

@@ -7,16 +7,19 @@ import traceback
 import uuid
 from loguru import logger
 
-from dev_opsgpt.connector.shcema.memory import Memory
-from dev_opsgpt.connector.connector_schema import (
-    Task, Role, Message, ActionStatus, Doc, CodeDoc
-    )
+from dev_opsgpt.connector.schema import (
+    Memory, Task, Env, Role, Message, ActionStatus, CodeDoc, Doc
+)
 from configs.server_config import SANDBOX_SERVER
 from dev_opsgpt.sandbox import PyCodeBox, CodeBoxResponse
 from dev_opsgpt.tools import DDGSTool, DocRetrieval, CodeRetrieval
-from dev_opsgpt.connector.configs.agent_config import REACT_PROMPT_INPUT
+from dev_opsgpt.connector.configs.prompts import BASE_PROMPT_INPUT, QUERY_CONTEXT_DOC_PROMPT_INPUT, BEGIN_PROMPT_INPUT
+from dev_opsgpt.connector.message_process import MessageUtils
+from dev_opsgpt.connector.configs.agent_config import REACT_PROMPT_INPUT, QUERY_CONTEXT_PROMPT_INPUT, PLAN_PROMPT_INPUT
 
 from dev_opsgpt.llm_models import getChatModel
+from dev_opsgpt.connector.utils import parse_section
+
 
 
 class BaseAgent:
@@ -34,67 +37,74 @@ class BaseAgent:
             stop: Union[List[str], str] = None,
             do_filter: bool = True,
             do_use_self_memory: bool = True,
-            # docs_prompt: str,
+            focus_agents: List[str] = [],
+            focus_message_keys: List[str] = [],
             # prompt_mamnger: PromptManager
             ):
         
         self.task = task
         self.role = role
+        self.message_utils = MessageUtils()
         self.llm = self.create_llm_engine(temperature, stop)
         self.memory = self.init_history(memory)
         self.chat_turn = chat_turn
         self.do_search = do_search
         self.do_doc_retrieval = do_doc_retrieval
         self.do_tool_retrieval = do_tool_retrieval
-        self.codebox = PyCodeBox(
-                    remote_url=SANDBOX_SERVER["url"],
-                    remote_ip=SANDBOX_SERVER["host"],
-                    remote_port=SANDBOX_SERVER["port"],
-                    token="mytoken",
-                    do_code_exe=True,
-                    do_remote=SANDBOX_SERVER["do_remote"],
-                    do_check_net=False
-                    )
+        self.focus_agents = focus_agents
+        self.focus_message_keys = focus_message_keys
         self.do_filter = do_filter
         self.do_use_self_memory = do_use_self_memory
-        # self.docs_prompt = docs_prompt
         # self.prompt_manager = None
 
-    def run(self, query: Message, history: Memory = None, background: Memory = None) -> Message:
-        '''llm inference'''
+    def run(self, query: Message, history: Memory = None, background: Memory = None, memory_pool: Memory=None) -> Message:
+        '''agent reponse from multi-message'''
+        message = None
+        for message in self.arun(query, history, background, memory_pool):
+            pass
+        return message
+    
+    def arun(self, query: Message, history: Memory = None, background: Memory = None, memory_pool: Memory=None) -> Message:
+        '''agent reponse from multi-message'''
         # insert query into memory
         query_c = copy.deepcopy(query)
         
         self_memory = self.memory if self.do_use_self_memory else None
-        prompt = self.create_prompt(query_c, self_memory, history, background)
+        # create your llm prompt
+        prompt = self.create_prompt(query_c, self_memory, history, background, memory_pool=memory_pool)
         content = self.llm.predict(prompt)
         logger.debug(f"{self.role.role_name} prompt: {prompt}")
-        # logger.debug(f"{self.role.role_name} content: {content}")
+        logger.debug(f"{self.role.role_name} content: {content}")
 
         output_message = Message(
             role_name=self.role.role_name,
             role_type="ai", #self.role.role_type,
             role_content=content,
             role_contents=[content],
+            step_content=content,
             input_query=query_c.input_query,
-            tools=query_c.tools
+            tools=query_c.tools,
+            parsed_output_list=[query.parsed_output]
             )
-        
-        output_message = self.parser(output_message)
+        # common parse llm' content to message
+        output_message = self.message_utils.parser(output_message)
         if self.do_filter:
-            output_message = self.filter(output_message)
+            output_message = self.message_utils.filter(output_message)
 
-
-        # 更新自身的回答
+        # update self_memory
         self.append_history(query_c)
         self.append_history(output_message)
-        logger.info(f"{self.role.role_name} step_run: {output_message.role_content}")
-        return output_message
+        # logger.info(f"{self.role.role_name} currenct question: {output_message.input_query}\nllm_step_run: {output_message.role_content}")
+        output_message.input_query = output_message.role_content
+        output_message.parsed_output_list.append(output_message.parsed_output)
+        # update memory pool
+        memory_pool.append(output_message)
+        yield output_message
 
     def create_prompt(
-            self, query: Message, memory: Memory =None, history: Memory = None, background: Memory = None, prompt_mamnger=None) -> str:
+            self, query: Message, memory: Memory =None, history: Memory = None, background: Memory = None, memory_pool: Memory=None, prompt_mamnger=None) -> str:
         '''
-        role\task\tools\docs\memory
+        prompt engineer, contains role\task\tools\docs\memory
         '''
         # 
         doc_infos = self.create_doc_prompt(query)
@@ -105,50 +115,105 @@ class BaseAgent:
         background_prompt = self.create_background_prompt(background, control_key="step_content")
         history_prompt = self.create_history_prompt(history)
         selfmemory_prompt = self.create_selfmemory_prompt(memory, control_key="step_content")
-        # 
+        
         # extra_system_prompt = self.role.role_prompt
+
         
         prompt = self.role.role_prompt.format(**{"formatted_tools": formatted_tools, "tool_names": tool_names})
+        #
+        memory_pool_select_by_agent_key = self.select_memory_by_agent_key(memory_pool)
+        memory_pool_select_by_agent_key_context = '\n\n'.join([f"*{k}*\n{v}" for parsed_output in memory_pool_select_by_agent_key.get_parserd_output_list() for k, v in parsed_output.items() if k not in ['Action Status']])
+        
+        # input_query = query.input_query
+
+        # # logger.debug(f"{self.role.role_name}  extra_system_prompt: {self.role.role_prompt}")
+        # # logger.debug(f"{self.role.role_name}  input_query: {input_query}")
+        # # logger.debug(f"{self.role.role_name}  doc_infos: {doc_infos}")
+        # # logger.debug(f"{self.role.role_name}  tool_names: {tool_names}")
+        # if "**Context:**" in self.role.role_prompt:
+        #     # logger.debug(f"parsed_output_list: {query.parsed_output_list}")
+        #     # input_query =  "'''" + "\n".join([f"###{k}###\n{v}" for i in query.parsed_output_list for k,v in i.items() if "Action Status" !=k]) + "'''"
+        #     context =  "\n".join([f"*{k}*\n{v}" for i in query.parsed_output_list for k,v in i.items() if "Action Status" !=k])
+        #     # context = history_prompt or '""'
+        #     # logger.debug(f"parsed_output_list: {t}")
+        #     prompt += "\n" + QUERY_CONTEXT_PROMPT_INPUT.format(**{"context": context, "query": query.origin_query})
+        # else:
+        #     prompt += "\n" + PLAN_PROMPT_INPUT.format(**{"query": input_query})
 
         task = query.task or self.task
         if task_prompt is not None:
             prompt += "\n" + task.task_prompt
 
+        DocInfos = ""
         if doc_infos is not None and doc_infos!="" and doc_infos!="不存在知识库辅助信息":
-            prompt += f"\n知识库信息: {doc_infos}"
+            DocInfos += f"\nDocument Information: {doc_infos}"
 
         if code_infos is not None and code_infos!="" and code_infos!="不存在代码库辅助信息":
-            prompt += f"\n代码库信息: {code_infos}"
+            DocInfos += f"\nCodeBase Infomation: {code_infos}"
+            
+        # if selfmemory_prompt:
+        #     prompt += "\n" + selfmemory_prompt
 
-        if background_prompt:
-            prompt += "\n" + background_prompt
+        # if background_prompt:
+        #     prompt += "\n" + background_prompt
 
-        if history_prompt:
-            prompt += "\n" + history_prompt
+        # if history_prompt:
+        #     prompt += "\n" + history_prompt
 
-        if selfmemory_prompt:
-            prompt += "\n" + selfmemory_prompt
-
-        # input_query = memory.to_tuple_messages(content_key="step_content")
-        # input_query = "\n".join([f"{k}: {v}" for k, v in input_query if v])
-
-        input_query = query.role_content
+        input_query = query.input_query
 
         # logger.debug(f"{self.role.role_name}  extra_system_prompt: {self.role.role_prompt}")
-        logger.debug(f"{self.role.role_name}  input_query: {input_query}")
+        # logger.debug(f"{self.role.role_name}  input_query: {input_query}")
         # logger.debug(f"{self.role.role_name}  doc_infos: {doc_infos}")
-        logger.debug(f"{self.role.role_name}  tool_names: {tool_names}")
-        prompt += "\n" + REACT_PROMPT_INPUT.format(**{"query": input_query})
+        # logger.debug(f"{self.role.role_name}  tool_names: {tool_names}")
+
+        # extra_system_prompt = self.role.role_prompt
+        input_keys = parse_section(self.role.role_prompt, 'Input Format')
+        prompt = self.role.role_prompt.format(**{"formatted_tools": formatted_tools, "tool_names": tool_names})
+        prompt += "\n" + BEGIN_PROMPT_INPUT
+        for input_key in input_keys:
+            if input_key == "Origin Query": 
+                prompt += "\n**Origin Query:**\n" + query.origin_query
+            elif input_key == "Context":
+                context =  "\n".join([f"*{k}*\n{v}" for i in query.parsed_output_list for k,v in i.items() if "Action Status" !=k])
+                if history:
+                    context = history_prompt + "\n" + context
+                if not context:
+                    context = "there is no context"
+
+                if self.focus_agents and memory_pool_select_by_agent_key_context:
+                    context = memory_pool_select_by_agent_key_context
+                prompt += "\n**Context:**\n" + context + "\n" + input_query
+            elif input_key == "DocInfos":
+                prompt += "\n**DocInfos:**\n" + DocInfos
+            elif input_key == "Question":
+                prompt += "\n**Question:**\n" + input_query
+
+        # if "**Context:**" in self.role.role_prompt:
+        #     # logger.debug(f"parsed_output_list: {query.parsed_output_list}")
+        #     # input_query =  "'''" + "\n".join([f"###{k}###\n{v}" for i in query.parsed_output_list for k,v in i.items() if "Action Status" !=k]) + "'''"
+        #     context =  "\n".join([f"*{k}*\n{v}" for i in query.parsed_output_list for k,v in i.items() if "Action Status" !=k])
+        #     if history:
+        #         context = history_prompt + "\n" + context
+
+        #     if not context:
+        #         context = "there is no context"
+
+        #     # logger.debug(f"parsed_output_list: {t}")
+        #     if "DocInfos" in prompt:
+        #         prompt += "\n" + QUERY_CONTEXT_DOC_PROMPT_INPUT.format(**{"context": context, "query": query.origin_query, "DocInfos": DocInfos})
+        #     else:
+        #         prompt += "\n" + QUERY_CONTEXT_PROMPT_INPUT.format(**{"context": context, "query": query.origin_query, "DocInfos": DocInfos})
+        # else:
+        #     prompt += "\n" + BASE_PROMPT_INPUT.format(**{"query": input_query})
+
         # prompt = extra_system_prompt.format(**{"query": input_query, "doc_infos": doc_infos, "formatted_tools": formatted_tools, "tool_names": tool_names})
         while "{{" in prompt or "}}" in prompt:
             prompt = prompt.replace("{{", "{")
             prompt = prompt.replace("}}", "}")
+
+        # logger.debug(f"{self.role.role_name}  prompt: {prompt}")
         return prompt
-    
-        # prompt_comp = [("system", extra_system_prompt)] + memory.to_tuple_messages()
-        # prompt = ChatPromptTemplate.from_messages(prompt_comp)
-        # prompt = prompt.format(**{"query": query.role_content, "doc_infos": doc_infos, "formatted_tools": formatted_tools, "tool_names": tool_names})
-        # return prompt
 
     def create_doc_prompt(self, message: Message) -> str: 
         ''''''
@@ -197,7 +262,7 @@ class BaseAgent:
         return "\n补充自身对话信息: " + selfmemory_message if selfmemory_message else None
     
     def init_history(self, memory: Memory = None) -> Memory:
-        return Memory([])
+        return Memory(messages=[])
     
     def update_history(self, message: Message):
         self.memory.append(message)
@@ -212,216 +277,389 @@ class BaseAgent:
     def create_llm_engine(self, temperature=0.2, stop=None):
         return getChatModel(temperature=temperature, stop=stop)
 
-    def filter(self, message: Message, stop=None) -> Message:
+    # def filter(self, message: Message, stop=None) -> Message:
 
-        tool_params = self.parser_spec_key(message.role_content, "tool_params")
-        code_content = self.parser_spec_key(message.role_content, "code_content")
-        plan = self.parser_spec_key(message.role_content, "plan")
-        plans = self.parser_spec_key(message.role_content, "plans", do_search=False)
-        content = self.parser_spec_key(message.role_content, "content", do_search=False)
+        # tool_params = self.parser_spec_key(message.role_content, "tool_params")
+        # code_content = self.parser_spec_key(message.role_content, "code_content")
+        # plan = self.parser_spec_key(message.role_content, "plan")
+        # plans = self.parser_spec_key(message.role_content, "plans", do_search=False)
+        # content = self.parser_spec_key(message.role_content, "content", do_search=False)
 
-        # logger.debug(f"tool_params: {tool_params}, code_content: {code_content}, plan: {plan}, plans: {plans}, content: {content}")
-        role_content = tool_params or code_content or plan or plans or content
-        message.role_content = role_content or message.role_content
-        return message
+        # # logger.debug(f"tool_params: {tool_params}, code_content: {code_content}, plan: {plan}, plans: {plans}, content: {content}")
+        # role_content = tool_params or code_content or plan or plans or content
+        # message.role_content = role_content or message.role_content
+        # return message
     
     def token_usage(self, ):
+        '''calculate the usage of token'''
         pass
 
-    def get_extra_infos(self, message: Message) -> Message:
-        ''''''
-        if self.do_search:
-            message = self.get_search_retrieval(message)
+    def select_memory_by_key(self, memory: Memory) -> Memory:
+        return Memory(
+            messages=[self.select_message_by_key(message) for message in memory.messages 
+                      if self.select_message_by_key(message) is not None]
+                      )
+
+    def select_memory_by_agent_key(self, memory: Memory) -> Memory:
+        return Memory(
+            messages=[self.select_message_by_agent_key(message) for message in memory.messages 
+                      if self.select_message_by_agent_key(message) is not None]
+                      )
+
+    def select_message_by_agent_key(self, message: Message) -> Message:
+        # assume we focus all agents
+        if self.focus_agents == []:
+            return message
+        return None if message is None or message.role_name not in self.focus_agents else self.select_message_by_key(message)
+    
+    def select_message_by_key(self, message: Message) -> Message:
+        # assume we focus all key contents
+        if message is None:
+            return message
         
-        if self.do_doc_retrieval:
-            message = self.get_doc_retrieval(message)
-
-        if self.do_tool_retrieval:
-            message = self.get_tool_retrieval(message)
+        if self.focus_message_keys == []:
+            return message
         
-        return message 
-    
-    def get_search_retrieval(self, message: Message,) -> Message:
-        SEARCH_ENGINES = {"duckduckgo": DDGSTool}
-        search_docs = []
-        for idx, doc in enumerate(SEARCH_ENGINES["duckduckgo"].run(message.role_content, 3)):
-            doc.update({"index": idx})
-            search_docs.append(Doc(**doc))
-        message.search_docs = search_docs
-        return message
-    
-    def get_doc_retrieval(self, message: Message) -> Message:
-        query = message.role_content
-        knowledge_basename = message.doc_engine_name
-        top_k = message.top_k
-        score_threshold = message.score_threshold
-        if knowledge_basename:
-            docs = DocRetrieval.run(query, knowledge_basename, top_k, score_threshold)
-            message.db_docs = [Doc(**doc) for doc in docs]
-        return message
-    
-    def get_code_retrieval(self, message: Message) -> Message:
-        # DocRetrieval.run("langchain是什么", "DSADSAD")
-        query = message.input_query
-        code_engine_name = message.code_engine_name
-        history_node_list = message.history_node_list
-        code_docs = CodeRetrieval.run(code_engine_name, query, code_limit=message.top_k, history_node_list=history_node_list)
-        message.code_docs = [CodeDoc(**doc) for doc in code_docs]
-        return message
-    
-    def get_tool_retrieval(self, message: Message) -> Message:
-        return message
-    
-    def step_router(self, message: Message) -> Message:
-        ''''''
-        # message = self.parser(message)
-        # logger.debug(f"message.action_status: {message.action_status}")
-        if message.action_status == ActionStatus.CODING:
-            message = self.code_step(message)
-        elif message.action_status == ActionStatus.TOOL_USING:
-            message = self.tool_step(message)
+        message_c = copy.deepcopy(message)
+        message_c.parsed_output = {k: v for k,v in message_c.parsed_output.items() if k in self.focus_message_keys}
+        message_c.parsed_output_list = [{k: v for k,v in parsed_output.items() if k in self.focus_message_keys} for parsed_output in message_c.parsed_output_list]
+        return message_c
 
-        return message
-
-    def code_step(self, message: Message) -> Message:
-        '''execute code'''
-        # logger.debug(f"message.role_content: {message.role_content}, message.code_content: {message.code_content}")
-        code_answer = self.codebox.chat('```python\n{}```'.format(message.code_content))
-        code_prompt = f"执行上述代码后存在报错信息为 {code_answer.code_exe_response}，需要进行修复" \
-                    if code_answer.code_exe_type == "error" else f"执行上述代码后返回信息为 {code_answer.code_exe_response}"
-        uid = str(uuid.uuid1())
-        if code_answer.code_exe_type == "image/png":
-            message.figures[uid] = code_answer.code_exe_response
-            message.code_answer = f"\n观察: 执行上述代码后生成一张图片, 图片名为{uid}\n"
-            message.observation = f"\n观察: 执行上述代码后生成一张图片, 图片名为{uid}\n"
-            message.step_content += f"\n观察: 执行上述代码后生成一张图片, 图片名为{uid}\n"
-            message.step_contents += [f"\n观察: 执行上述代码后生成一张图片, 图片名为{uid}\n"]
-            message.role_content += f"\n观察:执行上述代码后生成一张图片, 图片名为{uid}\n"
-        else:
-            message.code_answer = code_answer.code_exe_response
-            message.observation = code_answer.code_exe_response
-            message.step_content += f"\n观察: {code_prompt}\n"
-            message.step_contents += [f"\n观察: {code_prompt}\n"]
-            message.role_content += f"\n观察: {code_prompt}\n"
-        # logger.info(f"观察: {message.action_status}, {message.observation}")
-        return message
-
-    def tool_step(self, message: Message) -> Message:
-        '''execute tool'''
-        # logger.debug(f"message: {message.action_status}, {message.tool_name}, {message.tool_params}")
-        tool_names = [tool.name for tool in message.tools]
-        if message.tool_name not in tool_names:
-            message.tool_answer = "不存在可以执行的tool"    
-            message.observation = "不存在可以执行的tool"    
-            message.role_content += f"\n观察: 不存在可以执行的tool\n"
-            message.step_content += f"\n观察: 不存在可以执行的tool\n"
-            message.step_contents += [f"\n观察: 不存在可以执行的tool\n"]
-        for tool in message.tools:
-            if tool.name == message.tool_name:
-                tool_res = tool.func(**message.tool_params)
-                message.tool_answer = tool_res    
-                message.observation = tool_res
-                message.role_content += f"\n观察: {tool_res}\n"
-                message.step_content += f"\n观察: {tool_res}\n"
-                message.step_contents += [f"\n观察: {tool_res}\n"]
-
-        # logger.info(f"观察: {message.action_status}, {message.observation}")
-        return message
-    
-    def parser(self, message: Message) -> Message:
-        ''''''
-        content = message.role_content
-        parser_keys = ["action", "code_content", "code_filename", "tool_params", "plans"]
-        try:
-            s_json = self._parse_json(content)
-            message.action_status = s_json.get("action")
-            message.code_content = s_json.get("code_content")
-            message.tool_params = s_json.get("tool_params")
-            message.tool_name = s_json.get("tool_name")
-            message.code_filename = s_json.get("code_filename")
-            message.plans = s_json.get("plans")
-            # for parser_key in parser_keys:
-            #     message.action_status = content.get(parser_key)
-        except Exception as e:
-            # logger.warning(f"{traceback.format_exc()}")
-            action_value = self._match(r"'action':\s*'([^']*)'", content) if "'action'" in content else self._match(r'"action":\s*"([^"]*)"', content)
-            code_content_value = self._match(r"'code_content':\s*'([^']*)'", content) if "'code_content'" in content else self._match(r'"code_content":\s*"([^"]*)"', content)
-            filename_value = self._match(r"'code_filename':\s*'([^']*)'", content) if "'code_filename'" in content else self._match(r'"code_filename":\s*"([^"]*)"', content)
-            tool_params_value = self._match(r"'tool_params':\s*(\{[^{}]*\})", content, do_json=True) if "'tool_params'" in content \
-                                    else self._match(r'"tool_params":\s*(\{[^{}]*\})', content, do_json=True)
-            tool_name_value = self._match(r"'tool_name':\s*'([^']*)'", content) if "'tool_name'" in content else self._match(r'"tool_name":\s*"([^"]*)"', content)
-            plans_value = self._match(r"'plans':\s*(\[.*?\])", content, do_search=False) if "'plans'" in content else self._match(r'"plans":\s*(\[.*?\])', content, do_search=False, )
-            # re解析
-            message.action_status = action_value or "default"
-            message.code_content = code_content_value
-            message.code_filename = filename_value
-            message.tool_params = tool_params_value
-            message.tool_name = tool_name_value
-            message.plans = plans_value
-
-        # logger.debug(f"确认当前的action: {message.action_status}")
-
-        return message
-    
-    def parser_spec_key(self, content, key, do_search=True, do_json=False) -> str:
-        ''''''
-        key2pattern = {
-            "'action'": r"'action':\s*'([^']*)'", '"action"': r'"action":\s*"([^"]*)"',
-            "'code_content'": r"'code_content':\s*'([^']*)'", '"code_content"': r'"code_content":\s*"([^"]*)"',
-            "'code_filename'": r"'code_filename':\s*'([^']*)'", '"code_filename"': r'"code_filename":\s*"([^"]*)"',
-            "'tool_params'": r"'tool_params':\s*(\{[^{}]*\})", '"tool_params"': r'"tool_params":\s*(\{[^{}]*\})',
-            "'tool_name'": r"'tool_name':\s*'([^']*)'", '"tool_name"': r'"tool_name":\s*"([^"]*)"',
-            "'plans'": r"'plans':\s*(\[.*?\])", '"plans"': r'"plans":\s*(\[.*?\])',
-            "'content'": r"'content':\s*'([^']*)'", '"content"': r'"content":\s*"([^"]*)"',
-            }
+    # def get_extra_infos(self, message: Message) -> Message:
+    #     ''''''
+    #     if self.do_search:
+    #         message = self.get_search_retrieval(message)
         
-        s_json = self._parse_json(content)
-        try:
-            if s_json and key in s_json:
-                return str(s_json[key])
-        except:
-            pass
+    #     if self.do_doc_retrieval:
+    #         message = self.get_doc_retrieval(message)
 
-        keystr = f"'{key}'" if f"'{key}'" in content else f'"{key}"'
-        return self._match(key2pattern.get(keystr, fr"'{key}':\s*'([^']*)'"), content, do_search=do_search, do_json=do_json)
+    #     if self.do_tool_retrieval:
+    #         message = self.get_tool_retrieval(message)
+        
+    #     return message 
     
-    def _match(self, pattern, s, do_search=True, do_json=False):
-        try:
-            if do_search:
-                match = re.search(pattern, s)
-                if match:
-                    value = match.group(1).replace("\\n", "\n")
-                    if do_json:
-                        value = json.loads(value)
-                else:
-                    value = None
-            else:
-                match = re.findall(pattern, s, re.DOTALL)
-                if match:
-                    value = match[0]
-                    if do_json:
-                        value = json.loads(value)
-                else:
-                    value = None
-        except Exception as e:
-            logger.warning(f"{traceback.format_exc()}")
+    # def get_search_retrieval(self, message: Message,) -> Message:
+    #     SEARCH_ENGINES = {"duckduckgo": DDGSTool}
+    #     search_docs = []
+    #     for idx, doc in enumerate(SEARCH_ENGINES["duckduckgo"].run(message.role_content, 3)):
+    #         doc.update({"index": idx})
+    #         search_docs.append(Doc(**doc))
+    #     message.search_docs = search_docs
+    #     return message
+    
+    # def get_doc_retrieval(self, message: Message) -> Message:
+    #     query = message.role_content
+    #     knowledge_basename = message.doc_engine_name
+    #     top_k = message.top_k
+    #     score_threshold = message.score_threshold
+    #     if knowledge_basename:
+    #         docs = DocRetrieval.run(query, knowledge_basename, top_k, score_threshold)
+    #         message.db_docs = [Doc(**doc) for doc in docs]
+    #     return message
+    
+    # def get_code_retrieval(self, message: Message) -> Message:
+    #     # DocRetrieval.run("langchain是什么", "DSADSAD")
+    #     query = message.input_query
+    #     code_engine_name = message.code_engine_name
+    #     history_node_list = message.history_node_list
+    #     code_docs = CodeRetrieval.run(code_engine_name, query, code_limit=message.top_k, history_node_list=history_node_list)
+    #     message.code_docs = [CodeDoc(**doc) for doc in code_docs]
+    #     return message
+    
+    # def get_tool_retrieval(self, message: Message) -> Message:
+    #     return message
+    
+    # def step_router(self, message: Message) -> tuple[Message, ...]:
+    #     ''''''
+    #     # message = self.parser(message)
+    #     # logger.debug(f"message.action_status: {message.action_status}")
+    #     observation_message = None
+    #     if message.action_status == ActionStatus.CODING:
+    #         message, observation_message = self.code_step(message)
+    #     elif message.action_status == ActionStatus.TOOL_USING:
+    #         message, observation_message = self.tool_step(message)
 
-        # logger.debug(f"pattern: {pattern}, s: {s}, match: {match}")
-        return value
+    #     return message, observation_message
+
+    # def code_step(self, message: Message) -> Message:
+    #     '''execute code'''
+    #     # logger.debug(f"message.role_content: {message.role_content}, message.code_content: {message.code_content}")
+    #     code_answer = self.codebox.chat('```python\n{}```'.format(message.code_content))
+    #     code_prompt = f"执行上述代码后存在报错信息为 {code_answer.code_exe_response}，需要进行修复" \
+    #                 if code_answer.code_exe_type == "error" else f"执行上述代码后返回信息为 {code_answer.code_exe_response}"
+        
+    #     observation_message = Message(
+    #             role_name="observation",
+    #             role_type="func", #self.role.role_type,
+    #             role_content="",
+    #             step_content="",
+    #             input_query=message.code_content,
+    #             )
+    #     uid = str(uuid.uuid1())
+    #     if code_answer.code_exe_type == "image/png":
+    #         message.figures[uid] = code_answer.code_exe_response
+    #         message.code_answer = f"\n**Observation:**: 执行上述代码后生成一张图片, 图片名为{uid}\n"
+    #         message.observation = f"\n**Observation:**: 执行上述代码后生成一张图片, 图片名为{uid}\n"
+    #         message.step_content += f"\n**Observation:**: 执行上述代码后生成一张图片, 图片名为{uid}\n"
+    #         message.step_contents += [f"\n**Observation:**: 执行上述代码后生成一张图片, 图片名为{uid}\n"]
+    #         # message.role_content += f"\n**Observation:**:执行上述代码后生成一张图片, 图片名为{uid}\n"
+    #         observation_message.role_content = f"\n**Observation:**: 执行上述代码后生成一张图片, 图片名为{uid}\n"
+    #         observation_message.parsed_output = {"Observation": f"执行上述代码后生成一张图片, 图片名为{uid}"}
+    #     else:
+    #         message.code_answer = code_answer.code_exe_response
+    #         message.observation = code_answer.code_exe_response
+    #         message.step_content += f"\n**Observation:**: {code_prompt}\n"
+    #         message.step_contents += [f"\n**Observation:**: {code_prompt}\n"]
+    #         # message.role_content += f"\n**Observation:**: {code_prompt}\n"
+    #         observation_message.role_content = f"\n**Observation:**: {code_prompt}\n"
+    #         observation_message.parsed_output = {"Observation": code_prompt}
+    #     # logger.info(f"**Observation:** {message.action_status}, {message.observation}")
+    #     return message, observation_message
+
+    # def tool_step(self, message: Message) -> Message:
+    #     '''execute tool'''
+    #     # logger.debug(f"{message}")
+    #     observation_message = Message(
+    #             role_name="observation",
+    #             role_type="function", #self.role.role_type,
+    #             role_content="\n**Observation:** there is no tool can execute\n"    ,
+    #             step_content="",
+    #             input_query=str(message.tool_params),
+    #             tools=message.tools,
+    #             )
+    #     # logger.debug(f"message: {message.action_status}, {message.tool_name}, {message.tool_params}")
+    #     tool_names = [tool.name for tool in message.tools]
+    #     if message.tool_name not in tool_names:
+    #         message.tool_answer = "\n**Observation:** there is no tool can execute\n"    
+    #         message.observation = "\n**Observation:** there is no tool can execute\n"    
+    #         # message.role_content += f"\n**Observation:**: 不存在可以执行的tool\n"
+    #         message.step_content += f"\n**Observation:** there is no tool can execute\n"
+    #         message.step_contents += [f"\n**Observation:** there is no tool can execute\n"]
+    #         observation_message.role_content = f"\n**Observation:** there is no tool can execute\n"
+    #         observation_message.parsed_output = {"Observation": "there is no tool can execute\n"}
+    #     for tool in message.tools:
+    #         if tool.name == message.tool_name:
+    #             tool_res = tool.func(**message.tool_params.get("tool_params", {}))
+    #             logger.debug(f"tool_res {tool_res}")
+    #             message.tool_answer = tool_res    
+    #             message.observation = tool_res
+    #             # message.role_content += f"\n**Observation:**: {tool_res}\n"
+    #             message.step_content += f"\n**Observation:** {tool_res}\n"
+    #             message.step_contents += [f"\n**Observation:** {tool_res}\n"]
+    #             observation_message.role_content = f"\n**Observation:** {tool_res}\n"
+    #             observation_message.parsed_output = {"Observation": tool_res}
+    #             break
+
+    #     # logger.info(f"**Observation:** {message.action_status}, {message.observation}")
+    #     return message, observation_message
     
-    def _parse_json(self, s):
-        try:
-            pattern = r"```([^`]+)```"
-            match = re.findall(pattern, s)
-            if match:
-                return eval(match[0])
-        except:
-            pass
-        return None
+    # def parser(self, message: Message) -> Message:
+    #     ''''''
+    #     content = message.role_content
+    #     parser_keys = ["action", "code_content", "code_filename", "tool_params", "plans"]
+    #     try:
+    #         s_json = self._parse_json(content)
+    #         message.action_status = s_json.get("action")
+    #         message.code_content = s_json.get("code_content")
+    #         message.tool_params = s_json.get("tool_params")
+    #         message.tool_name = s_json.get("tool_name")
+    #         message.code_filename = s_json.get("code_filename")
+    #         message.plans = s_json.get("plans")
+    #         # for parser_key in parser_keys:
+    #         #     message.action_status = content.get(parser_key)
+    #     except Exception as e:
+    #         # logger.warning(f"{traceback.format_exc()}")
+    #         def parse_text_to_dict(text):
+    #             # Define a regular expression pattern to capture the key and value
+    #             main_pattern = r"\*\*(.+?):\*\*\s*(.*?)\s*(?=\*\*|$)"
+    #             list_pattern = r'```python\n(.*?)```'
+
+    #             # Use re.findall to find all main matches in the text
+    #             main_matches = re.findall(main_pattern, text, re.DOTALL)
+
+    #             # Convert main matches to a dictionary
+    #             parsed_dict = {key.strip(): value.strip() for key, value in main_matches}
+
+    #             for k, v in parsed_dict.items():
+    #                 for pattern in [list_pattern]:
+    #                     if "PLAN" != k: continue
+    #                     match_value = re.search(pattern, v, re.DOTALL)
+    #                     if match_value:
+    #                         # Add the code block to the dictionary
+    #                         parsed_dict[k] = eval(match_value.group(1).strip())
+    #                         break
+
+    #             return parsed_dict
+            
+    #         def extract_content_from_backticks(text):
+    #             code_blocks = []
+    #             lines = text.split('\n')
+    #             is_code_block = False
+    #             code_block = ''
+    #             language = ''
+    #             for line in lines:
+    #                 if line.startswith('```') and not is_code_block:
+    #                     is_code_block = True
+    #                     language = line[3:]
+    #                     code_block = ''
+    #                 elif line.startswith('```') and is_code_block:
+    #                     is_code_block = False
+    #                     code_blocks.append({language.strip(): code_block.strip()})
+    #                 elif is_code_block:
+    #                     code_block += line + '\n'
+    #             return code_blocks
+            
+    #         def parse_dict_to_dict(parsed_dict):
+    #             code_pattern = r'```python\n(.*?)```'
+    #             tool_pattern = r'```tool_params\n(.*?)```'
+                
+    #             pattern_dict = {"code": code_pattern, "tool_params": tool_pattern}
+    #             spec_parsed_dict = copy.deepcopy(parsed_dict)
+    #             for key, pattern in pattern_dict.items():
+    #                 for k, text in parsed_dict.items():
+    #                     # Search for the code block
+    #                     if not isinstance(text, str): continue
+    #                     _match = re.search(pattern, text, re.DOTALL)
+    #                     if _match:
+    #                         # Add the code block to the dictionary
+    #                         try:
+    #                             spec_parsed_dict[key] = json.loads(_match.group(1).strip())
+    #                         except:
+    #                             spec_parsed_dict[key] = _match.group(1).strip()
+    #                         break
+    #             return spec_parsed_dict
+            # def parse_dict_to_dict(parsed_dict):
+            #     code_pattern = r'```python\n(.*?)```'
+            #     tool_pattern = r'```json\n(.*?)```'
+                
+            #     pattern_dict = {"code": code_pattern, "json": tool_pattern}
+            #     spec_parsed_dict = copy.deepcopy(parsed_dict)
+            #     for key, pattern in pattern_dict.items():
+            #         for k, text in parsed_dict.items():
+            #             # Search for the code block
+            #             if not isinstance(text, str): continue
+            #             _match = re.search(pattern, text, re.DOTALL)
+            #             if _match:
+            #                 # Add the code block to the dictionary
+            #                 logger.debug(f"dsadsa {text}")
+            #                 try:
+            #                     spec_parsed_dict[key] = json.loads(_match.group(1).strip())
+            #                 except:
+            #                     spec_parsed_dict[key] = _match.group(1).strip()
+            #                 break
+            #     return spec_parsed_dict
+
+    #         parsed_dict = parse_text_to_dict(content)
+    #         spec_parsed_dict = parse_dict_to_dict(parsed_dict)
+    #         action_value = parsed_dict.get('Action Status')
+    #         if action_value:
+    #             action_value = action_value.lower()
+    #         logger.info(f'{self.role.role_name}: action_value: {action_value}')
+    #         # action_value = self._match(r"'action':\s*'([^']*)'", content) if "'action'" in content else self._match(r'"action":\s*"([^"]*)"', content)
+            
+            # code_content_value = spec_parsed_dict.get('code')
+            # # code_content_value = self._match(r"'code_content':\s*'([^']*)'", content) if "'code_content'" in content else self._match(r'"code_content":\s*"([^"]*)"', content)
+            # filename_value = self._match(r"'code_filename':\s*'([^']*)'", content) if "'code_filename'" in content else self._match(r'"code_filename":\s*"([^"]*)"', content)
+            # tool_params_value = spec_parsed_dict.get('tool_params')
+            # # tool_params_value = self._match(r"'tool_params':\s*(\{[^{}]*\})", content, do_json=True) if "'tool_params'" in content \
+            # #                         else self._match(r'"tool_params":\s*(\{[^{}]*\})', content, do_json=True)
+            # tool_name_value = self._match(r"'tool_name':\s*'([^']*)'", content) if "'tool_name'" in content else self._match(r'"tool_name":\s*"([^"]*)"', content)
+            # plans_value = self._match(r"'plans':\s*(\[.*?\])", content, do_search=False) if "'plans'" in content else self._match(r'"plans":\s*(\[.*?\])', content, do_search=False, )
+    #         # re解析
+    #         message.action_status = action_value or "default"
+    #         message.code_content = code_content_value
+    #         message.code_filename = filename_value
+    #         message.tool_params = tool_params_value
+    #         message.tool_name = tool_name_value
+    #         message.plans = plans_value
+    #         message.parsed_output = parsed_dict
+    #         message.spec_parsed_output = spec_parsed_dict
+            # code_content_value = spec_parsed_dict.get('code')
+            # # code_content_value = self._match(r"'code_content':\s*'([^']*)'", content) if "'code_content'" in content else self._match(r'"code_content":\s*"([^"]*)"', content)
+            # filename_value = self._match(r"'code_filename':\s*'([^']*)'", content) if "'code_filename'" in content else self._match(r'"code_filename":\s*"([^"]*)"', content)
+            # logger.debug(spec_parsed_dict)
+
+            # if action_value == 'tool_using':
+            #     tool_params_value = spec_parsed_dict.get('json')
+            # else:
+            #     tool_params_value = None
+            # # tool_params_value = self._match(r"'tool_params':\s*(\{[^{}]*\})", content, do_json=True) if "'tool_params'" in content \
+            # #                         else self._match(r'"tool_params":\s*(\{[^{}]*\})', content, do_json=True)
+            # tool_name_value = self._match(r"'tool_name':\s*'([^']*)'", content) if "'tool_name'" in content else self._match(r'"tool_name":\s*"([^"]*)"', content)
+            # plans_value = self._match(r"'plans':\s*(\[.*?\])", content, do_search=False) if "'plans'" in content else self._match(r'"plans":\s*(\[.*?\])', content, do_search=False, )
+            # # re解析
+            # message.action_status = action_value or "default"
+            # message.code_content = code_content_value
+            # message.code_filename = filename_value
+            # message.tool_params = tool_params_value
+            # message.tool_name = tool_name_value
+            # message.plans = plans_value
+            # message.parsed_output = parsed_dict
+            # message.spec_parsed_output = spec_parsed_dict
+
+    #     # logger.debug(f"确认当前的action: {message.action_status}")
+
+    #     return message
+    
+    # def parser_spec_key(self, content, key, do_search=True, do_json=False) -> str:
+    #     ''''''
+    #     key2pattern = {
+    #         "'action'": r"'action':\s*'([^']*)'", '"action"': r'"action":\s*"([^"]*)"',
+    #         "'code_content'": r"'code_content':\s*'([^']*)'", '"code_content"': r'"code_content":\s*"([^"]*)"',
+    #         "'code_filename'": r"'code_filename':\s*'([^']*)'", '"code_filename"': r'"code_filename":\s*"([^"]*)"',
+    #         "'tool_params'": r"'tool_params':\s*(\{[^{}]*\})", '"tool_params"': r'"tool_params":\s*(\{[^{}]*\})',
+    #         "'tool_name'": r"'tool_name':\s*'([^']*)'", '"tool_name"': r'"tool_name":\s*"([^"]*)"',
+    #         "'plans'": r"'plans':\s*(\[.*?\])", '"plans"': r'"plans":\s*(\[.*?\])',
+    #         "'content'": r"'content':\s*'([^']*)'", '"content"': r'"content":\s*"([^"]*)"',
+    #         }
+        
+    #     s_json = self._parse_json(content)
+    #     try:
+    #         if s_json and key in s_json:
+    #             return str(s_json[key])
+    #     except:
+    #         pass
+
+    #     keystr = f"'{key}'" if f"'{key}'" in content else f'"{key}"'
+    #     return self._match(key2pattern.get(keystr, fr"'{key}':\s*'([^']*)'"), content, do_search=do_search, do_json=do_json)
+    
+    # def _match(self, pattern, s, do_search=True, do_json=False):
+    #     try:
+    #         if do_search:
+    #             match = re.search(pattern, s)
+    #             if match:
+    #                 value = match.group(1).replace("\\n", "\n")
+    #                 if do_json:
+    #                     value = json.loads(value)
+    #             else:
+    #                 value = None
+    #         else:
+    #             match = re.findall(pattern, s, re.DOTALL)
+    #             if match:
+    #                 value = match[0]
+    #                 if do_json:
+    #                     value = json.loads(value)
+    #             else:
+    #                 value = None
+    #     except Exception as e:
+    #         logger.warning(f"{traceback.format_exc()}")
+
+    #     # logger.debug(f"pattern: {pattern}, s: {s}, match: {match}")
+    #     return value
+    
+    # def _parse_json(self, s):
+    #     try:
+    #         pattern = r"```([^`]+)```"
+    #         match = re.findall(pattern, s)
+    #         if match:
+    #             return eval(match[0])
+    #     except:
+    #         pass
+        # return None
 
     
-    def get_memory(self, ):
+    def get_memory(self, content_key="role_content"):
         return self.memory.to_tuple_messages(content_key="step_content")
     
-    def get_memory_str(self, ):
+    def get_memory_str(self, content_key="role_content"):
         return "\n".join([": ".join(i) for i in self.memory.to_tuple_messages(content_key="step_content")])

dev_opsgpt/connector/agents/check_agent.py

@@ -0,0 +1,110 @@
+from pydantic import BaseModel
+from typing import List, Union
+import re
+import json
+import traceback
+import copy
+from loguru import logger
+
+from langchain.prompts.chat import ChatPromptTemplate
+
+from dev_opsgpt.connector.schema import (
+    Memory, Task, Env, Role, Message, ActionStatus
+)
+from dev_opsgpt.llm_models import getChatModel
+from dev_opsgpt.connector.configs.agent_config import REACT_PROMPT_INPUT, CONTEXT_PROMPT_INPUT, QUERY_CONTEXT_PROMPT_INPUT
+
+from .base_agent import BaseAgent
+
+
+class CheckAgent(BaseAgent):
+    def __init__(
+            self, 
+            role: Role,
+            task: Task = None,
+            memory: Memory = None,
+            chat_turn: int = 1,
+            do_search: bool = False,
+            do_doc_retrieval: bool = False,
+            do_tool_retrieval: bool = False,
+            temperature: float = 0.2,
+            stop: Union[List[str], str] = None,
+            do_filter: bool = True,
+            do_use_self_memory: bool = True,
+            focus_agents: List[str] = [],
+            focus_message_keys: List[str] = [],
+            # prompt_mamnger: PromptManager
+            ):
+        
+        super().__init__(role, task, memory, chat_turn, do_search, do_doc_retrieval, 
+                         do_tool_retrieval, temperature, stop, do_filter,do_use_self_memory,
+                         focus_agents, focus_message_keys
+                         )
+
+    def create_prompt(
+            self, query: Message, memory: Memory =None, history: Memory = None, background: Memory = None, memory_pool: Memory=None, prompt_mamnger=None) -> str:
+        '''
+        role\task\tools\docs\memory
+        '''
+        # 
+        doc_infos = self.create_doc_prompt(query)
+        code_infos = self.create_codedoc_prompt(query)
+        # 
+        formatted_tools, tool_names = self.create_tools_prompt(query)
+        task_prompt = self.create_task_prompt(query)
+        background_prompt = self.create_background_prompt(background)
+        history_prompt = self.create_history_prompt(history)
+        selfmemory_prompt = self.create_selfmemory_prompt(memory, control_key="step_content")
+        
+        # react 流程是自身迭代过程，另外二次触发的是需要作为历史对话信息
+        # input_query = react_memory.to_tuple_messages(content_key="step_content")
+        input_query = query.input_query
+
+        # logger.debug(f"{self.role.role_name}  extra_system_prompt: {self.role.role_prompt}")
+        # logger.debug(f"{self.role.role_name}  input_query: {input_query}")
+        # logger.debug(f"{self.role.role_name}  doc_infos: {doc_infos}")
+        # logger.debug(f"{self.role.role_name}  tool_names: {tool_names}")
+        # prompt += "\n" + CHECK_PROMPT_INPUT.format(**{"query": input_query})
+        # prompt.format(**{"query": input_query})
+        
+        # extra_system_prompt = self.role.role_prompt
+        prompt = self.role.role_prompt.format(**{"query": input_query, "formatted_tools": formatted_tools, "tool_names": tool_names})
+
+        if "**Context:**" in self.role.role_prompt:
+            # logger.debug(f"parsed_output_list: {query.parsed_output_list}")
+            # input_query =  "'''" + "\n".join([f"*{k}*\n{v}" for i in background.get_parserd_output_list() for k,v in i.items() if "Action Status" !=k]) + "'''"
+            context =  "\n".join([f"*{k}*\n{v}" for i in background.get_parserd_output_list() for k,v in i.items() if "Action Status" !=k])
+            # logger.debug(context)
+            # logger.debug(f"parsed_output_list: {t}")
+            prompt += "\n" + QUERY_CONTEXT_PROMPT_INPUT.format(**{"query": query.origin_query, "context": context})
+        else:
+            prompt += "\n" + REACT_PROMPT_INPUT.format(**{"query": input_query})
+
+
+        task = query.task or self.task
+        if task_prompt is not None:
+            prompt += "\n" + task.task_prompt
+
+        # if doc_infos is not None and doc_infos!="" and doc_infos!="不存在知识库辅助信息":
+        #     prompt += f"\n知识库信息: {doc_infos}"
+
+        # if code_infos is not None and code_infos!="" and code_infos!="不存在代码库辅助信息":
+        #     prompt += f"\n代码库信息: {code_infos}"
+
+        # if background_prompt:
+        #     prompt += "\n" + background_prompt
+
+        # if history_prompt:
+        #     prompt += "\n" + history_prompt
+
+        # if selfmemory_prompt:
+        #     prompt += "\n" + selfmemory_prompt
+
+        # prompt = extra_system_prompt.format(**{"query": input_query, "doc_infos": doc_infos, "formatted_tools": formatted_tools, "tool_names": tool_names})
+        while "{{" in prompt or "}}" in prompt:
+            prompt = prompt.replace("{{", "{")
+            prompt = prompt.replace("}}", "}")
+
+        # logger.debug(f"{self.role.role_name}  prompt: {prompt}")
+        return prompt
+    

dev_opsgpt/connector/agents/executor_agent.py

@@ -0,0 +1,214 @@
+from pydantic import BaseModel
+from typing import List, Union, Tuple, Any
+import re
+import json
+import traceback
+import copy
+from loguru import logger
+
+from langchain.prompts.chat import ChatPromptTemplate
+
+from dev_opsgpt.connector.schema import (
+    Memory, Task, Env, Role, Message, ActionStatus
+)
+from dev_opsgpt.llm_models import getChatModel
+from dev_opsgpt.connector.configs.prompts import EXECUTOR_PROMPT_INPUT, BEGIN_PROMPT_INPUT
+from dev_opsgpt.connector.utils import parse_section
+
+from .base_agent import BaseAgent
+
+
+class ExecutorAgent(BaseAgent):
+    def __init__(
+            self, 
+            role: Role,
+            task: Task = None,
+            memory: Memory = None,
+            chat_turn: int = 1,
+            do_search: bool = False,
+            do_doc_retrieval: bool = False,
+            do_tool_retrieval: bool = False,
+            temperature: float = 0.2,
+            stop: Union[List[str], str] = None,
+            do_filter: bool = True,
+            do_use_self_memory: bool = True,
+            focus_agents: List[str] = [],
+            focus_message_keys: List[str] = [],
+            # prompt_mamnger: PromptManager
+            ):
+        
+        super().__init__(role, task, memory, chat_turn, do_search, do_doc_retrieval, 
+                         do_tool_retrieval, temperature, stop, do_filter,do_use_self_memory,
+                         focus_agents, focus_message_keys
+                         )
+        self.do_all_task = True # run all tasks
+
+    def arun(self, query: Message, history: Memory = None, background: Memory = None, memory_pool: Memory=None) -> Message:
+        '''agent reponse from multi-message'''
+        # insert query into memory
+        task_executor_memory = Memory(messages=[])
+        # insert query
+        output_message = Message(
+                role_name=self.role.role_name,
+                role_type="ai", #self.role.role_type,
+                role_content=query.input_query,
+                step_content="",
+                input_query=query.input_query,
+                tools=query.tools,
+                parsed_output_list=[query.parsed_output]
+            )
+        
+        self_memory = self.memory if self.do_use_self_memory else None
+
+        plan_step = int(query.parsed_output.get("PLAN_STEP", 0))
+        # 如果存在plan字段且plan字段为str的时候
+        if "PLAN" not in query.parsed_output or isinstance(query.parsed_output.get("PLAN", []), str) or plan_step >= len(query.parsed_output.get("PLAN", [])):
+            query_c = copy.deepcopy(query)
+            query_c.parsed_output = {"Question": query_c.input_query}
+            task_executor_memory.append(query_c)
+            for output_message, task_executor_memory in self._arun_step(output_message, query_c, self_memory, history, background, memory_pool, task_executor_memory):
+                pass
+            # task_executor_memory.append(query_c)
+            # content = "the execution step of the plan is exceed the planned scope."
+            # output_message.parsed_dict = {"Thought": content, "Action Status": "finished", "Action": content}
+            # task_executor_memory.append(output_message)
+
+        elif "PLAN" in query.parsed_output:
+            logger.debug(f"{query.parsed_output['PLAN']}")
+            if self.do_all_task:
+                # run all tasks step by step
+                for task_content in query.parsed_output["PLAN"][plan_step:]:
+                    # create your llm prompt
+                    query_c = copy.deepcopy(query)
+                    query_c.parsed_output = {"Question": task_content}
+                    task_executor_memory.append(query_c)
+                    for output_message, task_executor_memory in self._arun_step(output_message, query_c, self_memory, history, background, memory_pool, task_executor_memory):
+                        pass
+                    yield output_message
+            else:
+                query_c = copy.deepcopy(query)
+                task_content = query_c.parsed_output["PLAN"][plan_step]
+                query_c.parsed_output = {"Question": task_content}
+                task_executor_memory.append(query_c)
+                for output_message, task_executor_memory in self._arun_step(output_message, query_c, self_memory, history, background, memory_pool, task_executor_memory):
+                    pass
+                output_message.parsed_output.update({"CURRENT_STEP": plan_step})
+        # update self_memory
+        self.append_history(query)
+        self.append_history(output_message)
+        # logger.info(f"{self.role.role_name} currenct question: {output_message.input_query}\nllm_executor_run: {output_message.step_content}")
+        # logger.info(f"{self.role.role_name} currenct parserd_output_list: {output_message.parserd_output_list}")
+        output_message.input_query = output_message.role_content
+        # update memory pool
+        memory_pool.append(output_message)
+        yield output_message
+
+    def _arun_step(self, output_message: Message, query: Message, self_memory: Memory, 
+            history: Memory, background: Memory, memory_pool: Memory, 
+            react_memory: Memory) -> Union[Message, Memory]:
+        '''execute the llm predict by created prompt'''
+        prompt = self.create_prompt(query, self_memory, history, background, memory_pool=memory_pool, react_memory=react_memory)
+        content = self.llm.predict(prompt)
+        # logger.debug(f"{self.role.role_name} prompt: {prompt}")
+        logger.debug(f"{self.role.role_name} content: {content}")
+
+        output_message.role_content = content
+        output_message.role_contents += [content]
+        output_message.step_content += "\n"+output_message.role_content
+        output_message.step_contents + [output_message.role_content]
+
+        output_message = self.message_utils.parser(output_message)
+        # according the output to choose one action for code_content or tool_content
+        output_message, observation_message = self.message_utils.step_router(output_message)
+        # logger.debug(f"{self.role.role_name} content: {content}")
+        # update parserd_output_list
+        output_message.parsed_output_list.append(output_message.parsed_output)
+
+        react_message = copy.deepcopy(output_message)
+        react_memory.append(react_message)
+        if observation_message:
+            react_memory.append(observation_message)
+            output_message.parsed_output_list.append(observation_message.parsed_output)
+            logger.debug(f"{observation_message.role_name} content: {observation_message.role_content}")
+        yield output_message, react_memory
+    
+    def create_prompt(
+            self, query: Message, memory: Memory =None, history: Memory = None, background: Memory = None, memory_pool: Memory=None, react_memory: Memory = None, prompt_mamnger=None) -> str:
+        '''
+        role\task\tools\docs\memory
+        '''
+        # 
+        doc_infos = self.create_doc_prompt(query)
+        code_infos = self.create_codedoc_prompt(query)
+        # 
+        formatted_tools, tool_names = self.create_tools_prompt(query)
+        task_prompt = self.create_task_prompt(query)
+        background_prompt = self.create_background_prompt(background, control_key="step_content")
+        history_prompt = self.create_history_prompt(history)
+        selfmemory_prompt = self.create_selfmemory_prompt(memory, control_key="step_content")
+        
+        #
+        memory_pool_select_by_agent_key = self.select_memory_by_agent_key(memory_pool)
+        memory_pool_select_by_agent_key_context = '\n\n'.join([
+            f"*{k}*\n{v}" for parsed_output in memory_pool_select_by_agent_key.get_parserd_output_list() for k, v in parsed_output.items() if k not in ['Action Status']
+            ])
+        
+        DocInfos = ""
+        if doc_infos is not None and doc_infos!="" and doc_infos!="不存在知识库辅助信息":
+            DocInfos += f"\nDocument Information: {doc_infos}"
+
+        if code_infos is not None and code_infos!="" and code_infos!="不存在代码库辅助信息":
+            DocInfos += f"\nCodeBase Infomation: {code_infos}"
+
+        # extra_system_prompt = self.role.role_prompt
+        prompt = self.role.role_prompt.format(**{"formatted_tools": formatted_tools, "tool_names": tool_names})
+        
+        # input_query = react_memory.to_tuple_messages(content_key="role_content")
+        # logger.debug(f"get_parserd_dict {react_memory.get_parserd_output()}")
+        input_query = "\n".join(["\n".join([f"**{k}:**\n{v}" for k,v in _dict.items()]) for _dict in react_memory.get_parserd_output()])
+        # input_query = query.input_query + "\n".join([f"{v}" for k, v in input_query if v])
+        last_agent_parsed_output = "\n".join(["\n".join([f"*{k}*\n{v}" for k,v in _dict.items()]) for _dict in query.parsed_output_list])
+        react_parsed_output = "\n".join(["\n".join([f"*{k}_context*\n{v}" for k,v in _dict.items()]) for _dict in react_memory.get_parserd_output()[:-1]])
+        # 
+        prompt += "\n" + BEGIN_PROMPT_INPUT
+
+        input_keys = parse_section(self.role.role_prompt, 'Input Format')
+        if input_keys:
+            for input_key in input_keys:
+                if input_key == "Origin Query": 
+                    prompt += "\n**Origin Query:**\n" + query.origin_query
+                elif input_key == "DocInfos":
+                    prompt += "\n**DocInfos:**\n" + DocInfos
+                elif input_key == "Context":
+                    if self.focus_agents and memory_pool_select_by_agent_key_context:
+                        context = memory_pool_select_by_agent_key_context
+                    else:
+                        context = last_agent_parsed_output
+                    prompt += "\n**Context:**\n" + context + f"\n{react_parsed_output}"
+                elif input_key == "Question":
+                    prompt += "\n**Question:**\n" + query.parsed_output.get("Question")
+        else:
+            prompt += "\n" + input_query
+
+        task = query.task or self.task
+        # if task_prompt is not None:
+        #     prompt += "\n" + task.task_prompt
+
+        # if selfmemory_prompt:
+        #     prompt += "\n" + selfmemory_prompt
+
+        # if background_prompt:
+        #     prompt += "\n" + background_prompt
+
+        # if history_prompt:
+        #     prompt += "\n" + history_prompt
+
+        # prompt = extra_system_prompt.format(**{"query": input_query, "doc_infos": doc_infos, "formatted_tools": formatted_tools, "tool_names": tool_names})
+        while "{{" in prompt or "}}" in prompt:
+            prompt = prompt.replace("{{", "{")
+            prompt = prompt.replace("}}", "}")
+        return prompt
+    
+    def set_task(self, do_all_task):
+        '''set task exec type'''
+        self.do_all_task = do_all_task

dev_opsgpt/connector/agents/react_agent.py

@@ -1,15 +1,16 @@
 from pydantic import BaseModel
 from typing import List, Union
 import re
+import json
 import traceback
 import copy
 from loguru import logger
 
 from langchain.prompts.chat import ChatPromptTemplate
 
-from dev_opsgpt.connector.connector_schema import Message
-from dev_opsgpt.connector.shcema.memory import Memory
-from dev_opsgpt.connector.connector_schema import Task, Env, Role, Message, ActionStatus
+from dev_opsgpt.connector.schema import (
+    Memory, Task, Env, Role, Message, ActionStatus
+)
 from dev_opsgpt.llm_models import getChatModel
 from dev_opsgpt.connector.configs.agent_config import REACT_PROMPT_INPUT
 
@@ -27,63 +28,93 @@ class ReactAgent(BaseAgent):
             do_doc_retrieval: bool = False,
             do_tool_retrieval: bool = False,
             temperature: float = 0.2,
-            stop: Union[List[str], str] = "观察",
+            stop: Union[List[str], str] = None,
             do_filter: bool = True,
             do_use_self_memory: bool = True,
-            # docs_prompt: str,
+            focus_agents: List[str] = [],
+            focus_message_keys: List[str] = [],
             # prompt_mamnger: PromptManager
             ):
+        
         super().__init__(role, task, memory, chat_turn, do_search, do_doc_retrieval, 
-                         do_tool_retrieval, temperature, stop, do_filter,do_use_self_memory
+                         do_tool_retrieval, temperature, stop, do_filter,do_use_self_memory,
+                         focus_agents, focus_message_keys
                          )
 
-    def run(self, query: Message, history: Memory = None, background: Memory = None) -> Message:
+    def run(self, query: Message, history: Memory = None, background: Memory = None, memory_pool: Memory = None) -> Message:
+        '''agent reponse from multi-message'''
+        for message in self.arun(query, history, background, memory_pool):
+            pass
+        return message
+
+    def arun(self, query: Message, history: Memory = None, background: Memory = None, memory_pool: Memory = None) -> Message:
+        '''agent reponse from multi-message'''
         step_nums = copy.deepcopy(self.chat_turn)
-        react_memory = Memory([])
-        # 问题插入
+        react_memory = Memory(messages=[])
+        # insert query
         output_message = Message(
                 role_name=self.role.role_name,
                 role_type="ai", #self.role.role_type,
                 role_content=query.input_query,
-                step_content=query.input_query,
+                step_content="",
                 input_query=query.input_query,
-                tools=query.tools
+                tools=query.tools,
+                parsed_output_list=[query.parsed_output]
                 )
-        react_memory.append(output_message)
+        query_c = copy.deepcopy(query)
+        query_c.parsed_output = {"Question": "\n".join([f"{v}" for k, v in query.parsed_output.items() if k not in ["Action Status"]])}
+        react_memory.append(query_c)
+        self_memory = self.memory if self.do_use_self_memory else None
         idx = 0
+        # start to react
         while step_nums > 0:
             output_message.role_content = output_message.step_content
-            self_memory = self.memory if self.do_use_self_memory else None
-            prompt = self.create_prompt(query, self_memory, history, background, react_memory)
+            prompt = self.create_prompt(query, self_memory, history, background, react_memory, memory_pool)
             try:
                 content = self.llm.predict(prompt)
             except Exception as e:
                 logger.warning(f"error prompt: {prompt}")
                 raise Exception(traceback.format_exc())
             
-            output_message.role_content = content
+            output_message.role_content = "\n"+content
             output_message.role_contents += [content]
-            output_message.step_content += output_message.role_content
+            output_message.step_content += "\n"+output_message.role_content
             output_message.step_contents + [output_message.role_content]
+            yield output_message
 
             # logger.debug(f"{self.role.role_name}, {idx} iteration prompt: {prompt}")
-            # logger.info(f"{self.role.role_name}, {idx} iteration step_run: {output_message.role_content}")
+            logger.info(f"{self.role.role_name}, {idx} iteration step_run: {output_message.role_content}")
 
-            output_message = self.parser(output_message)
+            output_message = self.message_utils.parser(output_message)
             # when get finished signal can stop early
             if output_message.action_status == ActionStatus.FINISHED: break
             # according the output to choose one action for code_content or tool_content
-            output_message = self.step_router(output_message)
-            logger.info(f"{self.role.role_name} react_run: {output_message.role_content}")
+            output_message, observation_message = self.message_utils.step_router(output_message)
+            output_message.parsed_output_list.append(output_message.parsed_output)
             
+            react_message = copy.deepcopy(output_message)
+            react_memory.append(react_message)
+            if observation_message:
+                react_memory.append(observation_message)
+                output_message.parsed_output_list.append(observation_message.parsed_output)
+                # logger.debug(f"{observation_message.role_name} content: {observation_message.role_content}")
+            # logger.info(f"{self.role.role_name} currenct question: {output_message.input_query}\nllm_react_run: {output_message.role_content}")
+
             idx += 1
             step_nums -= 1
+            yield output_message
         # react' self_memory saved at last
         self.append_history(output_message)
-        return output_message
-
+        # update memory pool
+        # memory_pool.append(output_message)
+        output_message.input_query = query.input_query
+        # update memory pool
+        memory_pool.append(output_message)
+        yield output_message
+    
     def create_prompt(
-            self, query: Message, memory: Memory =None, history: Memory = None, background: Memory = None, react_memory: Memory = None, prompt_mamnger=None) -> str:
+            self, query: Message, memory: Memory =None, history: Memory = None, background: Memory = None, react_memory: Memory = None, memory_pool: Memory= None, 
+            prompt_mamnger=None) -> str:
         '''
         role\task\tools\docs\memory
         '''
@@ -100,35 +131,39 @@ class ReactAgent(BaseAgent):
         # extra_system_prompt = self.role.role_prompt
         prompt = self.role.role_prompt.format(**{"formatted_tools": formatted_tools, "tool_names": tool_names})
 
+        # react 流程是自身迭代过程，另外二次触发的是需要作为历史对话信息
+        # input_query = react_memory.to_tuple_messages(content_key="step_content")
+        # # input_query = query.input_query + "\n" + "\n".join([f"{v}" for k, v in input_query if v])
+        # input_query = "\n".join([f"{v}" for k, v in input_query if v])
+        input_query = "\n".join(["\n".join([f"**{k}:**\n{v}" for k,v in _dict.items()]) for _dict in react_memory.get_parserd_output()])
+        logger.debug(f"input_query: {input_query}")
+        
+        prompt += "\n" + REACT_PROMPT_INPUT.format(**{"query": input_query})
 
         task = query.task or self.task
-        if task_prompt is not None:
-            prompt += "\n" + task.task_prompt
+        # if task_prompt is not None:
+        #     prompt += "\n" + task.task_prompt
 
-        if doc_infos is not None and doc_infos!="" and doc_infos!="不存在知识库辅助信息":
-            prompt += f"\n知识库信息: {doc_infos}"
+        # if doc_infos is not None and doc_infos!="" and doc_infos!="不存在知识库辅助信息":
+        #     prompt += f"\n知识库信息: {doc_infos}"
 
-        if code_infos is not None and code_infos!="" and code_infos!="不存在代码库辅助信息":
-            prompt += f"\n代码库信息: {code_infos}"
+        # if code_infos is not None and code_infos!="" and code_infos!="不存在代码库辅助信息":
+        #     prompt += f"\n代码库信息: {code_infos}"
 
-        if background_prompt:
-            prompt += "\n" + background_prompt
+        # if background_prompt:
+        #     prompt += "\n" + background_prompt
 
-        if history_prompt:
-            prompt += "\n" + history_prompt
+        # if history_prompt:
+        #     prompt += "\n" + history_prompt
 
-        if selfmemory_prompt:
-            prompt += "\n" + selfmemory_prompt
-        
-        # react 流程是自身迭代过程，另外二次触发的是需要作为历史对话信息
-        input_query = react_memory.to_tuple_messages(content_key="step_content")
-        input_query = "\n".join([f"{v}" for k, v in input_query if v])
+        # if selfmemory_prompt:
+        #     prompt += "\n" + selfmemory_prompt
 
         # logger.debug(f"{self.role.role_name}  extra_system_prompt: {self.role.role_prompt}")
         # logger.debug(f"{self.role.role_name}  input_query: {input_query}")
         # logger.debug(f"{self.role.role_name}  doc_infos: {doc_infos}")
         # logger.debug(f"{self.role.role_name}  tool_names: {tool_names}")
-        prompt += "\n" + REACT_PROMPT_INPUT.format(**{"query": input_query})
+        # prompt += "\n" + REACT_PROMPT_INPUT.format(**{"query": input_query})
 
         # prompt = extra_system_prompt.format(**{"query": input_query, "doc_infos": doc_infos, "formatted_tools": formatted_tools, "tool_names": tool_names})
         while "{{" in prompt or "}}" in prompt:

dev_opsgpt/connector/agents/selector_agent.py

@@ -0,0 +1,109 @@
+from pydantic import BaseModel
+from typing import List, Union
+import re
+import json
+import traceback
+import copy
+from loguru import logger
+
+from langchain.prompts.chat import ChatPromptTemplate
+
+from dev_opsgpt.connector.schema import (
+    Memory, Task, Env, Role, Message, ActionStatus
+)
+from dev_opsgpt.llm_models import getChatModel
+from dev_opsgpt.connector.configs.agent_config import REACT_PROMPT_INPUT, CONTEXT_PROMPT_INPUT, QUERY_CONTEXT_PROMPT_INPUT
+
+from .base_agent import BaseAgent
+
+
+class SelectorAgent(BaseAgent):
+    def __init__(
+            self, 
+            role: Role,
+            task: Task = None,
+            memory: Memory = None,
+            chat_turn: int = 1,
+            do_search: bool = False,
+            do_doc_retrieval: bool = False,
+            do_tool_retrieval: bool = False,
+            temperature: float = 0.2,
+            stop: Union[List[str], str] = None,
+            do_filter: bool = True,
+            do_use_self_memory: bool = True,
+            focus_agents: List[str] = [],
+            focus_message_keys: List[str] = [],
+            # prompt_mamnger: PromptManager
+            ):
+        
+        super().__init__(role, task, memory, chat_turn, do_search, do_doc_retrieval, 
+                         do_tool_retrieval, temperature, stop, do_filter,do_use_self_memory,
+                         focus_agents, focus_message_keys
+                         )
+
+    def create_prompt(
+            self, query: Message, memory: Memory =None, history: Memory = None, background: Memory = None, memory_pool: Memory=None, prompt_mamnger=None) -> str:
+        '''
+        role\task\tools\docs\memory
+        '''
+        # 
+        doc_infos = self.create_doc_prompt(query)
+        code_infos = self.create_codedoc_prompt(query)
+        # 
+        formatted_tools, tool_names = self.create_tools_prompt(query)
+        task_prompt = self.create_task_prompt(query)
+        background_prompt = self.create_background_prompt(background)
+        history_prompt = self.create_history_prompt(history)
+        selfmemory_prompt = self.create_selfmemory_prompt(memory, control_key="step_content")
+        
+        # react 流程是自身迭代过程，另外二次触发的是需要作为历史对话信息
+        # input_query = react_memory.to_tuple_messages(content_key="step_content")
+        input_query = query.input_query
+
+        # logger.debug(f"{self.role.role_name}  extra_system_prompt: {self.role.role_prompt}")
+        logger.debug(f"{self.role.role_name}  input_query: {input_query}")
+        # logger.debug(f"{self.role.role_name}  doc_infos: {doc_infos}")
+        # logger.debug(f"{self.role.role_name}  tool_names: {tool_names}")
+        # prompt += "\n" + CHECK_PROMPT_INPUT.format(**{"query": input_query})
+        # prompt.format(**{"query": input_query})
+        
+        # extra_system_prompt = self.role.role_prompt
+        prompt = self.role.role_prompt.format(**{"query": input_query, "formatted_tools": formatted_tools, "tool_names": tool_names})
+
+        if "**Context:**" in self.role.role_prompt:
+            # logger.debug(f"parsed_output_list: {query.parsed_output_list}")
+            # input_query =  "'''" + "\n".join([f"*{k}*\n{v}" for i in background.get_parserd_output_list() for k,v in i.items() if "Action Status" !=k]) + "'''"
+            context =  "\n".join([f"*{k}*\n{v}" for i in background.get_parserd_output_list() for k,v in i.items() if "Action Status" !=k])
+            # logger.debug(f"parsed_output_list: {t}")
+            prompt += "\n" + QUERY_CONTEXT_PROMPT_INPUT.format(**{"query": query.origin_query, "context": context})
+        else:
+            prompt += "\n" + REACT_PROMPT_INPUT.format(**{"query": input_query})
+
+
+        task = query.task or self.task
+        if task_prompt is not None:
+            prompt += "\n" + task.task_prompt
+
+        # if doc_infos is not None and doc_infos!="" and doc_infos!="不存在知识库辅助信息":
+        #     prompt += f"\n知识库信息: {doc_infos}"
+
+        # if code_infos is not None and code_infos!="" and code_infos!="不存在代码库辅助信息":
+        #     prompt += f"\n代码库信息: {code_infos}"
+
+        # if background_prompt:
+        #     prompt += "\n" + background_prompt
+
+        # if history_prompt:
+        #     prompt += "\n" + history_prompt
+
+        # if selfmemory_prompt:
+        #     prompt += "\n" + selfmemory_prompt
+
+        # prompt = extra_system_prompt.format(**{"query": input_query, "doc_infos": doc_infos, "formatted_tools": formatted_tools, "tool_names": tool_names})
+        while "{{" in prompt or "}}" in prompt:
+            prompt = prompt.replace("{{", "{")
+            prompt = prompt.replace("}}", "}")
+
+        # logger.debug(f"{self.role.role_name}  prompt: {prompt}")
+        return prompt
+    

dev_opsgpt/connector/chains/base_chain.py

@@ -7,13 +7,14 @@ import traceback
 import uuid
 import copy
 
-from dev_opsgpt.connector.agents import BaseAgent
+from dev_opsgpt.connector.agents import BaseAgent, CheckAgent
 from dev_opsgpt.tools.base_tool import BaseTools, Tool
-from dev_opsgpt.connector.shcema.memory import Memory
-from dev_opsgpt.connector.connector_schema import (
-    Role, Message, ActionStatus, ChainConfig,
+
+from dev_opsgpt.connector.schema import (
+    Memory, Role, Message, ActionStatus, ChainConfig,
     load_role_configs
 )
+from dev_opsgpt.connector.message_process import MessageUtils
 from dev_opsgpt.sandbox import PyCodeBox, CodeBoxResponse
 
 
@@ -37,7 +38,7 @@ class BaseChain:
         self.agents = agents
         self.chat_turn = chat_turn
         self.do_checker = do_checker
-        self.checker = BaseAgent(role=role_configs["checker"].role,
+        self.checker = CheckAgent(role=role_configs["checker"].role,
             task = None,
             memory = None,
             do_search = role_configs["checker"].do_search,
@@ -45,37 +46,64 @@ class BaseChain:
             do_tool_retrieval = role_configs["checker"].do_tool_retrieval,
             do_filter=False, do_use_self_memory=False)
         
-        self.global_memory = Memory([])
-        self.local_memory = Memory([])
-        self.do_code_exec = do_code_exec
-        self.codebox = PyCodeBox(
-            remote_url=SANDBOX_SERVER["url"],
-            remote_ip=SANDBOX_SERVER["host"],
-            remote_port=SANDBOX_SERVER["port"],
-            token="mytoken",
-            do_code_exe=True,
-            do_remote=SANDBOX_SERVER["do_remote"],
-            do_check_net=False
-            )
+        self.do_agent_selector = False
+        self.agent_selector = CheckAgent(role=role_configs["checker"].role,
+            task = None,
+            memory = None,
+            do_search = role_configs["checker"].do_search,
+            do_doc_retrieval = role_configs["checker"].do_doc_retrieval,
+            do_tool_retrieval = role_configs["checker"].do_tool_retrieval,
+            do_filter=False, do_use_self_memory=False)
+        
+        self.messageUtils = MessageUtils()
+        # all memory created by agent until instance deleted
+        self.global_memory = Memory(messages=[])
+        # self.do_code_exec = do_code_exec
+        # self.codebox = PyCodeBox(
+        #     remote_url=SANDBOX_SERVER["url"],
+        #     remote_ip=SANDBOX_SERVER["host"],
+        #     remote_port=SANDBOX_SERVER["port"],
+        #     token="mytoken",
+        #     do_code_exe=True,
+        #     do_remote=SANDBOX_SERVER["do_remote"],
+        #     do_check_net=False
+        #     )
 
-    def step(self, query: Message, history: Memory = None, background: Memory = None) -> Message:
+    def step(self, query: Message, history: Memory = None, background: Memory = None, memory_pool: Memory = None) -> Message:
         '''execute chain'''
-        local_memory = Memory([])
+        for output_message, local_memory in self.astep(query, history, background, memory_pool):
+            pass
+        return output_message, local_memory
+    
+    def astep(self, query: Message, history: Memory = None, background: Memory = None, memory_pool: Memory = None) -> Message:
+        '''execute chain'''
+        local_memory = Memory(messages=[])
         input_message = copy.deepcopy(query)
         step_nums = copy.deepcopy(self.chat_turn)
         check_message = None
 
         self.global_memory.append(input_message)
-        local_memory.append(input_message)
+        # local_memory.append(input_message)
         while step_nums > 0:
 
-            for agent in self.agents:
-                output_message = agent.run(input_message, history, background=background)
-                output_message = self.inherit_extrainfo(input_message, output_message)
+            if self.do_agent_selector:
+                agent_message = copy.deepcopy(query)
+                agent_message.agents = self.agents
+                for selectory_message in self.agent_selector.arun(query, background=self.global_memory, memory_pool=memory_pool):
+                    pass
+                selectory_message = self.messageUtils.parser(selectory_message)
+                selectory_message = self.messageUtils.filter(selectory_message)
+                agent = self.agents[selectory_message.agent_index]
+                # selector agent to execure next task
+                for output_message in agent.arun(input_message, history, background=background, memory_pool=memory_pool):
+                    # logger.debug(f"local_memory {local_memory + output_message}")
+                    yield output_message, local_memory + output_message
+                
+                output_message = self.messageUtils.inherit_extrainfo(input_message, output_message)
                 # according the output to choose one action for code_content or tool_content
-                logger.info(f"{agent.role.role_name} message: {output_message.role_content}")
-                output_message = self.parser(output_message)
-                # output_message = self.step_router(output_message)
+                # logger.info(f"{agent.role.role_name}\nmessage: {output_message.step_content}\nquery: {output_message.input_query}")
+                output_message = self.messageUtils.parser(output_message)
+                yield output_message, local_memory + output_message
 
                 input_message = output_message
                 self.global_memory.append(output_message)
@@ -84,192 +112,243 @@ class BaseChain:
                 # when get finished signal can stop early
                 if output_message.action_status == ActionStatus.FINISHED: 
                     break
+            else:
+                for agent in self.agents:
+                    for output_message in agent.arun(input_message, history, background=background, memory_pool=memory_pool):
+                        # logger.debug(f"local_memory {local_memory + output_message}")
+                        yield output_message, local_memory + output_message
+                        
+                    output_message = self.messageUtils.inherit_extrainfo(input_message, output_message)
+                    # according the output to choose one action for code_content or tool_content
+                    # logger.info(f"{agent.role.role_name}  currenct message: {output_message.step_content}\n next llm question: {output_message.input_query}")
+                    output_message = self.messageUtils.parser(output_message)
+                    yield output_message, local_memory + output_message
+                    # output_message = self.step_router(output_message)
 
-            if self.do_checker:
-                logger.debug(f"{self.checker.role.role_name} input global memory: {self.global_memory.to_str_messages(content_key='step_content')}")
-                check_message = self.checker.run(query, background=self.global_memory)
-                check_message = self.parser(check_message)
-                check_message = self.filter(check_message)
-                check_message = self.inherit_extrainfo(output_message, check_message)
-                logger.debug(f"{self.checker.role.role_name}: {check_message.role_content}")
+                    input_message = output_message
+                    self.global_memory.append(output_message)
 
-                if check_message.action_status == ActionStatus.FINISHED: 
-                    self.global_memory.append(check_message)
-                    break
+                    local_memory.append(output_message)
+                    # when get finished signal can stop early
+                    if output_message.action_status == ActionStatus.FINISHED:
+                        action_status = False
+                        break
+
+                if self.do_checker and self.chat_turn > 1:
+                    # logger.debug(f"{self.checker.role.role_name} input global memory: {self.global_memory.to_str_messages(content_key='step_content', return_all=False)}")
+                    for check_message in self.checker.arun(query, background=local_memory, memory_pool=memory_pool):
+                        pass
+                    check_message = self.messageUtils.parser(check_message)
+                    check_message = self.messageUtils.filter(check_message)
+                    check_message = self.messageUtils.inherit_extrainfo(output_message, check_message)
+                    logger.debug(f"{self.checker.role.role_name}: {check_message.role_content}")
+
+                    if check_message.action_status == ActionStatus.FINISHED: 
+                        self.global_memory.append(check_message)
+                        break
 
             step_nums -= 1
-
-        return check_message or output_message, local_memory
-    
-    def step_router(self, message: Message) -> Message:
-        ''''''
-        # message = self.parser(message)
-        # logger.debug(f"message.action_status: {message.action_status}")
-        if message.action_status == ActionStatus.CODING:
-            message = self.code_step(message)
-        elif message.action_status == ActionStatus.TOOL_USING:
-            message = self.tool_step(message)
-
-        return message
-
-    def code_step(self, message: Message) -> Message:
-        '''execute code'''
-        # logger.debug(f"message.role_content: {message.role_content}, message.code_content: {message.code_content}")
-        code_answer = self.codebox.chat('```python\n{}```'.format(message.code_content))
-        uid = str(uuid.uuid1())
-        if code_answer.code_exe_type == "image/png":
-            message.figures[uid] = code_answer.code_exe_response
-            message.code_answer = f"\n观察: 执行代码后获得输出一张图片, 文件名为{uid}\n"
-            message.observation = f"\n观察: 执行代码后获得输出一张图片, 文件名为{uid}\n"
-            message.step_content += f"\n观察: 执行代码后获得输出一张图片, 文件名为{uid}\n"
-            message.step_contents += [f"\n观察: 执行代码后获得输出一张图片, 文件名为{uid}\n"]
-            message.role_content += f"\n执行代码后获得输出一张图片, 文件名为{uid}\n"
-        else:
-            message.code_answer = code_answer.code_exe_response
-            message.observation = code_answer.code_exe_response
-            message.step_content += f"\n观察: 执行代码后获得输出是 {code_answer.code_exe_response}\n"
-            message.step_contents += [f"\n观察: 执行代码后获得输出是 {code_answer.code_exe_response}\n"]
-            message.role_content += f"\n观察: 执行代码后获得输出是 {code_answer.code_exe_response}\n"
-        logger.info(f"观察: {message.action_status}, {message.observation}")
-        return message
-
-    def tool_step(self, message: Message) -> Message:
-        '''execute tool'''
-        # logger.debug(f"message: {message.action_status}, {message.tool_name}, {message.tool_params}")
-        tool_names = [tool.name for tool in message.tools]
-        if message.tool_name not in tool_names:
-            message.tool_answer = "不存在可以执行的tool"    
-            message.observation = "不存在可以执行的tool"    
-            message.role_content += f"\n观察: 不存在可以执行的tool\n"
-            message.step_content += f"\n观察: 不存在可以执行的tool\n"
-            message.step_contents += [f"\n观察: 不存在可以执行的tool\n"]
-        for tool in message.tools:
-            if tool.name == message.tool_name:
-                tool_res = tool.func(**message.tool_params)
-                message.tool_answer = tool_res    
-                message.observation = tool_res
-                message.role_content += f"\n观察: {tool_res}\n"
-                message.step_content += f"\n观察: {tool_res}\n"
-                message.step_contents += [f"\n观察: {tool_res}\n"]
-        return message
-
-    def filter(self, message: Message, stop=None) -> Message:
-
-        tool_params = self.parser_spec_key(message.role_content, "tool_params")
-        code_content = self.parser_spec_key(message.role_content, "code_content")
-        plan = self.parser_spec_key(message.role_content, "plan")
-        plans = self.parser_spec_key(message.role_content, "plans", do_search=False)
-        content = self.parser_spec_key(message.role_content, "content", do_search=False)
-
-        # logger.debug(f"tool_params: {tool_params}, code_content: {code_content}, plan: {plan}, plans: {plans}, content: {content}")
-        role_content = tool_params or code_content or plan or plans or content
-        message.role_content = role_content or message.role_content
-        return message
-    
-    def parser(self, message: Message) -> Message:
-        ''''''
-        content = message.role_content
-        parser_keys = ["action", "code_content", "code_filename", "tool_params", "plans"]
-        try:
-            s_json = self._parse_json(content)
-            message.action_status = s_json.get("action")
-            message.code_content = s_json.get("code_content")
-            message.tool_params = s_json.get("tool_params")
-            message.tool_name = s_json.get("tool_name")
-            message.code_filename = s_json.get("code_filename")
-            message.plans = s_json.get("plans")
-            # for parser_key in parser_keys:
-            #     message.action_status = content.get(parser_key)
-        except Exception as e:
-            # logger.warning(f"{traceback.format_exc()}")
-            action_value = self._match(r"'action':\s*'([^']*)'", content) if "'action'" in content else self._match(r'"action":\s*"([^"]*)"', content)
-            code_content_value = self._match(r"'code_content':\s*'([^']*)'", content) if "'code_content'" in content else self._match(r'"code_content":\s*"([^"]*)"', content)
-            filename_value = self._match(r"'code_filename':\s*'([^']*)'", content) if "'code_filename'" in content else self._match(r'"code_filename":\s*"([^"]*)"', content)
-            tool_params_value = self._match(r"'tool_params':\s*(\{[^{}]*\})", content, do_json=True) if "'tool_params'" in content \
-                                    else self._match(r'"tool_params":\s*(\{[^{}]*\})', content, do_json=True)
-            tool_name_value = self._match(r"'tool_name':\s*'([^']*)'", content) if "'tool_name'" in content else self._match(r'"tool_name":\s*"([^"]*)"', content)
-            plans_value = self._match(r"'plans':\s*(\[.*?\])", content, do_search=False) if "'plans'" in content else self._match(r'"plans":\s*(\[.*?\])', content, do_search=False, )
-            # re解析
-            message.action_status = action_value or "default"
-            message.code_content = code_content_value
-            message.code_filename = filename_value
-            message.tool_params = tool_params_value
-            message.tool_name = tool_name_value
-            message.plans = plans_value
-
-        logger.debug(f"确认当前的action: {message.action_status}")
-
-        return message
-    
-    def parser_spec_key(self, content, key, do_search=True, do_json=False) -> str:
-        ''''''
-        key2pattern = {
-            "'action'": r"'action':\s*'([^']*)'", '"action"': r'"action":\s*"([^"]*)"',
-            "'code_content'": r"'code_content':\s*'([^']*)'", '"code_content"': r'"code_content":\s*"([^"]*)"',
-            "'code_filename'": r"'code_filename':\s*'([^']*)'", '"code_filename"': r'"code_filename":\s*"([^"]*)"',
-            "'tool_params'": r"'tool_params':\s*(\{[^{}]*\})", '"tool_params"': r'"tool_params":\s*(\{[^{}]*\})',
-            "'tool_name'": r"'tool_name':\s*'([^']*)'", '"tool_name"': r'"tool_name":\s*"([^"]*)"',
-            "'plans'": r"'plans':\s*(\[.*?\])", '"plans"': r'"plans":\s*(\[.*?\])',
-            "'content'": r"'content':\s*'([^']*)'", '"content"': r'"content":\s*"([^"]*)"',
-            }
+        # 
+        output_message = check_message or output_message # 返回chain和checker的结果
+        output_message.input_query = query.input_query # chain和chain之间消息通信不改变问题
+        yield output_message, local_memory
         
-        s_json = self._parse_json(content)
-        try:
-            if s_json and key in s_json:
-                return str(s_json[key])
-        except:
-            pass
+    # def step_router(self, message: Message) -> Message:
+    #     ''''''
+    #     # message = self.parser(message)
+    #     # logger.debug(f"message.action_status: {message.action_status}")
+    #     if message.action_status == ActionStatus.CODING:
+    #         message = self.code_step(message)
+    #     elif message.action_status == ActionStatus.TOOL_USING:
+    #         message = self.tool_step(message)
 
-        keystr = f"'{key}'" if f"'{key}'" in content else f'"{key}"'
-        return self._match(key2pattern.get(keystr, fr"'{key}':\s*'([^']*)'"), content, do_search=do_search, do_json=do_json)
-    
-    def _match(self, pattern, s, do_search=True, do_json=False):
-        try:
-            if do_search:
-                match = re.search(pattern, s)
-                if match:
-                    value = match.group(1).replace("\\n", "\n")
-                    if do_json:
-                        value = json.loads(value)
-                else:
-                    value = None
-            else:
-                match = re.findall(pattern, s, re.DOTALL)
-                if match:
-                    value = match[0]
-                    if do_json:
-                        value = json.loads(value)
-                else:
-                    value = None
-        except Exception as e:
-            logger.warning(f"{traceback.format_exc()}")
+    #     return message
 
-        # logger.debug(f"pattern: {pattern}, s: {s}, match: {match}")
-        return value
-    
-    def _parse_json(self, s):
-        try:
-            pattern = r"```([^`]+)```"
-            match = re.findall(pattern, s)
-            if match:
-                return eval(match[0])
-        except:
-            pass
-        return None
+    # def code_step(self, message: Message) -> Message:
+    #     '''execute code'''
+    #     # logger.debug(f"message.role_content: {message.role_content}, message.code_content: {message.code_content}")
+    #     code_answer = self.codebox.chat('```python\n{}```'.format(message.code_content))
+    #     uid = str(uuid.uuid1())
+    #     if code_answer.code_exe_type == "image/png":
+    #         message.figures[uid] = code_answer.code_exe_response
+    #         message.code_answer = f"\n观察: 执行代码后获得输出一张图片, 文件名为{uid}\n"
+    #         message.observation = f"\n观察: 执行代码后获得输出一张图片, 文件名为{uid}\n"
+    #         message.step_content += f"\n观察: 执行代码后获得输出一张图片, 文件名为{uid}\n"
+    #         message.step_contents += [f"\n观察: 执行代码后获得输出一张图片, 文件名为{uid}\n"]
+    #         message.role_content += f"\n执行代码后获得输出一张图片, 文件名为{uid}\n"
+    #     else:
+    #         message.code_answer = code_answer.code_exe_response
+    #         message.observation = code_answer.code_exe_response
+    #         message.step_content += f"\n观察: 执行代码后获得输出是 {code_answer.code_exe_response}\n"
+    #         message.step_contents += [f"\n观察: 执行代码后获得输出是 {code_answer.code_exe_response}\n"]
+    #         message.role_content += f"\n观察: 执行代码后获得输出是 {code_answer.code_exe_response}\n"
+    #     logger.info(f"观察: {message.action_status}, {message.observation}")
+    #     return message
 
-    def inherit_extrainfo(self, input_message: Message, output_message: Message):
-        output_message.db_docs = input_message.db_docs
-        output_message.search_docs = input_message.search_docs
-        output_message.code_docs = input_message.code_docs
-        output_message.figures.update(input_message.figures)
-        return output_message
+    # def tool_step(self, message: Message) -> Message:
+    #     '''execute tool'''
+    #     # logger.debug(f"message: {message.action_status}, {message.tool_name}, {message.tool_params}")
+    #     tool_names = [tool.name for tool in message.tools]
+    #     if message.tool_name not in tool_names:
+    #         message.tool_answer = "不存在可以执行的tool"    
+    #         message.observation = "不存在可以执行的tool"    
+    #         message.role_content += f"\n观察: 不存在可以执行的tool\n"
+    #         message.step_content += f"\n观察: 不存在可以执行的tool\n"
+    #         message.step_contents += [f"\n观察: 不存在可以执行的tool\n"]
+    #     for tool in message.tools:
+    #         if tool.name == message.tool_name:
+    #             tool_res = tool.func(**message.tool_params)
+    #             message.tool_answer = tool_res    
+    #             message.observation = tool_res
+    #             message.role_content += f"\n观察: {tool_res}\n"
+    #             message.step_content += f"\n观察: {tool_res}\n"
+    #             message.step_contents += [f"\n观察: {tool_res}\n"]
+    #     return message
+
+    # def filter(self, message: Message, stop=None) -> Message:
+
+    #     tool_params = self.parser_spec_key(message.role_content, "tool_params")
+    #     code_content = self.parser_spec_key(message.role_content, "code_content")
+    #     plan = self.parser_spec_key(message.role_content, "plan")
+    #     plans = self.parser_spec_key(message.role_content, "plans", do_search=False)
+    #     content = self.parser_spec_key(message.role_content, "content", do_search=False)
+
+    #     # logger.debug(f"tool_params: {tool_params}, code_content: {code_content}, plan: {plan}, plans: {plans}, content: {content}")
+    #     role_content = tool_params or code_content or plan or plans or content
+    #     message.role_content = role_content or message.role_content
+    #     return message
     
-    def get_memory(self, do_all_memory=True, content_key="role_content") -> Memory:
-        memory = self.global_memory if do_all_memory else self.local_memory
+    # def parser(self, message: Message) -> Message:
+    #     ''''''
+    #     content = message.role_content
+    #     parser_keys = ["action", "code_content", "code_filename", "tool_params", "plans"]
+    #     try:
+    #         s_json = self._parse_json(content)
+    #         message.action_status = s_json.get("action")
+    #         message.code_content = s_json.get("code_content")
+    #         message.tool_params = s_json.get("tool_params")
+    #         message.tool_name = s_json.get("tool_name")
+    #         message.code_filename = s_json.get("code_filename")
+    #         message.plans = s_json.get("plans")
+    #         # for parser_key in parser_keys:
+    #         #     message.action_status = content.get(parser_key)
+    #     except Exception as e:
+    #         # logger.warning(f"{traceback.format_exc()}")
+    #         def parse_text_to_dict(text):
+    #             # Define a regular expression pattern to capture the key and value
+    #             main_pattern = r"\*\*(.+?):\*\*\s*(.*?)\s*(?=\*\*|$)"
+    #             code_pattern = r'```python\n(.*?)```'
+
+    #             # Use re.findall to find all main matches in the text
+    #             main_matches = re.findall(main_pattern, text, re.DOTALL)
+
+    #             # Convert main matches to a dictionary
+    #             parsed_dict = {key.strip(): value.strip() for key, value in main_matches}
+
+    #             # Search for the code block
+    #             code_match = re.search(code_pattern, text, re.DOTALL)
+    #             if code_match:
+    #                 # Add the code block to the dictionary
+    #                 parsed_dict['code'] = code_match.group(1).strip()
+
+    #             return parsed_dict
+            
+    #         parsed_dict = parse_text_to_dict(content)
+    #         action_value = parsed_dict.get('Action Status')
+    #         if action_value:
+    #             action_value = action_value.lower()
+    #         logger.debug(f'action_value: {action_value}')
+    #         # action_value = self._match(r"'action':\s*'([^']*)'", content) if "'action'" in content else self._match(r'"action":\s*"([^"]*)"', content)
+            
+    #         code_content_value = parsed_dict.get('code')
+    #         # code_content_value = self._match(r"'code_content':\s*'([^']*)'", content) if "'code_content'" in content else self._match(r'"code_content":\s*"([^"]*)"', content)
+    #         filename_value = self._match(r"'code_filename':\s*'([^']*)'", content) if "'code_filename'" in content else self._match(r'"code_filename":\s*"([^"]*)"', content)
+    #         tool_params_value = self._match(r"'tool_params':\s*(\{[^{}]*\})", content, do_json=True) if "'tool_params'" in content \
+    #                                 else self._match(r'"tool_params":\s*(\{[^{}]*\})', content, do_json=True)
+    #         tool_name_value = self._match(r"'tool_name':\s*'([^']*)'", content) if "'tool_name'" in content else self._match(r'"tool_name":\s*"([^"]*)"', content)
+    #         plans_value = self._match(r"'plans':\s*(\[.*?\])", content, do_search=False) if "'plans'" in content else self._match(r'"plans":\s*(\[.*?\])', content, do_search=False, )
+    #         # re解析
+    #         message.action_status = action_value or "default"
+    #         message.code_content = code_content_value
+    #         message.code_filename = filename_value
+    #         message.tool_params = tool_params_value
+    #         message.tool_name = tool_name_value
+    #         message.plans = plans_value
+
+    #     # logger.debug(f"确认当前的action: {message.action_status}")
+
+    #     return message
+    
+    # def parser_spec_key(self, content, key, do_search=True, do_json=False) -> str:
+    #     ''''''
+    #     key2pattern = {
+    #         "'action'": r"'action':\s*'([^']*)'", '"action"': r'"action":\s*"([^"]*)"',
+    #         "'code_content'": r"'code_content':\s*'([^']*)'", '"code_content"': r'"code_content":\s*"([^"]*)"',
+    #         "'code_filename'": r"'code_filename':\s*'([^']*)'", '"code_filename"': r'"code_filename":\s*"([^"]*)"',
+    #         "'tool_params'": r"'tool_params':\s*(\{[^{}]*\})", '"tool_params"': r'"tool_params":\s*(\{[^{}]*\})',
+    #         "'tool_name'": r"'tool_name':\s*'([^']*)'", '"tool_name"': r'"tool_name":\s*"([^"]*)"',
+    #         "'plans'": r"'plans':\s*(\[.*?\])", '"plans"': r'"plans":\s*(\[.*?\])',
+    #         "'content'": r"'content':\s*'([^']*)'", '"content"': r'"content":\s*"([^"]*)"',
+    #         }
+        
+    #     s_json = self._parse_json(content)
+    #     try:
+    #         if s_json and key in s_json:
+    #             return str(s_json[key])
+    #     except:
+    #         pass
+
+    #     keystr = f"'{key}'" if f"'{key}'" in content else f'"{key}"'
+    #     return self._match(key2pattern.get(keystr, fr"'{key}':\s*'([^']*)'"), content, do_search=do_search, do_json=do_json)
+    
+    # def _match(self, pattern, s, do_search=True, do_json=False):
+    #     try:
+    #         if do_search:
+    #             match = re.search(pattern, s)
+    #             if match:
+    #                 value = match.group(1).replace("\\n", "\n")
+    #                 if do_json:
+    #                     value = json.loads(value)
+    #             else:
+    #                 value = None
+    #         else:
+    #             match = re.findall(pattern, s, re.DOTALL)
+    #             if match:
+    #                 value = match[0]
+    #                 if do_json:
+    #                     value = json.loads(value)
+    #             else:
+    #                 value = None
+    #     except Exception as e:
+    #         logger.warning(f"{traceback.format_exc()}")
+
+    #     # logger.debug(f"pattern: {pattern}, s: {s}, match: {match}")
+    #     return value
+    
+    # def _parse_json(self, s):
+    #     try:
+    #         pattern = r"```([^`]+)```"
+    #         match = re.findall(pattern, s)
+    #         if match:
+    #             return eval(match[0])
+    #     except:
+    #         pass
+    #     return None
+
+    # def inherit_extrainfo(self, input_message: Message, output_message: Message):
+    #     output_message.db_docs = input_message.db_docs
+    #     output_message.search_docs = input_message.search_docs
+    #     output_message.code_docs = input_message.code_docs
+    #     output_message.origin_query = input_message.origin_query
+    #     output_message.figures.update(input_message.figures)
+    #     return output_message
+    
+    def get_memory(self, content_key="role_content") -> Memory:
+        memory = self.global_memory
         return memory.to_tuple_messages(content_key=content_key)
     
-    def get_memory_str(self, do_all_memory=True, content_key="role_content") -> Memory:
-        memory = self.global_memory if do_all_memory else self.local_memory
+    def get_memory_str(self, content_key="role_content") -> Memory:
+        memory = self.global_memory
         # for i in memory.to_tuple_messages(content_key=content_key):
         #     logger.debug(f"{i}")
         return "\n".join([": ".join(i) for i in memory.to_tuple_messages(content_key=content_key)])

dev_opsgpt/connector/chains/chains.py

@@ -1,28 +1,18 @@
 from typing import List
+from loguru import logger
+import copy
 from dev_opsgpt.connector.agents import BaseAgent
 from .base_chain import BaseChain
 
+from dev_opsgpt.connector.agents import BaseAgent, CheckAgent
+from dev_opsgpt.connector.schema import (
+    Memory, Role, Message, ActionStatus, ChainConfig,
+    load_role_configs
+)
 
 
-class simpleChatChain(BaseChain):
-
-    def __init__(self, agents: List[BaseAgent], do_code_exec: bool = False) -> None:
-        super().__init__(agents, do_code_exec)
-
-
-class toolChatChain(BaseChain):
-
-    def __init__(self, agents: List[BaseAgent], do_code_exec: bool = False) -> None:
-        super().__init__(agents, do_code_exec)
-
-
-class dataAnalystChain(BaseChain):
-
-    def __init__(self, agents: List[BaseAgent], do_code_exec: bool = False) -> None:
-        super().__init__(agents, do_code_exec)
-
-
-class plannerChain(BaseChain):
+
+class ExecutorRefineChain(BaseChain):
 
     def __init__(self, agents: List[BaseAgent], do_code_exec: bool = False) -> None:
         super().__init__(agents, do_code_exec)

dev_opsgpt/connector/configs/agent_config.py

@@ -1,213 +1,33 @@
 from enum import Enum
+from .prompts import (
+    REACT_PROMPT_INPUT, CHECK_PROMPT_INPUT, EXECUTOR_PROMPT_INPUT, CONTEXT_PROMPT_INPUT, QUERY_CONTEXT_PROMPT_INPUT,PLAN_PROMPT_INPUT,
+    RECOGNIZE_INTENTION_PROMPT,
+    CHECKER_TEMPLATE_PROMPT,
+    CONV_SUMMARY_PROMPT,
+    QA_PROMPT, CODE_QA_PROMPT, QA_TEMPLATE_PROMPT,
+    EXECUTOR_TEMPLATE_PROMPT,
+    REFINE_TEMPLATE_PROMPT,
+    PLANNER_TEMPLATE_PROMPT, GENERAL_PLANNER_PROMPT, DATA_PLANNER_PROMPT, TOOL_PLANNER_PROMPT,
+    PRD_WRITER_METAGPT_PROMPT, DESIGN_WRITER_METAGPT_PROMPT, TASK_WRITER_METAGPT_PROMPT, CODE_WRITER_METAGPT_PROMPT,
+    REACT_TEMPLATE_PROMPT,
+    REACT_TOOL_PROMPT, REACT_CODE_PROMPT, REACT_TOOL_AND_CODE_PLANNER_PROMPT, REACT_TOOL_AND_CODE_PROMPT
+)
+
 
 
 class AgentType:
     REACT = "ReactAgent"
+    EXECUTOR = "ExecutorAgent"
     ONE_STEP = "BaseAgent"
     DEFAULT = "BaseAgent"
 
 
-REACT_TOOL_PROMPT = """尽可能地以有帮助和准确的方式回应人类。您可以使用以下工具:
-{formatted_tools}
-使用json blob来指定一个工具，提供一个action关键字（工具名称）和一个tool_params关键字（工具输入）。
-有效的"action"值为："finished" 或 "tool_using" (使用工具来回答问题)
-有效的"tool_name"值为：{tool_names}
-请仅在每个$JSON_BLOB中提供一个action，如下所示：
-```
-{{{{
-"action": $ACTION,
-"tool_name": $TOOL_NAME
-"tool_params": $INPUT
-}}}}
-```
-
-按照以下格式进行回应：
-问题：输入问题以回答
-思考：考虑之前和之后的步骤
-行动：
-```
-$JSON_BLOB
-```
-观察：行动结果
-...（重复思考/行动/观察N次）
-思考：我知道该如何回应
-行动：
-```
-{{{{
-"action": "finished",
-"tool_name": "notool"
-"tool_params": "最终返回答案给到用户"
-}}}}
-```
-"""
-
-REACT_PROMPT_INPUT = '''下面开始！记住根据问题进行返回需要生成的答案
-问题: {query}'''
-
-
-REACT_CODE_PROMPT = """尽可能地以有帮助和准确的方式回应人类，能够逐步编写可执行并打印变量的代码来解决问题
-使用 JSON Blob 来指定一个返回的内容，提供一个 action（行动）和一个 code （生成代码）。
-有效的 'action' 值为：'coding'(结合总结下述思维链过程编写下一步的可执行代码) or 'finished' （总结下述思维链过程可回答问题）。
-在每个 $JSON_BLOB 中仅提供一个 action，如下所示：
-```
-{{{{'action': $ACTION,'code_content': $CODE}}}}
-```
-
-按照以下思维链格式进行回应：
-问题：输入问题以回答
-思考：考虑之前和之后的步骤
-行动：
-```
-$JSON_BLOB
-```
-观察：行动结果
-...（重复思考/行动/观察N次）
-思考：我知道该如何回应
-行动：
-```
-{{{{
-"action": "finished",
-"code_content": "总结上述思维链过程回答问题"
-}}}}
-```
-"""
-
-GENERAL_PLANNER_PROMPT = """你是一个通用计划拆解助手，将问题拆解问题成各个详细明确的步骤计划或直接回答问题，尽可能地以有帮助和准确的方式回应人类，
-使用 JSON Blob 来指定一个返回的内容，提供一个 action（行动）和一个 plans （生成的计划）。
-有效的 'action' 值为：'planning'(拆解计划) or 'only_answer' （不需要拆解问题即可直接回答问题）。
-有效的 'plans' 值为: 一个任务列表，按顺序写出需要执行的计划
-在每个 $JSON_BLOB 中仅提供一个 action，如下所示：
-```
-{{'action': 'planning', 'plans': [$PLAN1, $PLAN2, $PLAN3, ..., $PLANN], }}
-或者
-{{'action': 'only_answer', 'plans': "直接回答问题", }}
-```
-
-按照以下格式进行回应：
-问题：输入问题以回答
-行动：
-```
-$JSON_BLOB
-```
-"""
-
-DATA_PLANNER_PROMPT = """你是一个数据分析助手，能够根据问题来制定一个详细明确的数据分析计划，尽可能地以有帮助和准确的方式回应人类，
-使用 JSON Blob 来指定一个返回的内容，提供一个 action（行动）和一个 plans （生成的计划）。
-有效的 'action' 值为：'planning'(拆解计划) or 'only_answer' （不需要拆解问题即可直接回答问题）。
-有效的 'plans' 值为: 一份数据分析计划清单，按顺序排列，用文本表示
-在每个 $JSON_BLOB 中仅提供一个 action，如下所示：
-```
-{{'action': 'planning', 'plans': '$PLAN1, $PLAN2, ..., $PLAN3' }}
-```
-
-按照以下格式进行回应：
-问题：输入问题以回答
-行动：
-```
-$JSON_BLOB
-```
-"""
-
-TOOL_PLANNER_PROMPT = """你是一个工具使用过程的计划拆解助手，将问题拆解为一系列的工具使用计划，若没有可用工具则直接回答问题，尽可能地以有帮助和准确的方式回应人类，你可以使用以下工具:
-{formatted_tools}
-使用 JSON Blob 来指定一个返回的内容，提供一个 action（行动）和一个 plans （生成的计划）。
-有效的 'action' 值为：'planning'(拆解计划) or 'only_answer' （不需要拆解问题即可直接回答问题）。
-有效的 'plans' 值为: 一个任务列表，按顺序写出需要使用的工具和使用该工具的理由
-在每个 $JSON_BLOB 中仅提供一个 action，如下两个示例所示：
-```
-{{'action': 'planning', 'plans': [$PLAN1, $PLAN2, $PLAN3, ..., $PLANN], }}
-```
-或者 若无法通过以上工具解决问题，则直接回答问题
-```
-{{'action': 'only_answer', 'plans': "直接回答问题", }}
-```
-
-按照以下格式进行回应：
-问题：输入问题以回答
-行动：
-```
-$JSON_BLOB
-```
-"""
-
-
-RECOGNIZE_INTENTION_PROMPT = """你是一个任务决策助手，能够将理解用户意图并决策采取最合适的行动，尽可能地以有帮助和准确的方式回应人类，
-使用 JSON Blob 来指定一个返回的内容，提供一个 action（行动）。
-有效的 'action' 值为：'planning'(需要先进行拆解计划) or 'only_answer' （不需要拆解问题即可直接回答问题）or "tool_using" (使用工具来回答问题) or 'coding'(生成可执行的代码)。
-在每个 $JSON_BLOB 中仅提供一个 action，如下所示：
-```
-{{'action': $ACTION}}
-```
-按照以下格式进行回应：
-问题：输入问题以回答
-行动：$ACTION
-```
-$JSON_BLOB
-```
-"""
-
-
-CHECKER_PROMPT = """尽可能地以有帮助和准确的方式回应人类，判断问题是否得到解答，同时展现解答的过程和内容
-使用 JSON Blob 来指定一个返回的内容，提供一个 action（行动）。
-有效的 'action' 值为：'finished'(任务已经可以通过“背景信息”和“对话信息”回答问题) or 'continue' （“背景信息”和“对话信息”不足以回答问题）。
-在每个 $JSON_BLOB 中仅提供一个 action，如下所示：
-```
-{{'action': $ACTION, 'content': '提取“背景信息”和“对话信息”中信息来回答问题'}}
-```
-按照以下格式进行回应：
-问题：输入问题以回答
-行动：$ACTION
-```
-$JSON_BLOB
-```
-"""
-
-CONV_SUMMARY_PROMPT = """尽可能地以有帮助和准确的方式回应人类，根据“背景信息”中的有效信息回答问题，
-使用 JSON Blob 来指定一个返回的内容，提供一个 action（行动）。
-有效的 'action' 值为：'finished'(任务已经可以通过上下文信息可以回答) or 'continue' （根据背景信息回答问题）。
-在每个 $JSON_BLOB 中仅提供一个 action，如下所示：
-```
-{{'action': $ACTION, 'content': '根据背景信息回答问题'}}
-```
-按照以下格式进行回应：
-问题：输入问题以回答
-行动：
-```
-$JSON_BLOB
-```
-"""
-
-CONV_SUMMARY_PROMPT = """尽可能地以有帮助和准确的方式回应人类
-根据“背景信息”中的有效信息回答问题，同时展现解答的过程和内容
-若能根“背景信息”回答问题，则直接回答
-否则，总结“背景信息”的内容
-"""
-
-
-
-QA_PROMPT = """根据已知信息，简洁和专业的来回答问题。如果无法从中得到答案，请说 “根据已知信息无法回答该问题”，不允许在答案中添加编造成分，答案请使用中文。 
-使用 JSON Blob 来指定一个返回的内容，提供一个 action（行动）。
-有效的 'action' 值为：'finished'(任务已经可以通过上下文信息可以回答) or 'continue' （上下文信息不足以回答问题）。
-在每个 $JSON_BLOB 中仅提供一个 action，如下所示：
-```
-{{'action': $ACTION, 'content': '总结对话内容'}}
-```
-按照以下格式进行回应：
-问题：输入问题以回答
-行动：$ACTION
-```
-$JSON_BLOB
-```
-"""
-
-CODE_QA_PROMPT = """【指令】根据已知信息来回答问题"""
-
 
 AGETN_CONFIGS = {
     "checker": {
         "role": {
-            "role_prompt": CHECKER_PROMPT,
-            "role_type": "ai",
+            "role_prompt": CHECKER_TEMPLATE_PROMPT,
+            "role_type": "assistant",
             "role_name": "checker",
             "role_desc": "",
             "agent_type": "BaseAgent"
@@ -220,7 +40,7 @@ AGETN_CONFIGS = {
     "conv_summary": {
         "role": {
             "role_prompt": CONV_SUMMARY_PROMPT,
-            "role_type": "ai",
+            "role_type": "assistant",
             "role_name": "conv_summary",
             "role_desc": "",
             "agent_type": "BaseAgent"
@@ -232,8 +52,8 @@ AGETN_CONFIGS = {
     },
     "general_planner": {
         "role": {
-            "role_prompt": GENERAL_PLANNER_PROMPT,
-            "role_type": "ai",
+            "role_prompt": PLANNER_TEMPLATE_PROMPT,
+            "role_type": "assistant",
             "role_name": "general_planner",
             "role_desc": "",
             "agent_type": "BaseAgent"
@@ -243,10 +63,37 @@ AGETN_CONFIGS = {
         "do_doc_retrieval": False,
         "do_tool_retrieval": False
     },
+    "executor": {
+        "role": {
+            "role_prompt": EXECUTOR_TEMPLATE_PROMPT,
+            "role_type": "assistant",
+            "role_name": "executor",
+            "role_desc": "",
+            "agent_type": "ExecutorAgent",
+        },
+        "stop": "\n**Observation:**",
+        "chat_turn": 1,
+        "do_search": False,
+        "do_doc_retrieval": False,
+        "do_tool_retrieval": False
+    },
+    "base_refiner": {
+        "role": {
+            "role_prompt": REFINE_TEMPLATE_PROMPT,
+            "role_type": "assistant",
+            "role_name": "base_refiner",
+            "role_desc": "",
+            "agent_type": "BaseAgent"
+        },
+        "chat_turn": 1,
+        "do_search": False,
+        "do_doc_retrieval": False,
+        "do_tool_retrieval": False
+    },
     "planner": {
         "role": {
             "role_prompt": DATA_PLANNER_PROMPT,
-            "role_type": "ai",
+            "role_type": "assistant",
             "role_name": "planner",
             "role_desc": "",
             "agent_type": "BaseAgent"
@@ -259,7 +106,7 @@ AGETN_CONFIGS = {
     "intention_recognizer": {
         "role": {
             "role_prompt": RECOGNIZE_INTENTION_PROMPT,
-            "role_type": "ai",
+            "role_type": "assistant",
             "role_name": "intention_recognizer",
             "role_desc": "",
             "agent_type": "BaseAgent"
@@ -272,7 +119,7 @@ AGETN_CONFIGS = {
     "tool_planner": {
         "role": {
             "role_prompt": TOOL_PLANNER_PROMPT,
-            "role_type": "ai",
+            "role_type": "assistant",
             "role_name": "tool_planner",
             "role_desc": "",
             "agent_type": "BaseAgent"
@@ -282,10 +129,37 @@ AGETN_CONFIGS = {
         "do_doc_retrieval": False,
         "do_tool_retrieval": False
     },
+    "tool_and_code_react": {
+        "role": {
+            "role_prompt": REACT_TOOL_AND_CODE_PROMPT,
+            "role_type": "assistant",
+            "role_name": "tool_and_code_react",
+            "role_desc": "",
+            "agent_type": "ReactAgent",
+        },
+        "stop": "\n**Observation:**",
+        "chat_turn": 7,
+        "do_search": False,
+        "do_doc_retrieval": False,
+        "do_tool_retrieval": False
+    },
+    "tool_and_code_planner": {
+        "role": {
+            "role_prompt": REACT_TOOL_AND_CODE_PLANNER_PROMPT,
+            "role_type": "assistant",
+            "role_name": "tool_and_code_planner",
+            "role_desc": "",
+            "agent_type": "BaseAgent"
+        },
+        "chat_turn": 1,
+        "do_search": False,
+        "do_doc_retrieval": False,
+        "do_tool_retrieval": False
+    },
     "tool_react": {
         "role": {
             "role_prompt": REACT_TOOL_PROMPT,
-            "role_type": "ai",
+            "role_type": "assistant",
             "role_name": "tool_react",
             "role_desc": "",
             "agent_type": "ReactAgent"
@@ -294,12 +168,12 @@ AGETN_CONFIGS = {
         "do_search": False,
         "do_doc_retrieval": False,
         "do_tool_retrieval": False,
-        "stop": "观察"
+        "stop": "\n**Observation:**"
     },
     "code_react": {
         "role": {
             "role_prompt": REACT_CODE_PROMPT,
-            "role_type": "ai",
+            "role_type": "assistant",
             "role_name": "code_react",
             "role_desc": "",
             "agent_type": "ReactAgent"
@@ -308,25 +182,25 @@ AGETN_CONFIGS = {
         "do_search": False,
         "do_doc_retrieval": False,
         "do_tool_retrieval": False,
-        "stop": "观察"
+        "stop": "\n**Observation:**"
     },
     "qaer": {
         "role": {
-            "role_prompt": QA_PROMPT,
-            "role_type": "ai",
+            "role_prompt": QA_TEMPLATE_PROMPT,
+            "role_type": "assistant",
             "role_name": "qaer",
             "role_desc": "",
             "agent_type": "BaseAgent"
         },
         "chat_turn": 1,
         "do_search": False,
-        "do_doc_retrieval": True,
+        "do_doc_retrieval": False,
         "do_tool_retrieval": False
     },
     "code_qaer": {
         "role": {
             "role_prompt": CODE_QA_PROMPT ,
-            "role_type": "ai",
+            "role_type": "assistant",
             "role_name": "code_qaer",
             "role_desc": "",
             "agent_type": "BaseAgent"
@@ -338,8 +212,8 @@ AGETN_CONFIGS = {
     },
     "searcher": {
         "role": {
-            "role_prompt": QA_PROMPT,
-            "role_type": "ai",
+            "role_prompt": QA_TEMPLATE_PROMPT,
+            "role_type": "assistant",
             "role_name": "searcher",
             "role_desc": "",
             "agent_type": "BaseAgent"
@@ -349,62 +223,65 @@ AGETN_CONFIGS = {
         "do_doc_retrieval": False,
         "do_tool_retrieval": False
     },
-    "answer": {
+    "metaGPT_PRD": {
         "role": {
-            "role_prompt": "",
-            "role_type": "ai",
-            "role_name": "answer",
+            "role_prompt": PRD_WRITER_METAGPT_PROMPT,
+            "role_type": "assistant",
+            "role_name": "metaGPT_PRD",
             "role_desc": "",
             "agent_type": "BaseAgent"
         },
         "chat_turn": 1,
         "do_search": False,
         "do_doc_retrieval": False,
-        "do_tool_retrieval": False
+        "do_tool_retrieval": False,
+        "focus_agents": [],
+        "focus_message_keys": [],
     },
-    "data_analyst": {
+
+    "metaGPT_DESIGN": {
         "role": {
-            "role_prompt": """你是一个数据分析的代码开发助手，能够编写可执行的代码来完成相关的数据分析问题，使用 JSON Blob 来指定一个返回的内容，通过提供一个 action（行动）和一个 code （生成代码）和 一个 file_name （指定保存文件）。\
-                有效的 'action' 值为：'coding'(生成可执行的代码) or 'finished' （不生成代码并直接返回答案）。在每个 $JSON_BLOB 中仅提供一个 action，如下所示：\
-                ```\n{{'action': $ACTION,'code_content': $CODE, 'code_filename': $FILE_NAME}}```\
-                下面开始！记住根据问题进行返回需要生成的答案，格式为 ```JSON_BLOB```""",
-            "role_type": "ai",
-            "role_name": "data_analyst",
+            "role_prompt": DESIGN_WRITER_METAGPT_PROMPT,
+            "role_type": "assistant",
+            "role_name": "metaGPT_DESIGN",
             "role_desc": "",
             "agent_type": "BaseAgent"
         },
         "chat_turn": 1,
         "do_search": False,
         "do_doc_retrieval": False,
-        "do_tool_retrieval": False
+        "do_tool_retrieval": False,
+        "focus_agents": ["metaGPT_PRD"],
+        "focus_message_keys": [],
     },
-    "deveploer": {
+    "metaGPT_TASK": {
         "role": {
-            "role_prompt": """你是一个代码开发助手，能够编写可执行的代码来完成问题，使用 JSON Blob 来指定一个返回的内容，通过提供一个 action（行动）和一个 code （生成代码）和 一个 file_name （指定保存文件）。\
-                有效的 'action' 值为：'coding'(生成可执行的代码) or 'finished' （不生成代码并直接返回答案）。在每个 $JSON_BLOB 中仅提供一个 action，如下所示：\
-                ```\n{{'action': $ACTION,'code_content': $CODE, 'code_filename': $FILE_NAME}}```\
-                下面开始！记住根据问题进行返回需要生成的答案，格式为 ```JSON_BLOB```""",
-            "role_type": "ai",
-            "role_name": "deveploer",
+            "role_prompt": TASK_WRITER_METAGPT_PROMPT,
+            "role_type": "assistant",
+            "role_name": "metaGPT_TASK",
             "role_desc": "",
             "agent_type": "BaseAgent"
         },
         "chat_turn": 1,
         "do_search": False,
         "do_doc_retrieval": False,
-        "do_tool_retrieval": False
+        "do_tool_retrieval": False,
+        "focus_agents": ["metaGPT_DESIGN"],
+        "focus_message_keys": [],
     },
-    "tester": {
+    "metaGPT_CODER": {
         "role": {
-            "role_prompt": "你是一个QA问答的助手，能够尽可能准确地回答问题，下面请逐步思考问题并回答",
-            "role_type": "ai",
-            "role_name": "tester",
+            "role_prompt": CODE_WRITER_METAGPT_PROMPT,
+            "role_type": "assistant",
+            "role_name": "metaGPT_CODER",
             "role_desc": "",
-            "agent_type": "BaseAgent"
+            "agent_type": "ExecutorAgent"
         },
         "chat_turn": 1,
         "do_search": False,
         "do_doc_retrieval": False,
-        "do_tool_retrieval": False
-    }
+        "do_tool_retrieval": False,
+        "focus_agents": ["metaGPT_DESIGN", "metaGPT_TASK"],
+        "focus_message_keys": [],
+    },
 }

dev_opsgpt/connector/configs/agent_prompt/design_writer.yaml

@@ -0,0 +1,99 @@
+You are a Architect, named Bob, your goal is Design a concise, usable, complete python system, and the constraint is Try to specify good open source tools as much as possible. 
+
+# Context
+## Original Requirements:
+Create a snake game.
+
+## Product Goals:
+Develop a highly addictive and engaging snake game.
+Provide a user-friendly and intuitive user interface.
+Implement various levels and challenges to keep the players entertained.
+## User Stories:
+As a user, I want to be able to control the snake's movement using arrow keys or touch gestures.
+As a user, I want to see my score and progress displayed on the screen.
+As a user, I want to be able to pause and resume the game at any time.
+As a user, I want to be challenged with different obstacles and levels as I progress.
+As a user, I want to have the option to compete with other players and compare my scores.
+## Competitive Analysis:
+Python Snake Game: A simple snake game implemented in Python with basic features and limited levels.
+Snake.io: A multiplayer online snake game with competitive gameplay and high engagement.
+Slither.io: Another multiplayer online snake game with a larger player base and addictive gameplay.
+Snake Zone: A mobile snake game with various power-ups and challenges.
+Snake Mania: A classic snake game with modern graphics and smooth controls.
+Snake Rush: A fast-paced snake game with time-limited challenges.
+Snake Master: A snake game with unique themes and customizable snakes.
+
+## Requirement Analysis:
+The product should be a highly addictive and engaging snake game with a user-friendly interface. It should provide various levels and challenges to keep the players entertained. The game should have smooth controls and allow the users to compete with each other.
+
+## Requirement Pool:
+```
+[
+    ["Implement different levels with increasing difficulty", "P0"],
+    ["Allow users to control the snake using arrow keys or touch gestures", "P0"],
+    ["Display the score and progress on the screen", "P1"],
+    ["Provide an option to pause and resume the game", "P1"],
+    ["Integrate leaderboards to enable competition among players", "P2"]
+]
+```
+## UI Design draft:
+The game will have a simple and clean interface. The main screen will display the snake, obstacles, and the score. The snake's movement can be controlled using arrow keys or touch gestures. There will be buttons to pause and resume the game. The level and difficulty will be indicated on the screen. The design will have a modern and visually appealing style with smooth animations.
+
+## Anything UNCLEAR:
+There are no unclear points.
+
+## Format example
+---
+## Implementation approach
+We will ...
+
+## Python package name
+```python
+"snake_game"
+```
+
+## File list
+```python
+[
+    "main.py",
+]
+```
+
+## Data structures and interface definitions
+```mermaid
+classDiagram
+    class Game{
+        +int score
+    }
+    ...
+    Game "1" -- "1" Food: has
+```
+
+## Program call flow
+```mermaid
+sequenceDiagram
+    participant M as Main
+    ...
+    G->>M: end game
+```
+
+## Anything UNCLEAR
+The requirement is clear to me.
+---
+-----
+Role: You are an architect; the goal is to design a SOTA PEP8-compliant python system; make the best use of good open source tools
+Requirement: Fill in the following missing information based on the context, note that all sections are response with code form separately
+Max Output: 8192 chars or 2048 tokens. Try to use them up.
+Attention: Use '##' to split sections, not '#', and '## <SECTION_NAME>' SHOULD WRITE BEFORE the code and triple quote.
+
+## Implementation approach: Provide as Plain text. Analyze the difficult points of the requirements, select the appropriate open-source framework.
+
+## Python package name: Provide as Python str with python triple quoto, concise and clear, characters only use a combination of all lowercase and underscores
+
+## File list: Provided as Python list[str], the list of ONLY REQUIRED files needed to write the program(LESS IS MORE!). Only need relative paths, comply with PEP8 standards. ALWAYS write a main.py or app.py here
+
+## Data structures and interface definitions: Use mermaid classDiagram code syntax, including classes (INCLUDING __init__ method) and functions (with type annotations), CLEARLY MARK the RELATIONSHIPS between classes, and comply with PEP8 standards. The data structures SHOULD BE VERY DETAILED and the API should be comprehensive with a complete design. 
+
+## Program call flow: Use sequenceDiagram code syntax, COMPLETE and VERY DETAILED, using CLASSES AND API DEFINED ABOVE accurately, covering the CRUD AND INIT of each object, SYNTAX MUST BE CORRECT.
+
+## Anything UNCLEAR: Provide as Plain text. Make clear here.

dev_opsgpt/connector/configs/agent_prompt/prd_writer.yaml

@@ -0,0 +1,101 @@
+You are a Product Manager, named Alice, your goal is Efficiently create a successful product, and the constraint is . 
+
+# Context
+## Original Requirements
+Create a snake game.
+
+## Search Information
+### Search Results
+
+### Search Summary
+
+## mermaid quadrantChart code syntax example. DONT USE QUOTO IN CODE DUE TO INVALID SYNTAX. Replace the <Campain X> with REAL COMPETITOR NAME
+```mermaid
+quadrantChart
+    title Reach and engagement of campaigns
+    x-axis Low Reach --> High Reach
+    y-axis Low Engagement --> High Engagement
+    quadrant-1 We should expand
+    quadrant-2 Need to promote
+    quadrant-3 Re-evaluate
+    quadrant-4 May be improved
+    "Campaign: A": [0.3, 0.6]
+    "Campaign B": [0.45, 0.23]
+    "Campaign C": [0.57, 0.69]
+    "Campaign D": [0.78, 0.34]
+    "Campaign E": [0.40, 0.34]
+    "Campaign F": [0.35, 0.78]
+    "Our Target Product": [0.5, 0.6]
+```
+
+## Format example
+---
+## Original Requirements
+The boss ... 
+
+## Product Goals
+```python
+[
+    "Create a ...",
+]
+```
+
+## User Stories
+```python
+[
+    "As a user, ...",
+]
+```
+
+## Competitive Analysis
+```python
+[
+    "Python Snake Game: ...",
+]
+```
+
+## Competitive Quadrant Chart
+```mermaid
+quadrantChart
+    title Reach and engagement of campaigns
+    ...
+    "Our Target Product": [0.6, 0.7]
+```
+
+## Requirement Analysis
+The product should be a ...
+
+## Requirement Pool
+```python
+[
+    ["End game ...", "P0"]
+]
+```
+
+## UI Design draft
+Give a basic function description, and a draft
+
+## Anything UNCLEAR
+There are no unclear points.
+---
+-----
+Role: You are a professional product manager; the goal is to design a concise, usable, efficient product
+Requirements: According to the context, fill in the following missing information, note that each sections are returned in Python code triple quote form seperatedly. If the requirements are unclear, ensure minimum viability and avoid excessive design
+ATTENTION: Use '##' to SPLIT SECTIONS, not '#'. AND '## <SECTION_NAME>' SHOULD WRITE BEFORE the code and triple quote. Output carefully referenced "Format example" in format.
+
+## Original Requirements: Provide as Plain text, place the polished complete original requirements here
+
+## Product Goals: Provided as Python list[str], up to 3 clear, orthogonal product goals. If the requirement itself is simple, the goal should also be simple
+
+## User Stories: Provided as Python list[str], up to 5 scenario-based user stories, If the requirement itself is simple, the user stories should also be less
+
+## Competitive Analysis: Provided as Python list[str], up to 7 competitive product analyses, consider as similar competitors as possible
+
+## Competitive Quadrant Chart: Use mermaid quadrantChart code syntax. up to 14 competitive products. Translation: Distribute these competitor scores evenly between 0 and 1, trying to conform to a normal distribution centered around 0.5 as much as possible.
+
+## Requirement Analysis: Provide as Plain text. Be simple. LESS IS MORE. Make your requirements less dumb. Delete the parts unnessasery.
+
+## Requirement Pool: Provided as Python list[list[str], the parameters are requirement description, priority(P0/P1/P2), respectively, comply with PEP standards; no more than 5 requirements and consider to make its difficulty lower
+
+## UI Design draft: Provide as Plain text. Be simple. Describe the elements and functions, also provide a simple style description and layout description.
+## Anything UNCLEAR: Provide as Plain text. Make clear here.

dev_opsgpt/connector/configs/agent_prompt/review_code.yaml

@@ -0,0 +1,177 @@
+
+NOTICE
+Role: You are a professional software engineer, and your main task is to review the code. You need to ensure that the code conforms to the PEP8 standards, is elegantly designed and modularized, easy to read and maintain, and is written in Python 3.9 (or in another programming language).
+ATTENTION: Use '##' to SPLIT SECTIONS, not '#'. Output format carefully referenced "Format example".
+
+## Code Review: Based on the following context and code, and following the check list, Provide key, clear, concise, and specific code modification suggestions, up to 5.
+```
+1. Check 0: Is the code implemented as per the requirements?
+2. Check 1: Are there any issues with the code logic?
+3. Check 2: Does the existing code follow the "Data structures and interface definitions"?
+4. Check 3: Is there a function in the code that is omitted or not fully implemented that needs to be implemented?
+5. Check 4: Does the code have unnecessary or lack dependencies?
+```
+
+## Rewrite Code: point.py Base on "Code Review" and the source code, rewrite code with triple quotes. Do your utmost to optimize THIS SINGLE FILE. 
+-----
+# Context
+## Implementation approach
+For the snake game, we can use the Pygame library, which is an open-source and easy-to-use library for game development in Python. Pygame provides a simple and efficient way to handle graphics, sound, and user input, making it suitable for developing a snake game.
+
+## Python package name
+```
+"snake_game"
+```
+## File list
+````
+[
+    "main.py",
+]
+```
+## Data structures and interface definitions
+```
+classDiagram
+    class Game:
+        -int score
+        -bool paused
+        +__init__()
+        +start_game()
+        +handle_input(key: int)
+        +update_game()
+        +draw_game()
+        +game_over()
+    
+    class Snake:
+        -list[Point] body
+        -Point dir
+        -bool alive
+        +__init__(start_pos: Point)
+        +move()
+        +change_direction(dir: Point)
+        +grow()
+        +get_head() -> Point
+        +get_body() -> list[Point]
+        +is_alive() -> bool
+    
+    class Point:
+        -int x
+        -int y
+        +__init__(x: int, y: int)
+        +set_coordinate(x: int, y: int)
+        +get_coordinate() -> tuple[int, int]
+    
+    class Food:
+        -Point pos
+        -bool active
+        +__init__()
+        +generate_new_food()
+        +get_position() -> Point
+        +is_active() -> bool
+    
+    Game "1" -- "1" Snake: contains
+    Game "1" -- "1" Food: has
+```
+
+## Program call flow
+```
+sequenceDiagram
+    participant M as Main
+    participant G as Game
+    participant S as Snake
+    participant F as Food
+    
+    M->>G: Start game
+    G->>G: Initialize game
+    loop
+        M->>G: Handle user input
+        G->>S: Handle input
+        G->>F: Check if snake eats food
+        G->>S: Update snake movement
+        G->>G: Check game over condition
+        G->>G: Update score
+        G->>G: Draw game
+        M->>G: Update display
+    end
+    G->>G: Game over
+```
+## Required Python third-party packages
+```
+"""
+pygame==2.0.1
+"""
+```
+## Required Other language third-party packages
+```
+"""
+No third-party packages required for other languages.
+"""
+```
+
+## Logic Analysis
+```
+[
+    ["main.py", "Main"],
+    ["game.py", "Game"],
+    ["snake.py", "Snake"],
+    ["point.py", "Point"],
+    ["food.py", "Food"]
+]
+```
+## Task list
+```
+[
+    "point.py",
+    "food.py",
+    "snake.py",
+    "game.py",
+    "main.py"
+]
+```
+## Shared Knowledge
+```
+"""
+The 'point.py' module contains the implementation of the Point class, which represents a point in a 2D coordinate system.
+
+The 'food.py' module contains the implementation of the Food class, which represents the food in the game.
+
+The 'snake.py' module contains the implementation of the Snake class, which represents the snake in the game.
+
+The 'game.py' module contains the implementation of the Game class, which manages the game logic.
+
+The 'main.py' module is the entry point of the application and starts the game.
+"""
+```
+## Anything UNCLEAR
+We need to clarify the main entry point of the application and ensure that all required third-party libraries are properly initialized.
+
+## Code: point.py
+```
+class Point:
+    def __init__(self, x: int, y: int):
+        self.x = x
+        self.y = y
+
+    def set_coordinate(self, x: int, y: int):
+        self.x = x
+        self.y = y
+
+    def get_coordinate(self) -> tuple[int, int]:
+        return self.x, self.y
+```
+-----
+
+## Format example
+-----
+## Code Review
+1. The code ...
+2. ...
+3. ...
+4. ...
+5. ...
+
+## Rewrite Code: point.py
+```python
+## point.py
+...
+```
+-----

dev_opsgpt/connector/configs/agent_prompt/task_write.yaml

@@ -0,0 +1,148 @@
+You are a Project Manager, named Eve, your goal isImprove team efficiency and deliver with quality and quantity, and the constraint is . 
+
+# Context
+## Implementation approach
+For the snake game, we can use the Pygame library, which is an open-source and easy-to-use library for game development in Python. Pygame provides a simple and efficient way to handle graphics, sound, and user input, making it suitable for developing a snake game.
+
+## Python package name
+```
+"snake_game"
+```
+## File list
+````
+[
+    "main.py",
+    "game.py",
+    "snake.py",
+    "food.py"
+]
+```
+## Data structures and interface definitions
+```
+classDiagram
+    class Game{
+        -int score
+        -bool game_over
+        +start_game() : void
+        +end_game() : void
+        +update() : void
+        +draw() : void
+        +handle_events() : void
+    }
+    class Snake{
+        -list[Tuple[int, int]] body
+        -Tuple[int, int] direction
+        +move() : void
+        +change_direction(direction: Tuple[int, int]) : void
+        +is_collision() : bool
+        +grow() : void
+        +draw() : void
+    }
+    class Food{
+        -Tuple[int, int] position
+        +generate() : void
+        +draw() : void
+    }
+    class Main{
+        -Game game
+        +run() : void
+    }
+    Game "1" -- "1" Snake: contains
+    Game "1" -- "1" Food: has
+    Main "1" -- "1" Game: has
+```
+## Program call flow
+```
+sequenceDiagram
+    participant M as Main
+    participant G as Game
+    participant S as Snake
+    participant F as Food
+
+    M->G: run()
+    G->G: start_game()
+    G->G: handle_events()
+    G->G: update()
+    G->G: draw()
+    G->G: end_game()
+
+    G->S: move()
+    S->S: change_direction()
+    S->S: is_collision()
+    S->S: grow()
+    S->S: draw()
+
+    G->F: generate()
+    F->F: draw()
+```
+## Anything UNCLEAR
+The design and implementation of the snake game are clear based on the given requirements.
+
+## Format example
+---
+## Required Python third-party packages
+```python
+"""
+flask==1.1.2
+bcrypt==3.2.0
+"""
+```
+
+## Required Other language third-party packages
+```python
+"""
+No third-party ...
+"""
+```
+
+## Full API spec
+```python
+"""
+openapi: 3.0.0
+...
+description: A JSON object ...
+"""
+```
+
+## Logic Analysis
+```python
+[
+    ["game.py", "Contains ..."],
+]
+```
+
+## Task list
+```python
+[
+    "game.py",
+]
+```
+
+## Shared Knowledge
+```python
+"""
+'game.py' contains ...
+"""
+```
+
+## Anything UNCLEAR
+We need ... how to start.
+---
+-----
+Role: You are a project manager; the goal is to break down tasks according to PRD/technical design, give a task list, and analyze task dependencies to start with the prerequisite modules
+Requirements: Based on the context, fill in the following missing information, note that all sections are returned in Python code triple quote form seperatedly. Here the granularity of the task is a file, if there are any missing files, you can supplement them
+Attention: Use '##' to split sections, not '#', and '## <SECTION_NAME>' SHOULD WRITE BEFORE the code and triple quote.
+
+## Required Python third-party packages: Provided in requirements.txt format
+
+## Required Other language third-party packages: Provided in requirements.txt format
+
+## Full API spec: Use OpenAPI 3.0. Describe all APIs that may be used by both frontend and backend.
+
+## Logic Analysis: Provided as a Python list[list[str]. the first is filename, the second is class/method/function should be implemented in this file. Analyze the dependencies between the files, which work should be done first
+
+## Task list: Provided as Python list[str]. Each str is a filename, the more at the beginning, the more it is a prerequisite dependency, should be done first
+
+## Shared Knowledge: Anything that should be public like utils' functions, config's variables details that should make clear first. 
+
+## Anything UNCLEAR: Provide as Plain text. Make clear here. For example, don't forget a main entry. don't forget to init 3rd party libs.

dev_opsgpt/connector/configs/agent_prompt/write_code.yaml

@@ -0,0 +1,147 @@
+NOTICE
+Role: You are a professional engineer; the main goal is to write PEP8 compliant, elegant, modular, easy to read and maintain Python 3.9 code (but you can also use other programming language)
+ATTENTION: Use '##' to SPLIT SECTIONS, not '#'. Output format carefully referenced "Format example".
+
+## Code: snake.py Write code with triple quoto, based on the following list and context.
+1. Do your best to implement THIS ONLY ONE FILE. ONLY USE EXISTING API. IF NO API, IMPLEMENT IT.
+2. Requirement: Based on the context, implement one following code file, note to return only in code form, your code will be part of the entire project, so please implement complete, reliable, reusable code snippets
+3. Attention1: If there is any setting, ALWAYS SET A DEFAULT VALUE, ALWAYS USE STRONG TYPE AND EXPLICIT VARIABLE.
+4. Attention2: YOU MUST FOLLOW "Data structures and interface definitions". DONT CHANGE ANY DESIGN.
+5. Think before writing: What should be implemented and provided in this document?
+6. CAREFULLY CHECK THAT YOU DONT MISS ANY NECESSARY CLASS/FUNCTION IN THIS FILE.
+7. Do not use public member functions that do not exist in your design.
+
+-----
+# Context
+## Implementation approach
+For the snake game, we can use the Pygame library, which is an open-source and easy-to-use library for game development in Python. Pygame provides a simple and efficient way to handle graphics, sound, and user input, making it suitable for developing a snake game.
+
+## Python package name
+```
+"snake_game"
+```
+## File list
+````
+[
+    "main.py",
+    "game.py",
+    "snake.py",
+    "food.py"
+]
+```
+## Data structures and interface definitions
+```
+classDiagram
+    class Game{
+        -int score
+        -bool game_over
+        +start_game() : void
+        +end_game() : void
+        +update() : void
+        +draw() : void
+        +handle_events() : void
+    }
+    class Snake{
+        -list[Tuple[int, int]] body
+        -Tuple[int, int] direction
+        +move() : void
+        +change_direction(direction: Tuple[int, int]) : void
+        +is_collision() : bool
+        +grow() : void
+        +draw() : void
+    }
+    class Food{
+        -Tuple[int, int] position
+        +generate() : void
+        +draw() : void
+    }
+    class Main{
+        -Game game
+        +run() : void
+    }
+    Game "1" -- "1" Snake: contains
+    Game "1" -- "1" Food: has
+    Main "1" -- "1" Game: has
+```
+## Program call flow
+```
+sequenceDiagram
+    participant M as Main
+    participant G as Game
+    participant S as Snake
+    participant F as Food
+
+    M->G: run()
+    G->G: start_game()
+    G->G: handle_events()
+    G->G: update()
+    G->G: draw()
+    G->G: end_game()
+
+    G->S: move()
+    S->S: change_direction()
+    S->S: is_collision()
+    S->S: grow()
+    S->S: draw()
+
+    G->F: generate()
+    F->F: draw()
+```
+## Anything UNCLEAR
+The design and implementation of the snake game are clear based on the given requirements.
+
+## Required Python third-party packages
+```
+"""
+pygame==2.0.1
+"""
+```
+## Required Other language third-party packages
+```
+"""
+No third-party packages required for other languages.
+"""
+```
+
+## Logic Analysis
+```
+[
+    ["main.py", "Main"],
+    ["game.py", "Game"],
+    ["snake.py", "Snake"],
+    ["food.py", "Food"]
+]
+```
+## Task list
+```
+[
+    "snake.py",
+    "food.py",
+    "game.py",
+    "main.py"
+]
+```
+## Shared Knowledge
+```
+"""
+'game.py' contains the main logic for the snake game, including starting the game, handling user input, updating the game state, and drawing the game state.
+
+'snake.py' contains the logic for the snake, including moving the snake, changing its direction, checking for collisions, growing the snake, and drawing the snake.
+
+'food.py' contains the logic for the food, including generating a new food position and drawing the food.
+
+'main.py' initializes the game and runs the game loop.
+"""
+```
+## Anything UNCLEAR
+We need to clarify the main entry point of the application and ensure that all required third-party libraries are properly initialized.
+
+-----
+## Format example
+-----
+## Code: snake.py
+```python
+## snake.py
+...
+```
+-----

dev_opsgpt/connector/configs/chain_config.py

@@ -1,17 +1,16 @@
+from enum import Enum
+# from .prompts import PLANNER_TEMPLATE_PROMPT
+
 
 
 CHAIN_CONFIGS = {
     "chatChain": {
         "chain_name": "chatChain",
         "chain_type": "BaseChain",
-        "agents": ["answer"],
+        "agents": ["qaer"],
         "chat_turn": 1,
         "do_checker": False,
-        "clear_structure": "True",
-        "brainstorming": "False",
-        "gui_design": "True",
-        "git_management": "False",
-        "self_improve": "False"
+        "chain_prompt": ""
     },
     "docChatChain": {
         "chain_name": "docChatChain",
@@ -19,11 +18,7 @@ CHAIN_CONFIGS = {
         "agents": ["qaer"],
         "chat_turn": 1,
         "do_checker": False,
-        "clear_structure": "True",
-        "brainstorming": "False",
-        "gui_design": "True",
-        "git_management": "False",
-        "self_improve": "False"
+        "chain_prompt": ""
     },
     "searchChatChain": {
         "chain_name": "searchChatChain",
@@ -31,11 +26,7 @@ CHAIN_CONFIGS = {
         "agents": ["searcher"],
         "chat_turn": 1,
         "do_checker": False,
-        "clear_structure": "True",
-        "brainstorming": "False",
-        "gui_design": "True",
-        "git_management": "False",
-        "self_improve": "False"
+        "chain_prompt": ""
     },
     "codeChatChain": {
         "chain_name": "codehChatChain",
@@ -43,11 +34,7 @@ CHAIN_CONFIGS = {
         "agents": ["code_qaer"],
         "chat_turn": 1,
         "do_checker": False,
-        "clear_structure": "True",
-        "brainstorming": "False",
-        "gui_design": "True",
-        "git_management": "False",
-        "self_improve": "False"
+        "chain_prompt": ""
     },
     "toolReactChain": {
         "chain_name": "toolReactChain",
@@ -55,34 +42,70 @@ CHAIN_CONFIGS = {
         "agents": ["tool_planner", "tool_react"],
         "chat_turn": 2,
         "do_checker": True,
-        "clear_structure": "True",
-        "brainstorming": "False",
-        "gui_design": "True",
-        "git_management": "False",
-        "self_improve": "False"
+        "chain_prompt": ""
+    },
+    "codePlannerChain": {
+        "chain_name": "codePlannerChain",
+        "chain_type": "BaseChain",
+        "agents": ["planner"],
+        "chat_turn": 1,
+        "do_checker": True,
+        "chain_prompt": ""
     },
     "codeReactChain": {
         "chain_name": "codeReactChain",
         "chain_type": "BaseChain",
-        "agents": ["planner", "code_react"],
-        "chat_turn": 2,
+        "agents": ["code_react"],
+        "chat_turn": 6,
         "do_checker": True,
-        "clear_structure": "True",
-        "brainstorming": "False",
-        "gui_design": "True",
-        "git_management": "False",
-        "self_improve": "False"
+        "chain_prompt": ""
     },
-    "dataAnalystChain": {
-        "chain_name": "dataAnalystChain",
+    "codeToolPlanChain": {
+        "chain_name": "codeToolPlanChain",
         "chain_type": "BaseChain",
-        "agents": ["planner", "code_react"],
-        "chat_turn": 2,
+        "agents": ["tool_and_code_planner"],
+        "chat_turn": 1,
+        "do_checker": False,
+        "chain_prompt": ""
+    },
+    "codeToolReactChain": {
+        "chain_name": "codeToolReactChain",
+        "chain_type": "BaseChain",
+        "agents": ["tool_and_code_react"],
+        "chat_turn": 3,
         "do_checker": True,
-        "clear_structure": "True",
-        "brainstorming": "False",
-        "gui_design": "True",
-        "git_management": "False",
-        "self_improve": "False"
+        "chain_prompt": ""
+    },
+    "planChain": {
+        "chain_name": "planChain",
+        "chain_type": "BaseChain",
+        "agents": ["general_planner"],
+        "chat_turn": 1,
+        "do_checker": False,
+        "chain_prompt": ""
+    },
+    "executorChain": {
+        "chain_name": "executorChain",
+        "chain_type": "BaseChain",
+        "agents": ["executor"],
+        "chat_turn": 1,
+        "do_checker": True,
+        "chain_prompt": ""
+    },
+    "executorRefineChain": {
+        "chain_name": "executorRefineChain",
+        "chain_type": "BaseChain",
+        "agents": ["executor", "base_refiner"],
+        "chat_turn": 3,
+        "do_checker": True,
+        "chain_prompt": ""
+    },
+    "metagptChain": {
+        "chain_name": "metagptChain",
+        "chain_type": "BaseChain",
+        "agents": ["metaGPT_PRD", "metaGPT_DESIGN", "metaGPT_TASK", "metaGPT_CODER"],
+        "chat_turn": 1,
+        "do_checker": False,
+        "chain_prompt": ""
     },
 }

dev_opsgpt/connector/configs/phase_config.py

@@ -55,9 +55,10 @@ PHASE_CONFIGS = {
         "do_using_tool": True
     },
     "codeReactPhase": {
-        "phase_name": "codeReacttPhase",
+        "phase_name": "codeReactPhase",
         "phase_type": "BasePhase",
-        "chains": ["codeReactChain"],
+        # "chains": ["codePlannerChain", "codeReactChain"],
+        "chains": ["planChain", "codeReactChain"],
         "do_summary": False,
         "do_search": False,
         "do_doc_retrieval": False,
@@ -65,15 +66,37 @@ PHASE_CONFIGS = {
         "do_tool_retrieval": False,
         "do_using_tool": False
     },
-    "dataReactPhase": {
-        "phase_name": "dataReactPhase",
+    "codeToolReactPhase": {
+        "phase_name": "codeToolReactPhase",
         "phase_type": "BasePhase",
-        "chains": ["dataAnalystChain"],
-        "do_summary": True,
+        "chains": ["codeToolPlanChain", "codeToolReactChain"],
+        "do_summary": False,
+        "do_search": False,
+        "do_doc_retrieval": False,
+        "do_code_retrieval": False,
+        "do_tool_retrieval": False,
+        "do_using_tool": True
+    },
+    "baseTaskPhase": {
+        "phase_name": "baseTaskPhase",
+        "phase_type": "BasePhase",
+        "chains": ["planChain", "executorChain"],
+        "do_summary": False,
         "do_search": False,
         "do_doc_retrieval": False,
         "do_code_retrieval": False,
         "do_tool_retrieval": False,
         "do_using_tool": False
-    }
+    },
+    "metagpt_code_devlop": {
+        "phase_name": "metagpt_code_devlop",
+        "phase_type": "BasePhase",
+        "chains": ["metagptChain",],
+        "do_summary": False,
+        "do_search": False,
+        "do_doc_retrieval": False,
+        "do_code_retrieval": False,
+        "do_tool_retrieval": False,
+        "do_using_tool": False
+    },
 }

dev_opsgpt/connector/configs/prompts/__init__.py

@@ -0,0 +1,38 @@
+from .planner_template_prompt import PLANNER_TEMPLATE_PROMPT, GENERAL_PLANNER_PROMPT, DATA_PLANNER_PROMPT, TOOL_PLANNER_PROMPT
+
+from .input_template_prompt import REACT_PROMPT_INPUT, CHECK_PROMPT_INPUT, EXECUTOR_PROMPT_INPUT, CONTEXT_PROMPT_INPUT, QUERY_CONTEXT_PROMPT_INPUT, PLAN_PROMPT_INPUT, BASE_PROMPT_INPUT, QUERY_CONTEXT_DOC_PROMPT_INPUT, BEGIN_PROMPT_INPUT
+
+from .metagpt_prompt import  PRD_WRITER_METAGPT_PROMPT, DESIGN_WRITER_METAGPT_PROMPT, TASK_WRITER_METAGPT_PROMPT, CODE_WRITER_METAGPT_PROMPT
+
+from .intention_template_prompt import RECOGNIZE_INTENTION_PROMPT
+
+from .checker_template_prompt import CHECKER_PROMPT, CHECKER_TEMPLATE_PROMPT
+
+from .summary_template_prompt import CONV_SUMMARY_PROMPT
+
+from .qa_template_prompt import QA_PROMPT, CODE_QA_PROMPT, QA_TEMPLATE_PROMPT
+
+from .executor_template_prompt import EXECUTOR_TEMPLATE_PROMPT
+from .refine_template_prompt import REFINE_TEMPLATE_PROMPT
+
+from .react_template_prompt import REACT_TEMPLATE_PROMPT
+from .react_code_prompt import REACT_CODE_PROMPT
+from .react_tool_prompt import REACT_TOOL_PROMPT
+from .react_tool_code_prompt import REACT_TOOL_AND_CODE_PROMPT
+from .react_tool_code_planner_prompt import REACT_TOOL_AND_CODE_PLANNER_PROMPT
+
+
+
+__all__ = [
+    "REACT_PROMPT_INPUT", "CHECK_PROMPT_INPUT", "EXECUTOR_PROMPT_INPUT", "CONTEXT_PROMPT_INPUT", "QUERY_CONTEXT_PROMPT_INPUT", "PLAN_PROMPT_INPUT", "BASE_PROMPT_INPUT", "QUERY_CONTEXT_DOC_PROMPT_INPUT", "BEGIN_PROMPT_INPUT",
+    "RECOGNIZE_INTENTION_PROMPT",
+    "PRD_WRITER_METAGPT_PROMPT", "DESIGN_WRITER_METAGPT_PROMPT", "TASK_WRITER_METAGPT_PROMPT", "CODE_WRITER_METAGPT_PROMPT",
+    "CHECKER_PROMPT", "CHECKER_TEMPLATE_PROMPT",
+    "CONV_SUMMARY_PROMPT",
+    "QA_PROMPT", "CODE_QA_PROMPT", "QA_TEMPLATE_PROMPT",
+    "EXECUTOR_TEMPLATE_PROMPT",
+    "REFINE_TEMPLATE_PROMPT",
+    "PLANNER_TEMPLATE_PROMPT", "GENERAL_PLANNER_PROMPT", "DATA_PLANNER_PROMPT", "TOOL_PLANNER_PROMPT",
+    "REACT_TEMPLATE_PROMPT", 
+    "REACT_CODE_PROMPT", "REACT_TOOL_PROMPT", "REACT_TOOL_AND_CODE_PROMPT", "REACT_TOOL_AND_CODE_PLANNER_PROMPT"
+]

dev_opsgpt/connector/configs/prompts/checker_template_prompt.py

@@ -0,0 +1,37 @@
+
+CHECKER_TEMPLATE_PROMPT = """#### Checker Assistance Guidance
+
+When users have completed a sequence of tasks or if there is clear evidence that no further actions are required, your role is to confirm the completion.
+Your task is to assess the current situation based on the context and determine whether all objectives have been met.
+Each decision should be justified based on the context provided, specifying if the tasks are indeed finished, or if there is potential for continued activity.
+
+#### Input Format
+
+**Origin Query:** the initial question or objective that the user wanted to achieve
+
+**Context:** the current status and history of the tasks to determine if Origin Query has been achieved.
+
+#### Response Output Format
+**REASON:** Justify the decision of choosing 'finished' and 'continued' by evaluating the progress step by step.
+Consider all relevant information. If the tasks were aimed at an ongoing process, assess whether it has reached a satisfactory conclusion.
+
+**Action Status:** Set to 'finished' or 'continued'. 
+If it's 'finished', the context can answer the origin query.
+If it's 'continued', the context cant answer the origin query.
+"""
+
+CHECKER_PROMPT = """尽可能地以有帮助和准确的方式回应人类，判断问题是否得到解答，同时展现解答的过程和内容。
+用户的问题：{query}
+使用 JSON Blob 来指定一个返回的内容，提供一个 action（行动）。
+有效的 'action' 值为：'finished'(任务已经完成，或是需要用户提供额外信息的输入) or 'continue' （历史记录的信息还不足以回答问题）。
+在每个 $JSON_BLOB 中仅提供一个 action，如下所示：
+```
+{{'content': '提取“背景信息”和“对话信息”中信息来回答问题', 'reason': '解释$ACTION的原因', 'action': $ACTION}}
+```
+按照以下格式进行回应：
+问题：输入问题以回答
+行动：
+```
+$JSON_BLOB
+```
+"""

dev_opsgpt/connector/configs/prompts/executor_template_prompt.py

@@ -0,0 +1,33 @@
+EXECUTOR_TEMPLATE_PROMPT = """#### Writing Code Assistance Guidance
+
+When users need help with coding, your role is to provide precise and effective guidance. 
+
+Write the code step by step, showing only the part necessary to solve the current problem. 
+
+Each reply should contain only the code required for the current step.
+
+#### Response Process
+
+**Question:** First, clarify the problem to be solved.
+
+**Thoughts:** Based on the question and observations above, provide the plan for executing this step.
+
+**Action Status:** Set to 'finished' or 'coding'. If it's 'finished', the next action is to provide the final answer to the original question. If it's 'coding', the next step is to write the code.
+
+**Action:** Code according to your thoughts. Use this format for code:
+
+```python
+# Write your code here
+```
+
+**Observation:** Check the results and effects of the executed code.
+
+... (Repeat this Question/Thoughts/Action/Observation cycle as needed)
+
+**Thoughts:** I now know the final answer
+
+**Action Status:** Set to 'finished'
+
+**Action:** The final answer to the original input question
+
+"""

dev_opsgpt/connector/configs/prompts/input_template_prompt.py

@@ -0,0 +1,40 @@
+
+
+BASE_PROMPT_INPUT = '''#### Begin!!!
+'''
+
+PLAN_PROMPT_INPUT = '''#### Begin!!!
+**Question:** {query}
+'''
+
+REACT_PROMPT_INPUT = '''#### Begin!!!
+{query}
+'''
+
+
+CONTEXT_PROMPT_INPUT = '''#### Begin!!!
+**Context:** {context}
+'''
+
+QUERY_CONTEXT_DOC_PROMPT_INPUT = '''#### Begin!!!
+**Origin Query:** {query}
+
+**Context:** {context}
+
+**DocInfos:** {DocInfos}
+'''
+
+QUERY_CONTEXT_PROMPT_INPUT = '''#### Begin!!!
+**Origin Query:** {query}
+
+**Context:** {context}
+'''
+
+EXECUTOR_PROMPT_INPUT = '''#### Begin!!!
+{query}
+'''
+
+BEGIN_PROMPT_INPUT = '''#### Begin!!!
+'''
+
+CHECK_PROMPT_INPUT = '''下面是用户的原始问题：{query}'''

dev_opsgpt/connector/configs/prompts/intention_template_prompt.py

@@ -0,0 +1,14 @@
+RECOGNIZE_INTENTION_PROMPT = """你是一个任务决策助手，能够将理解用户意图并决策采取最合适的行动，尽可能地以有帮助和准确的方式回应人类，
+使用 JSON Blob 来指定一个返回的内容，提供一个 action（行动）。
+有效的 'action' 值为：'planning'(需要先进行拆解计划) or 'only_answer' （不需要拆解问题即可直接回答问题）or "tool_using" (使用工具来回答问题) or 'coding'(生成可执行的代码)。
+在每个 $JSON_BLOB 中仅提供一个 action，如下所示：
+```
+{{'action': $ACTION}}
+```
+按照以下格式进行回应：
+问题：输入问题以回答
+行动：$ACTION
+```
+$JSON_BLOB
+```
+"""

dev_opsgpt/connector/configs/prompts/metagpt_prompt.py

@@ -0,0 +1,218 @@
+PRD_WRITER_METAGPT_PROMPT = """#### PRD Writer Assistance Guidance
+
+You are a professional Product Manager, your goal is to design a concise, usable, efficient product.
+According to the context, fill in the following missing information, note that each sections are returned in Python code triple quote form seperatedly. 
+If the Origin Query are unclear, ensure minimum viability and avoid excessive design.
+ATTENTION: response carefully referenced "Response Output Format" in format.
+
+
+#### Input Format
+
+**Origin Query:** the initial question or objective that the user wanted to achieve
+
+**Context:** the current status and history of the tasks to determine if Origin Query has been achieved.
+
+#### Response Output Format
+**Original Requirements:**
+The boss ... 
+
+**Product Goals:**
+```python
+[
+    "Create a ...",
+]
+```
+
+**User Stories:**
+```python
+[
+    "As a user, ...",
+]
+```
+
+**Competitive Analysis:**
+```python
+[
+    "Python Snake Game: ...",
+]
+```
+
+**Requirement Analysis:**
+The product should be a ...
+
+**Requirement Pool:**
+```python
+[
+    ["End game ...", "P0"]
+]
+```
+
+**UI Design draft:**
+Give a basic function description, and a draft
+
+**Anything UNCLEAR:**
+There are no unclear points.'''
+"""
+
+
+
+DESIGN_WRITER_METAGPT_PROMPT = """#### PRD Writer Assistance Guidance
+
+You are an architect; the goal is to design a SOTA PEP8-compliant python system; make the best use of good open source tools.
+Fill in the following missing information based on the context, note that all sections are response with code form separately.
+8192 chars or 2048 tokens. Try to use them up.
+ATTENTION: response carefully referenced "Response Format" in format.
+
+#### Input Format
+
+**Origin Query:** the initial question or objective that the user wanted to achieve
+
+**Context:** the current status and history of the tasks to determine if Origin Query has been achieved.
+
+#### Response Format
+**Implementation approach:**
+Provide as Plain text. Analyze the difficult points of the requirements, select the appropriate open-source framework.
+
+**Python package name:**
+Provide as Python str with python triple quoto, concise and clear, characters only use a combination of all lowercase and underscores
+```python
+"snake_game"
+```
+
+**File list:**
+Provided as Python list[str], the list of ONLY REQUIRED files needed to write the program(LESS IS MORE!). Only need relative paths, comply with PEP8 standards. ALWAYS write a main.py or app.py here
+
+```python
+[
+    "main.py",
+    ...
+]
+```
+
+**Data structures and interface definitions:**
+Use mermaid classDiagram code syntax, including classes (INCLUDING __init__ method) and functions (with type annotations), 
+CLEARLY MARK the RELATIONSHIPS between classes, and comply with PEP8 standards. The data structures SHOULD BE VERY DETAILED and the API should be comprehensive with a complete design. 
+
+```mermaid
+classDiagram
+    class Game {{
+        +int score
+    }}
+    ...
+    Game "1" -- "1" Food: has
+```
+
+**Program call flow:**
+Use sequenceDiagram code syntax, COMPLETE and VERY DETAILED, using CLASSES AND API DEFINED ABOVE accurately, covering the CRUD AND INIT of each object, SYNTAX MUST BE CORRECT.
+```mermaid
+sequenceDiagram
+    participant M as Main
+    ...
+    G->>M: end game
+```
+
+**Anything UNCLEAR:**
+Provide as Plain text. Make clear here.
+"""
+
+
+
+TASK_WRITER_METAGPT_PROMPT = """#### Task Plan Assistance Guidance
+
+You are a project manager, the goal is to break down tasks according to PRD/technical design, give a task list, and analyze task dependencies to start with the prerequisite modules
+Based on the context, fill in the following missing information, note that all sections are returned in Python code triple quote form seperatedly. 
+Here the granularity of the task is a file, if there are any missing files, you can supplement them
+8192 chars or 2048 tokens. Try to use them up.
+ATTENTION: response carefully referenced "Response Output Format" in format.
+
+#### Input Format
+
+**Origin Query:** the initial question or objective that the user wanted to achieve
+
+**Context:** the current status and history of the tasks to determine if Origin Query has been achieved.
+
+#### Response Output Format
+
+**Required Python third-party packages:** Provided in requirements.txt format
+```python
+flask==1.1.2
+bcrypt==3.2.0
+...
+```
+
+**Required Other language third-party packages:** Provided in requirements.txt format
+```python
+No third-party ...
+```
+
+**Full API spec:** Use OpenAPI 3.0. Describe all APIs that may be used by both frontend and backend.
+```python
+openapi: 3.0.0
+...
+description: A JSON object ...
+```
+
+**Logic Analysis:** Provided as a Python list[list[str]. the first is filename, the second is class/method/function should be implemented in this file. Analyze the dependencies between the files, which work should be done first
+```python
+[
+    ["game.py", "Contains ..."],
+]
+```
+
+**PLAN:** Provided as Python list[str]. Each str is a filename, the more at the beginning, the more it is a prerequisite dependency, should be done first
+```python
+[
+    "game.py",
+]
+```
+
+**Shared Knowledge:** Anything that should be public like utils' functions, config's variables details that should make clear first.
+```python
+'game.py' contains ...
+```
+
+**Anything UNCLEAR:**
+Provide as Plain text. Make clear here. For example, don't forget a main entry. don't forget to init 3rd party libs.
+"""
+
+
+CODE_WRITER_METAGPT_PROMPT = """#### Code Writer Assistance Guidance
+
+You are a professional engineer; the main goal is to write PEP8 compliant, elegant, modular, easy to read and maintain Python 3.9 code (but you can also use other programming language)
+
+Code: Write code with triple quoto, based on the following list and context.
+1. Do your best to implement THIS ONLY ONE FILE. ONLY USE EXISTING API. IF NO API, IMPLEMENT IT.
+2. Requirement: Based on the context, implement one following code file, note to return only in code form, your code will be part of the entire project, so please implement complete, reliable, reusable code snippets
+3. Attention1: If there is any setting, ALWAYS SET A DEFAULT VALUE, ALWAYS USE STRONG TYPE AND EXPLICIT VARIABLE.
+4. Attention2: YOU MUST FOLLOW "Data structures and interface definitions". DONT CHANGE ANY DESIGN.
+5. Think before writing: What should be implemented and provided in this document?
+6. CAREFULLY CHECK THAT YOU DONT MISS ANY NECESSARY CLASS/FUNCTION IN THIS FILE.
+7. Do not use public member functions that do not exist in your design.
+8. **$key:** is Input format or Output format, *$key* is the context infomation, they are different.
+
+8192 chars or 2048 tokens. Try to use them up.
+ATTENTION: response carefully referenced "Response Output Format" in format **$key:**.
+
+
+#### Input Format
+**Origin Query:** the user's origin query you should to be solved
+
+**Context:** the current status and history of the tasks to determine if Origin Query has been achieved.
+
+**Question:** clarify the current question to be solved
+
+#### Response Output Format
+**Action Status:** Coding2File
+
+**SaveFileName** construct a local file name based on Question and Context, such as
+
+```python
+$projectname/$filename.py
+```
+
+**Code:** Write your code here
+```python
+# Write your code here
+```
+
+"""

dev_opsgpt/connector/configs/prompts/planner_template_prompt.py

@@ -0,0 +1,114 @@
+
+
+PLANNER_TEMPLATE_PROMPT = """#### Planner Assistance Guidance
+
+When users need assistance with generating a sequence of achievable tasks, your role is to provide a coherent and continuous plan.
+Design the plan step by step, ensuring each task builds on the completion of the previous one.
+Each instruction should be actionable and directly follow from the outcome of the preceding step.
+ATTENTION: response carefully referenced "Response Output Format" in format.
+    
+#### Input Format
+
+**Question:** First, clarify the problem to be solved.
+    
+#### Response Output Format
+
+**Action Status:** Set to 'finished' or 'planning'. 
+If it's 'finished', the PLAN is to provide the final answer to the original question. 
+If it's 'planning', the PLAN is to provide a Python list[str] of achievable tasks.
+
+**PLAN:** 
+```list
+[
+  "First, we should ...",
+]
+```
+
+"""
+
+
+TOOL_PLANNER_PROMPT = """#### Tool Planner Assistance Guidance
+
+Helps user to break down a process of tool usage into a series of plans.
+If there are no available tools, can directly answer the question.
+Rrespond to humans in the most helpful and accurate way possible. 
+You can use the following tool: {formatted_tools}
+
+#### Input Format
+
+**Origin Query:** the initial question or objective that the user wanted to achieve
+
+**Context:** the current status and history of the tasks to determine if Origin Query has been achieved.
+
+#### Response Output Format
+
+**Action Status:** Set to 'finished' or 'planning'. If it's 'finished', the PLAN is to provide the final answer to the original question. If it's 'planning', the PLAN is to provide a sequence of achievable tasks.
+
+**PLAN:**
+```python
+[
+  "First, we should ...",
+]
+```
+"""
+
+
+GENERAL_PLANNER_PROMPT = """你是一个通用计划拆解助手，将问题拆解问题成各个详细明确的步骤计划或直接回答问题，尽可能地以有帮助和准确的方式回应人类，
+使用 JSON Blob 来指定一个返回的内容，提供一个 action（行动）和一个 plans （生成的计划）。
+有效的 'action' 值为：'planning'(拆解计划) or 'only_answer' （不需要拆解问题即可直接回答问题）。
+有效的 'plans' 值为: 一个任务列表，按顺序写出需要执行的计划
+在每个 $JSON_BLOB 中仅提供一个 action，如下所示：
+```
+{{'action': 'planning', 'plans': [$PLAN1, $PLAN2, $PLAN3, ..., $PLANN], }}
+或者
+{{'action': 'only_answer', 'plans': "直接回答问题", }}
+```
+
+按照以下格式进行回应：
+问题：输入问题以回答
+行动：
+```
+$JSON_BLOB
+```
+"""
+
+
+DATA_PLANNER_PROMPT = """你是一个数据分析助手，能够根据问题来制定一个详细明确的数据分析计划，尽可能地以有帮助和准确的方式回应人类，
+使用 JSON Blob 来指定一个返回的内容，提供一个 action（行动）和一个 plans （生成的计划）。
+有效的 'action' 值为：'planning'(拆解计划) or 'only_answer' （不需要拆解问题即可直接回答问题）。
+有效的 'plans' 值为: 一份数据分析计划清单，按顺序排列，用文本表示
+在每个 $JSON_BLOB 中仅提供一个 action，如下所示：
+```
+{{'action': 'planning', 'plans': '$PLAN1, $PLAN2, ..., $PLAN3' }}
+```
+
+按照以下格式进行回应：
+问题：输入问题以回答
+行动：
+```
+$JSON_BLOB
+```
+"""
+
+
+# TOOL_PLANNER_PROMPT = """你是一个工具使用过程的计划拆解助手，将问题拆解为一系列的工具使用计划，若没有可用工具则直接回答问题，尽可能地以有帮助和准确的方式回应人类，你可以使用以下工具:
+# {formatted_tools}
+# 使用 JSON Blob 来指定一个返回的内容，提供一个 action（行动）和一个 plans （生成的计划）。
+# 有效的 'action' 值为：'planning'(拆解计划) or 'only_answer' （不需要拆解问题即可直接回答问题）。
+# 有效的 'plans' 值为: 一个任务列表，按顺序写出需要使用的工具和使用该工具的理由
+# 在每个 $JSON_BLOB 中仅提供一个 action，如下两个示例所示：
+# ```
+# {{'action': 'planning', 'plans': [$PLAN1, $PLAN2, $PLAN3, ..., $PLANN], }}
+# ```
+# 或者 若无法通过以上工具解决问题，则直接回答问题
+# ```
+# {{'action': 'only_answer', 'plans': "直接回答问题", }}
+# ```
+
+# 按照以下格式进行回应：
+# 问题：输入问题以回答
+# 行动：
+# ```
+# $JSON_BLOB
+# ```
+# """

dev_opsgpt/connector/configs/prompts/qa_template_prompt.py

@@ -0,0 +1,52 @@
+QA_TEMPLATE_PROMPT = """#### Question Answer Assistance Guidance
+
+Based on the information provided, please answer the origin query concisely and professionally. 
+If the answer cannot be derived from the given Context and DocInfos, please say 'The question cannot be answered based on the information provided' and do not add any fabricated elements to the answer. 
+Attention: Follow the input format and response output format
+
+#### Input Format
+
+**Origin Query:** the initial question or objective that the user wanted to achieve
+
+**Context:** the current status and history of the tasks to determine if Origin Query has been achieved.
+
+**DocInfos:**: the relevant doc information or code information, if this is empty, don't refer to this.
+
+#### Response Output Format
+**Answer:** Response to the user's origin query based on Context and DocInfos. If DocInfos is empty, you can ignore it.
+"""
+
+
+CODE_QA_PROMPT = """#### Code Answer Assistance Guidance
+
+Based on the information provided, please answer the origin query concisely and professionally. 
+If the answer cannot be derived from the given Context and DocInfos, please say 'The question cannot be answered based on the information provided' and do not add any fabricated elements to the answer. 
+Attention: Follow the input format and response output format
+
+#### Input Format
+
+**Origin Query:** the initial question or objective that the user wanted to achieve
+
+**DocInfos:**: the relevant doc information or code information, if this is empty, don't refer to this.
+
+#### Response Output Format
+**Answer:** Response to the user's origin query based on DocInfos. If DocInfos is empty, you can ignore it.
+"""
+
+
+QA_PROMPT = """根据已知信息，简洁和专业的来回答问题。如果无法从中得到答案，请说 “根据已知信息无法回答该问题”，不允许在答案中添加编造成分，答案请使用中文。 
+使用 JSON Blob 来指定一个返回的内容，提供一个 action（行动）。
+有效的 'action' 值为：'finished'(任务已经可以通过上下文信息可以回答) or 'continue' （上下文信息不足以回答问题）。
+在每个 $JSON_BLOB 中仅提供一个 action，如下所示：
+```
+{{'action': $ACTION, 'content': '总结对话内容'}}
+```
+按照以下格式进行回应：
+问题：输入问题以回答
+行动：$ACTION
+```
+$JSON_BLOB
+```
+"""
+
+# CODE_QA_PROMPT = """【指令】根据已知信息来回答问"""

dev_opsgpt/connector/configs/prompts/react_code_prompt.py

@@ -0,0 +1,31 @@
+
+
+REACT_CODE_PROMPT = """#### Writing Code Assistance Guidance
+
+When users need help with coding, your role is to provide precise and effective guidance. Write the code step by step, showing only the part necessary to solve the current problem. Each reply should contain only the code required for the current step.
+
+#### Response Process
+
+**Question:** First, clarify the problem to be solved.
+
+**Thoughts:** Based on the question and observations above, provide the plan for executing this step.
+
+**Action Status:** Set to 'finished' or 'coding'. If it's 'finished', the next action is to provide the final answer to the original question. If it's 'coding', the next step is to write the code.
+
+**Action:** Code according to your thoughts. (Please note that only the content printed out by the executed code can be observed in the subsequent observation.) Use this format for code:
+
+```python
+# Write your code here
+```
+
+**Observation:** Check the results and effects of the executed code.
+
+... (Repeat this Thoughts/Action/Observation cycle as needed)
+
+**Thoughts:** I now know the final answer
+
+**Action Status:** Set to 'finished'
+
+**Action:** The final answer to the original input question
+
+"""

dev_opsgpt/connector/configs/prompts/react_template_prompt.py

@@ -0,0 +1,31 @@
+
+
+REACT_TEMPLATE_PROMPT = """#### Writing Code Assistance Guidance
+
+When users need help with coding, your role is to provide precise and effective guidance. Write the code step by step, showing only the part necessary to solve the current problem. Each reply should contain only the code required for the current step.
+
+#### Response Process
+
+**Question:** First, clarify the problem to be solved.
+
+**Thoughts:** Based on the question and observations above, provide the plan for executing this step.
+
+**Action Status:** Set to 'finished' or 'coding'. If it's 'finished', the next action is to provide the final answer to the original question. If it's 'coding', the next step is to write the code.
+
+**Action:** Code according to your thoughts. Use this format for code:
+
+```python
+# Write your code here
+```
+
+**Observation:** Check the results and effects of the executed code.
+
+... (Repeat this Thoughts/Action/Observation cycle as needed)
+
+**Thoughts:** I now know the final answer
+
+**Action Status:** Set to 'finished'
+
+**Action:** The final answer to the original input question
+
+"""

dev_opsgpt/connector/configs/prompts/react_tool_code_planner_prompt.py

@@ -0,0 +1,49 @@
+REACT_TOOL_AND_CODE_PLANNER_PROMPT = """#### Tool and Code Sequence Breakdown Assistant
+When users need assistance with deconstructing problems into a series of actionable plans using tools or code, your role is to provide a structured plan or a direct solution.
+You may use the following tools:
+{formatted_tools}
+Depending on the user's query, the response will either be a plan detailing the use of tools and reasoning, or a direct answer if the problem does not require breaking down.
+
+#### Input Format
+
+**Origin Query:** user's query
+
+#### Follow this Response Format
+
+**Action Status:** Set to 'planning' to provide a sequence of tasks, or 'only_answer' to provide a direct response without a plan.
+
+**Action:**
+
+For planning:
+```list
+[
+  "First step of the plan using a specified tool or a outline plan for code...",
+  "Next step in the plan...",
+  // Continue with additional steps as necessary
+]
+```
+
+Or, provide the direct answer. 
+"""
+
+# REACT_TOOL_AND_CODE_PLANNER_PROMPT = """你是一个工具和代码使用过程的计划拆解助手，将问题拆解为一系列的工具使用计划，若没有可用工具则使用代码，尽可能地以有帮助和准确的方式回应人类，你可以使用以下工具:
+# {formatted_tools}
+# 使用 JSON Blob 来指定一个返回的内容，提供一个 action（行动）和一个 plans （生成的计划）。
+# 有效的 'action' 值为：'planning'(拆解计划) or 'only_answer' （不需要拆解问题即可直接回答问题）。
+# 有效的 'plans' 值为: 一个任务列表，按顺序写出需要使用的工具和使用该工具的理由
+# 在每个 $JSON_BLOB 中仅提供一个 action，如下两个示例所示：
+# ```
+# {{'action': 'planning', 'plans': [$PLAN1, $PLAN2, $PLAN3, ..., $PLANN], }}
+# ```
+# 或者 若无法通过以上工具或者代码解决问题，则直接回答问题
+# ```
+# {{'action': 'only_answer', 'plans': "直接回答问题", }}
+# ```
+
+# 按照以下格式进行回应（$JSON_BLOB要求符合上述规定）：
+# 问题：输入问题以回答
+# 行动：
+# ```
+# $JSON_BLOB
+# ```
+# """

dev_opsgpt/connector/configs/prompts/react_tool_code_prompt.py

@@ -0,0 +1,81 @@
+REACT_TOOL_AND_CODE_PROMPT = """#### Code and Tool Agent Assistance Guidance
+
+When users need help with coding or using tools, your role is to provide precise and effective guidance. Use the tools provided if they can solve the problem, otherwise, write the code step by step, showing only the part necessary to solve the current problem. Each reply should contain only the guidance required for the current step either by tool usage or code. 
+
+#### Tool Infomation
+
+You can use these tools:\n{formatted_tools}
+
+Valid "tool_name" value:\n{tool_names}
+
+#### Response Process
+
+**Question:** Start by understanding the input question to be answered.
+
+**Thoughts:** Considering the user's question, previously executed steps, and the plan, decide whether the current step requires the use of a tool or coding. Solve the problem step by step, only displaying the thought process necessary for the current step of solving the problem. If a tool can be used, provide its name and parameters. If coding is required, outline the plan for executing this step.
+
+**Action Status:** finished, tool_using, or coding. (Choose one from these three statuses. If the task is done, set it to 'finished'. If using a tool, set it to 'tool_using'. If writing code, set it to 'coding'.)
+
+**Action:** 
+
+If using a tool, output the following format to call the tool:
+
+```json
+{{
+  "tool_name": "$TOOL_NAME",
+  "tool_params": "$INPUT"
+}}
+```
+
+If the problem cannot be solved with a tool at the moment, then proceed to solve the issue using code. Output the following format to execute the code:
+
+```python
+# Write your code here
+```
+
+**Observation:** Check the results and effects of the executed action.
+
+... (Repeat this Thoughts/Action/Observation cycle as needed)
+
+**Thoughts:** Conclude the final response to the input question.
+
+**Action Status:** finished
+
+**Action:** The final answer or guidance to the original input question.
+"""
+
+# REACT_TOOL_AND_CODE_PROMPT = """你是一个使用工具与代码的助手。
+# 如果现有工具不足以完成整个任务，请不要添加不存在的工具，只使用现有工具完成可能的部分。
+# 如果当前步骤不能使用工具完成，将由代码来完成。
+# 有效的"action"值为："finished"（已经完成用户的任务） 、 "tool_using" (使用工具来回答问题) 或 'coding'(结合总结下述思维链过程编写下一步的可执行代码)。
+# 尽可能地以有帮助和准确的方式回应人类，你可以使用以下工具:
+# {formatted_tools}
+# 如果现在的步骤可以用工具解决问题，请仅在每个$JSON_BLOB中提供一个action，如下所示：
+# ```
+# {{{{
+# "action": $ACTION,
+# "tool_name": $TOOL_NAME
+# "tool_params": $INPUT
+# }}}}
+# ```
+# 若当前无法通过工具解决问题，则使用代码解决问题
+# 请仅在每个$JSON_BLOB中提供一个action，如下所示：
+# ```
+# {{{{'action': $ACTION,'code_content': $CODE}}}}
+# ```
+
+# 按照以下思维链格式进行回应（$JSON_BLOB要求符合上述规定）：
+# 问题：输入问题以回答
+# 思考：考虑之前和之后的步骤
+# 行动：
+# ```
+# $JSON_BLOB
+# ```
+# 观察：行动结果
+# ...（重复思考/行动/观察N次）
+# 思考：我知道该如何回应
+# 行动：
+# ```
+# $JSON_BLOB
+# ```
+# """

dev_opsgpt/connector/configs/prompts/react_tool_prompt.py

@@ -0,0 +1,81 @@
+REACT_TOOL_PROMPT = """#### Tool Agent Assistance Guidance
+
+When interacting with users, your role is to respond in a helpful and accurate manner using the tools available. Follow the steps below to ensure efficient and effective use of the tools.
+
+Please note that all the tools you can use are listed below. You can only choose from these tools for use. If there are no suitable tools, please do not invent any tools. Just let the user know that you do not have suitable tools to use.
+
+#### Tool List
+
+you can use these tools:\n{formatted_tools}
+
+valid "tool_name" value is:\n{tool_names}
+
+#### Response Process
+
+**Question:** Start by understanding the input question to be answered.
+
+**Thoughts:** Based on the question and previous observations, plan the approach for using the tool effectively.
+
+**Action Status:** Set to either 'finished' or 'tool_using'. If 'finished', provide the final response to the original question. If 'tool_using', proceed with using the specified tool.
+
+**Action:** Use the tools by formatting the tool action in JSON. The format should be:
+
+```json
+{{
+  "tool_name": "$TOOL_NAME",
+  "tool_params": "$INPUT"
+}}
+```
+
+**Observation:** Evaluate the outcome of the tool's usage.
+
+... (Repeat this Thoughts/Action/Observation cycle as needed)
+
+**Thoughts:** Determine the final response based on the results.
+
+**Action Status:** Set to 'finished'
+
+**Action:** Conclude with the final response to the original question in this format:
+
+```json
+{{
+  "tool_params": "Final response to be provided to the user",
+  "tool_name": "notool",
+}}
+```
+"""
+
+
+# REACT_TOOL_PROMPT = """尽可能地以有帮助和准确的方式回应人类。您可以使用以下工具:
+# {formatted_tools}
+# 使用json blob来指定一个工具，提供一个action关键字（工具名称）和一个tool_params关键字（工具输入）。
+# 有效的"action"值为："finished" 或 "tool_using" (使用工具来回答问题)
+# 有效的"tool_name"值为：{tool_names}
+# 请仅在每个$JSON_BLOB中提供一个action，如下所示：
+# ```
+# {{{{
+# "action": $ACTION,
+# "tool_name": $TOOL_NAME,
+# "tool_params": $INPUT
+# }}}}
+# ```
+
+# 按照以下格式进行回应：
+# 问题：输入问题以回答
+# 思考：考虑之前和之后的步骤
+# 行动：
+# ```
+# $JSON_BLOB
+# ```
+# 观察：行动结果
+# ...（重复思考/行动/观察N次）
+# 思考：我知道该如何回应
+# 行动：
+# ```
+# {{{{
+# "action": "finished",
+# "tool_name": "notool",
+# "tool_params": "最终返回答案给到用户"
+# }}}}
+# ```
+# """

dev_opsgpt/connector/configs/prompts/refine_template_prompt.py

@@ -0,0 +1,30 @@
+REFINE_TEMPLATE_PROMPT = """#### Refiner Assistance Guidance
+
+When users have a sequence of tasks that require optimization or adjustment based on feedback from the context, your role is to refine the existing plan.
+Your task is to identify where improvements can be made and provide a revised plan that is more efficient or effective.
+Each instruction should be an enhancement of the existing plan and should specify the step from which the changes should be implemented.
+
+#### Input Format
+
+**Context:** Review the history of the plan and feedback to identify areas for improvement. 
+Take into consideration all feedback information from the current step. If there is no existing plan, generate a new one.
+
+#### Response Output Format
+
+**REASON:** think the reason of why choose 'finished', 'unchanged' or 'adjusted' step by step.
+
+**Action Status:** Set to 'finished', 'unchanged' or 'adjusted'. 
+If it's 'finished', all tasks are accomplished, and no adjustments are needed, so PLAN_STEP is set to -1.
+If it's 'unchanged', this PLAN has no problem, just set PLAN_STEP to CURRENT_STEP+1.
+If it's 'adjusted', the PLAN is to provide an optimized version of the original plan. 
+
+**PLAN:**
+```list
+[
+  "First, we should ...",
+]
+```
+
+**PLAN_STEP:** Set to the plan index from which the changes should start. Index range from 0 to n-1 or -1
+If it's 'finished', the PLAN_STEP is -1. If it's 'adjusted', the PLAN_STEP is the index of the first revised task in the sequence.
+"""

dev_opsgpt/connector/configs/prompts/summary_template_prompt.py

@@ -0,0 +1,20 @@
+CONV_SUMMARY_PROMPT = """尽可能地以有帮助和准确的方式回应人类，根据“背景信息”中的有效信息回答问题，
+使用 JSON Blob 来指定一个返回的内容，提供一个 action（行动）。
+有效的 'action' 值为：'finished'(任务已经可以通过上下文信息可以回答) or 'continue' （根据背景信息回答问题）。
+在每个 $JSON_BLOB 中仅提供一个 action，如下所示：
+```
+{{'action': $ACTION, 'content': '根据背景信息回答问题'}}
+```
+按照以下格式进行回应：
+问题：输入问题以回答
+行动：
+```
+$JSON_BLOB
+```
+"""
+
+CONV_SUMMARY_PROMPT = """尽可能地以有帮助和准确的方式回应人类
+根据“背景信息”中的有效信息回答问题，同时展现解答的过程和内容
+若能根“背景信息”回答问题，则直接回答
+否则，总结“背景信息”的内容
+"""

dev_opsgpt/connector/message_process.py

@@ -0,0 +1,364 @@
+import re, traceback, uuid, copy, json, os
+from loguru import logger
+
+
+from configs.server_config import SANDBOX_SERVER
+from configs.model_config import JUPYTER_WORK_PATH
+from dev_opsgpt.connector.schema import (
+    Memory, Task, Env, Role, Message, ActionStatus, CodeDoc, Doc
+)
+from dev_opsgpt.tools import DDGSTool, DocRetrieval, CodeRetrieval
+from dev_opsgpt.sandbox import PyCodeBox, CodeBoxResponse
+
+
+class MessageUtils:
+    def __init__(self, role: Role = None) -> None:
+        self.role = role
+        self.codebox = PyCodeBox(
+                    remote_url=SANDBOX_SERVER["url"],
+                    remote_ip=SANDBOX_SERVER["host"],
+                    remote_port=SANDBOX_SERVER["port"],
+                    token="mytoken",
+                    do_code_exe=True,
+                    do_remote=SANDBOX_SERVER["do_remote"],
+                    do_check_net=False
+                    )
+        
+    def filter(self, message: Message, stop=None) -> Message:
+
+        tool_params = self.parser_spec_key(message.role_content, "tool_params")
+        code_content = self.parser_spec_key(message.role_content, "code_content")
+        plan = self.parser_spec_key(message.role_content, "plan")
+        plans = self.parser_spec_key(message.role_content, "plans", do_search=False)
+        content = self.parser_spec_key(message.role_content, "content", do_search=False)
+
+        # logger.debug(f"tool_params: {tool_params}, code_content: {code_content}, plan: {plan}, plans: {plans}, content: {content}")
+        role_content = tool_params or code_content or plan or plans or content
+        message.role_content = role_content or message.role_content
+        return message
+    
+    def inherit_extrainfo(self, input_message: Message, output_message: Message):
+        output_message.db_docs = input_message.db_docs
+        output_message.search_docs = input_message.search_docs
+        output_message.code_docs = input_message.code_docs
+        output_message.figures.update(input_message.figures)
+        output_message.origin_query = input_message.origin_query
+        return output_message
+
+    def get_extrainfo_step(self, message: Message, do_search, do_doc_retrieval, do_code_retrieval, do_tool_retrieval) -> Message:
+        ''''''
+        if do_search:
+            message = self.get_search_retrieval(message)
+        
+        if do_doc_retrieval:
+            message = self.get_doc_retrieval(message)
+
+        if do_code_retrieval:
+            input_message = self.get_code_retrieval(message)
+
+        if do_tool_retrieval:
+            message = self.get_tool_retrieval(message)
+        
+        return message 
+    
+    def get_search_retrieval(self, message: Message,) -> Message:
+        SEARCH_ENGINES = {"duckduckgo": DDGSTool}
+        search_docs = []
+        for idx, doc in enumerate(SEARCH_ENGINES["duckduckgo"].run(message.role_content, 3)):
+            doc.update({"index": idx})
+            search_docs.append(Doc(**doc))
+        message.search_docs = search_docs
+        return message
+    
+    def get_doc_retrieval(self, message: Message) -> Message:
+        query = message.role_content
+        knowledge_basename = message.doc_engine_name
+        top_k = message.top_k
+        score_threshold = message.score_threshold
+        if knowledge_basename:
+            docs = DocRetrieval.run(query, knowledge_basename, top_k, score_threshold)
+            message.db_docs = [Doc(**doc) for doc in docs]
+        return message
+    
+    def get_code_retrieval(self, message: Message) -> Message:
+        # DocRetrieval.run("langchain是什么", "DSADSAD")
+        query = message.input_query
+        code_engine_name = message.code_engine_name
+        history_node_list = message.history_node_list
+        code_docs = CodeRetrieval.run(code_engine_name, query, code_limit=message.top_k, history_node_list=history_node_list, search_type=message.cb_search_type)
+        message.code_docs = [CodeDoc(**doc) for doc in code_docs]
+        return message
+    
+    def get_tool_retrieval(self, message: Message) -> Message:
+        return message
+    
+    def step_router(self, message: Message) -> tuple[Message, ...]:
+        ''''''
+        # message = self.parser(message)
+        # logger.debug(f"message.action_status: {message.action_status}")
+        observation_message = None
+        if message.action_status == ActionStatus.CODING:
+            message, observation_message = self.code_step(message)
+        elif message.action_status == ActionStatus.TOOL_USING:
+            message, observation_message = self.tool_step(message)
+        elif message.action_status == ActionStatus.CODING2FILE:
+            self.save_code2file(message)
+
+        return message, observation_message
+
+    def code_step(self, message: Message) -> Message:
+        '''execute code'''
+        # logger.debug(f"message.role_content: {message.role_content}, message.code_content: {message.code_content}")
+        code_answer = self.codebox.chat('```python\n{}```'.format(message.code_content))
+        code_prompt = f"The return error after executing the above code is {code_answer.code_exe_response}，need to recover" \
+                    if code_answer.code_exe_type == "error" else f"The return information after executing the above code is {code_answer.code_exe_response}"
+        
+        observation_message = Message(
+                role_name="observation",
+                role_type="func", #self.role.role_type,
+                role_content="",
+                step_content="",
+                input_query=message.code_content,
+                )
+        uid = str(uuid.uuid1())
+        if code_answer.code_exe_type == "image/png":
+            message.figures[uid] = code_answer.code_exe_response
+            message.code_answer = f"\n**Observation:**: The return figure name is {uid} after executing the above code.\n"
+            message.observation = f"\n**Observation:**: The return figure name is {uid} after executing the above code.\n"
+            message.step_content += f"\n**Observation:**: The return figure name is {uid} after executing the above code.\n"
+            message.step_contents += [f"\n**Observation:**:The return figure name is {uid} after executing the above code.\n"]
+            # message.role_content += f"\n**Observation:**:执行上述代码后生成一张图片, 图片名为{uid}\n"
+            observation_message.role_content = f"\n**Observation:**: The return figure name is {uid} after executing the above code.\n"
+            observation_message.parsed_output = {"Observation": f"The return figure name is {uid} after executing the above code."}
+        else:
+            message.code_answer = code_answer.code_exe_response
+            message.observation = code_answer.code_exe_response
+            message.step_content += f"\n**Observation:**: {code_prompt}\n"
+            message.step_contents += [f"\n**Observation:**: {code_prompt}\n"]
+            # message.role_content += f"\n**Observation:**: {code_prompt}\n"
+            observation_message.role_content = f"\n**Observation:**: {code_prompt}\n"
+            observation_message.parsed_output = {"Observation": code_prompt}
+        # logger.info(f"**Observation:** {message.action_status}, {message.observation}")
+        return message, observation_message
+
+    def tool_step(self, message: Message) -> Message:
+        '''execute tool'''
+        # logger.debug(f"{message}")
+        observation_message = Message(
+                role_name="observation",
+                role_type="function", #self.role.role_type,
+                role_content="\n**Observation:** there is no tool can execute\n"    ,
+                step_content="",
+                input_query=str(message.tool_params),
+                tools=message.tools,
+                )
+        # logger.debug(f"message: {message.action_status}, {message.tool_name}, {message.tool_params}")
+        tool_names = [tool.name for tool in message.tools]
+        if message.tool_name not in tool_names:
+            message.tool_answer = "\n**Observation:** there is no tool can execute\n"    
+            message.observation = "\n**Observation:** there is no tool can execute\n"    
+            # message.role_content += f"\n**Observation:**: 不存在可以执行的tool\n"
+            message.step_content += f"\n**Observation:** there is no tool can execute\n"
+            message.step_contents += [f"\n**Observation:** there is no tool can execute\n"]
+            observation_message.role_content = f"\n**Observation:** there is no tool can execute\n"
+            observation_message.parsed_output = {"Observation": "there is no tool can execute\n"}
+        for tool in message.tools:
+            if tool.name == message.tool_name:
+                tool_res = tool.func(**message.tool_params.get("tool_params", {}))
+                logger.debug(f"tool_res {tool_res}")
+                message.tool_answer = tool_res    
+                message.observation = tool_res
+                # message.role_content += f"\n**Observation:**: {tool_res}\n"
+                message.step_content += f"\n**Observation:** {tool_res}\n"
+                message.step_contents += [f"\n**Observation:** {tool_res}\n"]
+                observation_message.role_content = f"\n**Observation:** {tool_res}\n"
+                observation_message.parsed_output = {"Observation": tool_res}
+                break
+
+        # logger.info(f"**Observation:** {message.action_status}, {message.observation}")
+        return message, observation_message
+    
+    def parser(self, message: Message) -> Message:
+        ''''''
+        content = message.role_content
+        parser_keys = ["action", "code_content", "code_filename", "tool_params", "plans"]
+        try:
+            s_json = self._parse_json(content)
+            message.action_status = s_json.get("action")
+            message.code_content = s_json.get("code_content")
+            message.tool_params = s_json.get("tool_params")
+            message.tool_name = s_json.get("tool_name")
+            message.code_filename = s_json.get("code_filename")
+            message.plans = s_json.get("plans")
+            # for parser_key in parser_keys:
+            #     message.action_status = content.get(parser_key)
+        except Exception as e:
+            # logger.warning(f"{traceback.format_exc()}")
+            def parse_text_to_dict(text):
+                # Define a regular expression pattern to capture the key and value
+                main_pattern = r"\*\*(.+?):\*\*\s*(.*?)\s*(?=\*\*|$)"
+                list_pattern = r'```python\n(.*?)```'
+
+                # Use re.findall to find all main matches in the text
+                main_matches = re.findall(main_pattern, text, re.DOTALL)
+
+                # Convert main matches to a dictionary
+                parsed_dict = {key.strip(): value.strip() for key, value in main_matches}
+
+                for k, v in parsed_dict.items():
+                    for pattern in [list_pattern]:
+                        if "PLAN" != k: continue
+                        v = v.replace("```list", "```python")
+                        match_value = re.search(pattern, v, re.DOTALL)
+                        if match_value:
+                            # Add the code block to the dictionary
+                            parsed_dict[k] = eval(match_value.group(1).strip())
+                            break
+
+                return parsed_dict
+            
+            def extract_content_from_backticks(text):
+                code_blocks = []
+                lines = text.split('\n')
+                is_code_block = False
+                code_block = ''
+                language = ''
+                for line in lines:
+                    if line.startswith('```') and not is_code_block:
+                        is_code_block = True
+                        language = line[3:]
+                        code_block = ''
+                    elif line.startswith('```') and is_code_block:
+                        is_code_block = False
+                        code_blocks.append({language.strip(): code_block.strip()})
+                    elif is_code_block:
+                        code_block += line + '\n'
+                return code_blocks
+            
+            def parse_dict_to_dict(parsed_dict) -> dict:
+                code_pattern = r'```python\n(.*?)```'
+                tool_pattern = r'```json\n(.*?)```'
+                
+                pattern_dict = {"code": code_pattern, "json": tool_pattern}
+                spec_parsed_dict = copy.deepcopy(parsed_dict)
+                for key, pattern in pattern_dict.items():
+                    for k, text in parsed_dict.items():
+                        # Search for the code block
+                        if not isinstance(text, str): continue
+                        _match = re.search(pattern, text, re.DOTALL)
+                        if _match:
+                            # Add the code block to the dictionary
+                            try:
+                                spec_parsed_dict[key] = json.loads(_match.group(1).strip())
+                            except:
+                                spec_parsed_dict[key] = _match.group(1).strip()
+                            break
+                return spec_parsed_dict
+
+            parsed_dict = parse_text_to_dict(content)
+            spec_parsed_dict = parse_dict_to_dict(parsed_dict)
+            action_value = parsed_dict.get('Action Status')
+            if action_value:
+                action_value = action_value.lower()
+            logger.info(f'{message.role_name}: action_value: {action_value}')
+            # action_value = self._match(r"'action':\s*'([^']*)'", content) if "'action'" in content else self._match(r'"action":\s*"([^"]*)"', content)
+            
+            code_content_value = spec_parsed_dict.get('code')
+            # code_content_value = self._match(r"'code_content':\s*'([^']*)'", content) if "'code_content'" in content else self._match(r'"code_content":\s*"([^"]*)"', content)
+            filename_value = self._match(r"'code_filename':\s*'([^']*)'", content) if "'code_filename'" in content else self._match(r'"code_filename":\s*"([^"]*)"', content)
+
+            if action_value == 'tool_using':
+                tool_params_value = spec_parsed_dict.get('json')
+            else:
+                tool_params_value = None
+            # tool_params_value = spec_parsed_dict.get('tool_params')
+            # tool_params_value = self._match(r"'tool_params':\s*(\{[^{}]*\})", content, do_json=True) if "'tool_params'" in content \
+            #                         else self._match(r'"tool_params":\s*(\{[^{}]*\})', content, do_json=True)
+            tool_name_value = self._match(r"'tool_name':\s*'([^']*)'", content) if "'tool_name'" in content else self._match(r'"tool_name":\s*"([^"]*)"', content)
+            plans_value = self._match(r"'plans':\s*(\[.*?\])", content, do_search=False) if "'plans'" in content else self._match(r'"plans":\s*(\[.*?\])', content, do_search=False, )
+            # re解析
+            message.action_status = action_value or "default"
+            message.code_content = code_content_value
+            message.code_filename = filename_value
+            message.tool_params = tool_params_value
+            message.tool_name = tool_name_value
+            message.plans = plans_value
+            message.parsed_output = parsed_dict
+            message.spec_parsed_output = spec_parsed_dict
+
+        # logger.debug(f"确认当前的action: {message.action_status}")
+
+        return message
+    
+    def parser_spec_key(self, content, key, do_search=True, do_json=False) -> str:
+        ''''''
+        key2pattern = {
+            "'action'": r"'action':\s*'([^']*)'", '"action"': r'"action":\s*"([^"]*)"',
+            "'code_content'": r"'code_content':\s*'([^']*)'", '"code_content"': r'"code_content":\s*"([^"]*)"',
+            "'code_filename'": r"'code_filename':\s*'([^']*)'", '"code_filename"': r'"code_filename":\s*"([^"]*)"',
+            "'tool_params'": r"'tool_params':\s*(\{[^{}]*\})", '"tool_params"': r'"tool_params":\s*(\{[^{}]*\})',
+            "'tool_name'": r"'tool_name':\s*'([^']*)'", '"tool_name"': r'"tool_name":\s*"([^"]*)"',
+            "'plans'": r"'plans':\s*(\[.*?\])", '"plans"': r'"plans":\s*(\[.*?\])',
+            "'content'": r"'content':\s*'([^']*)'", '"content"': r'"content":\s*"([^"]*)"',
+            }
+        
+        s_json = self._parse_json(content)
+        try:
+            if s_json and key in s_json:
+                return str(s_json[key])
+        except:
+            pass
+
+        keystr = f"'{key}'" if f"'{key}'" in content else f'"{key}"'
+        return self._match(key2pattern.get(keystr, fr"'{key}':\s*'([^']*)'"), content, do_search=do_search, do_json=do_json)
+    
+    def _match(self, pattern, s, do_search=True, do_json=False):
+        try:
+            if do_search:
+                match = re.search(pattern, s)
+                if match:
+                    value = match.group(1).replace("\\n", "\n")
+                    if do_json:
+                        value = json.loads(value)
+                else:
+                    value = None
+            else:
+                match = re.findall(pattern, s, re.DOTALL)
+                if match:
+                    value = match[0]
+                    if do_json:
+                        value = json.loads(value)
+                else:
+                    value = None
+        except Exception as e:
+            logger.warning(f"{traceback.format_exc()}")
+
+        # logger.debug(f"pattern: {pattern}, s: {s}, match: {match}")
+        return value
+    
+    def _parse_json(self, s):
+        try:
+            pattern = r"```([^`]+)```"
+            match = re.findall(pattern, s)
+            if match:
+                return eval(match[0])
+        except:
+            pass
+        return None
+    
+
+    def save_code2file(self, message: Message, project_dir=JUPYTER_WORK_PATH):
+        filename = message.parsed_output.get("SaveFileName")
+        code = message.spec_parsed_output.get("code")
+
+        for k, v in {"&gt;": ">", "&ge;": ">=", "&lt;": "<", "&le;": "<="}.items():
+            code = code.replace(k, v)
+
+        file_path = os.path.join(project_dir, filename)
+
+        if not os.path.exists(file_path):
+            os.makedirs(os.path.dirname(file_path), exist_ok=True)
+
+        with open(file_path, "w") as f:
+            f.write(code)
+        

dev_opsgpt/connector/phase/base_phase.py

@@ -8,14 +8,13 @@ from loguru import logger
 from dev_opsgpt.connector.agents import BaseAgent
 from dev_opsgpt.connector.chains import BaseChain
 from dev_opsgpt.tools.base_tool import BaseTools, Tool
-from dev_opsgpt.connector.shcema.memory import Memory
-from dev_opsgpt.connector.connector_schema import (
-    Task, Env, Role, Message, Doc, Docs, AgentConfig, ChainConfig, PhaseConfig, CodeDoc,
+
+from dev_opsgpt.connector.schema import (
+    Memory, Task, Env, Role, Message, Doc, Docs, AgentConfig, ChainConfig, PhaseConfig, CodeDoc,
     load_chain_configs, load_phase_configs, load_role_configs
 )
 from dev_opsgpt.connector.configs import AGETN_CONFIGS, CHAIN_CONFIGS, PHASE_CONFIGS
-from dev_opsgpt.tools import DDGSTool, DocRetrieval, CodeRetrieval
-
+from dev_opsgpt.connector.message_process import MessageUtils
 
 role_configs = load_role_configs(AGETN_CONFIGS)
 chain_configs = load_chain_configs(CHAIN_CONFIGS)
@@ -56,57 +55,71 @@ class BasePhase:
             chain_config = chain_config,
             role_config = role_config,
             )
+        
+        self.message_utils = MessageUtils()
         self.phase_name = phase_name
         self.do_summary = do_summary
         self.do_search = do_search
         self.do_code_retrieval = do_code_retrieval
         self.do_doc_retrieval = do_doc_retrieval
         self.do_tool_retrieval = do_tool_retrieval
-
-        self.global_message = Memory([])
+        # 
+        self.global_memory = Memory(messages=[])
         # self.chain_message = Memory([])
         self.phase_memory: List[Memory] = []
+        # memory_pool dont have specific order
+        self.memory_pool = Memory(messages=[])
 
-    def step(self, query: Message, history: Memory = None) -> Tuple[Message, Memory]:
+    def astep(self, query: Message, history: Memory = None) -> Tuple[Message, Memory]:
         summary_message = None
-        chain_message = Memory([])
-        local_memory = Memory([])
+        chain_message = Memory(messages=[])
+        local_phase_memory = Memory(messages=[])
         # do_search、do_doc_search、do_code_search
-        query = self.get_extrainfo_step(query)
+        query = self.message_utils.get_extrainfo_step(query, self.do_search, self.do_doc_retrieval, self.do_code_retrieval, self.do_tool_retrieval)
         input_message = copy.deepcopy(query)
         
-        self.global_message.append(input_message)
+        self.global_memory.append(input_message)
+        local_phase_memory.append(input_message)
         for chain in self.chains:
             # chain can supply background and query to next chain
-            output_message, chain_memory = chain.step(input_message, history, background=chain_message)
-            output_message = self.inherit_extrainfo(input_message, output_message)
+            for output_message, local_chain_memory in chain.astep(input_message, history, background=chain_message, memory_pool=self.memory_pool):
+                # logger.debug(f"local_memory: {local_memory + chain_memory}")
+                yield output_message, local_phase_memory + local_chain_memory
+
+            output_message = self.message_utils.inherit_extrainfo(input_message, output_message)
             input_message = output_message
             logger.info(f"{chain.chainConfig.chain_name} phase_step: {output_message.role_content}")
+            # 这一段也有问题
+            self.global_memory.extend(local_chain_memory)
+            local_phase_memory.extend(local_chain_memory)
 
-            self.global_message.append(output_message)
-            local_memory.extend(chain_memory)
-
-            # whether use summary_llm
+            # whether to use summary_llm
             if self.do_summary:
-                logger.info(f"{self.conv_summary_agent.role.role_name} input global memory: {self.global_message.to_str_messages(content_key='step_content')}")
-                logger.info(f"{self.conv_summary_agent.role.role_name} input global memory: {self.global_message.to_str_messages(content_key='role_content')}")
-                summary_message = self.conv_summary_agent.run(query, background=self.global_message)
+                logger.info(f"{self.conv_summary_agent.role.role_name} input global memory: {local_phase_memory.to_str_messages(content_key='step_content')}")
+                for summary_message in self.conv_summary_agent.arun(query, background=local_phase_memory, memory_pool=self.memory_pool):
+                    pass
+                # summary_message = Message(**summary_message)
                 summary_message.role_name = chain.chainConfig.chain_name
-                summary_message = self.conv_summary_agent.parser(summary_message)
-                summary_message = self.conv_summary_agent.filter(summary_message)
-                summary_message = self.inherit_extrainfo(output_message, summary_message)
+                summary_message = self.conv_summary_agent.message_utils.parser(summary_message)
+                summary_message = self.conv_summary_agent.message_utils.filter(summary_message)
+                summary_message = self.message_utils.inherit_extrainfo(output_message, summary_message)
                 chain_message.append(summary_message)
-        
+
+                message = summary_message or output_message
+                yield message, local_phase_memory
+
         # 由于不会存在多轮chain执行，所以直接保留memory即可
         for chain in self.chains:
             self.phase_memory.append(chain.global_memory)
-
+        # TODO：local_memory缺少添加summary的过程
         message = summary_message or output_message
         message.role_name = self.phase_name
-        # message.db_docs = query.db_docs
-        # message.code_docs = query.code_docs
-        # message.search_docs = query.search_docs
-        return summary_message or output_message, local_memory
+        yield message, local_phase_memory
+
+    def step(self, query: Message, history: Memory = None) -> Tuple[Message, Memory]:
+        for message, local_phase_memory in self.astep(query, history=history):
+            pass
+        return message, local_phase_memory
 
     def init_chains(self, phase_name,  phase_config, chain_config,
             role_config, task=None, memory=None) -> List[BaseChain]:
@@ -118,25 +131,33 @@ class BasePhase:
         chains = []
         self.chain_module = importlib.import_module("dev_opsgpt.connector.chains")
         self.agent_module = importlib.import_module("dev_opsgpt.connector.agents")
-        phase = phase_configs.get(phase_name)
-        for chain_name in phase.chains:
-            logger.info(f"chain_name: {chain_name}")
-            # chain_class = getattr(self.chain_module, chain_name)
-            logger.debug(f"{chain_configs.keys()}")
-            chain_config = chain_configs[chain_name]
 
-            agents = [
-                getattr(self.agent_module, role_configs[agent_name].role.agent_type)(
-                    role_configs[agent_name].role, 
+        phase = phase_configs.get(phase_name)
+        logger.info(f"start to init the phase, the phase_name is {phase_name}, it contains these chains such as {phase.chains}")
+        
+        for chain_name in phase.chains:
+            # logger.debug(f"{chain_configs.keys()}")
+            chain_config = chain_configs[chain_name]
+            logger.info(f"start to init the chain, the chain_name is {chain_name}, it contains these agents such as {chain_config.agents}")
+
+            agents = []
+            for agent_name in chain_config.agents:
+                agent_config = role_configs[agent_name]
+                baseAgent: BaseAgent = getattr(self.agent_module, agent_config.role.agent_type)
+                base_agent = baseAgent(
+                    agent_config.role, 
                     task = task,
                     memory = memory,
-                    chat_turn=role_configs[agent_name].chat_turn,
-                    do_search = role_configs[agent_name].do_search,
-                    do_doc_retrieval = role_configs[agent_name].do_doc_retrieval,
-                    do_tool_retrieval = role_configs[agent_name].do_tool_retrieval,
+                    chat_turn=agent_config.chat_turn,
+                    do_search = agent_config.do_search,
+                    do_doc_retrieval = agent_config.do_doc_retrieval,
+                    do_tool_retrieval = agent_config.do_tool_retrieval,
+                    stop= agent_config.stop,
+                    focus_agents=agent_config.focus_agents,
+                    focus_message_keys=agent_config.focus_message_keys,
                 ) 
-                for agent_name in chain_config.agents
-            ]
+                agents.append(base_agent)
+            
             chain_instance = BaseChain(
                 chain_config, agents, chain_config.chat_turn, 
                 do_checker=chain_configs[chain_name].do_checker, 
@@ -145,56 +166,57 @@ class BasePhase:
 
         return chains
 
-    def get_extrainfo_step(self, input_message):
-        if self.do_doc_retrieval:
-            input_message = self.get_doc_retrieval(input_message)
+    # def get_extrainfo_step(self, input_message):
+    #     if self.do_doc_retrieval:
+    #         input_message = self.get_doc_retrieval(input_message)
 
-        logger.debug(F"self.do_code_retrieval: {self.do_code_retrieval}")
-        if self.do_code_retrieval:
-            input_message = self.get_code_retrieval(input_message)
+    #     # logger.debug(F"self.do_code_retrieval: {self.do_code_retrieval}")
+    #     if self.do_code_retrieval:
+    #         input_message = self.get_code_retrieval(input_message)
 
-        if self.do_search:
-            input_message = self.get_search_retrieval(input_message)
+    #     if self.do_search:
+    #         input_message = self.get_search_retrieval(input_message)
 
-        return input_message
+    #     return input_message
 
-    def inherit_extrainfo(self, input_message: Message, output_message: Message):
-        output_message.db_docs = input_message.db_docs
-        output_message.search_docs = input_message.search_docs
-        output_message.code_docs = input_message.code_docs
-        output_message.figures.update(input_message.figures)
-        return output_message
+    # def inherit_extrainfo(self, input_message: Message, output_message: Message):
+    #     output_message.db_docs = input_message.db_docs
+    #     output_message.search_docs = input_message.search_docs
+    #     output_message.code_docs = input_message.code_docs
+    #     output_message.figures.update(input_message.figures)
+    #     output_message.origin_query = input_message.origin_query
+    #     return output_message
         
-    def get_search_retrieval(self, message: Message,) -> Message:
-        SEARCH_ENGINES = {"duckduckgo": DDGSTool}
-        search_docs = []
-        for idx, doc in enumerate(SEARCH_ENGINES["duckduckgo"].run(message.role_content, 3)):
-            doc.update({"index": idx})
-            search_docs.append(Doc(**doc))
-        message.search_docs = search_docs
-        return message
+    # def get_search_retrieval(self, message: Message,) -> Message:
+    #     SEARCH_ENGINES = {"duckduckgo": DDGSTool}
+    #     search_docs = []
+    #     for idx, doc in enumerate(SEARCH_ENGINES["duckduckgo"].run(message.role_content, 3)):
+    #         doc.update({"index": idx})
+    #         search_docs.append(Doc(**doc))
+    #     message.search_docs = search_docs
+    #     return message
     
-    def get_doc_retrieval(self, message: Message) -> Message:
-        query = message.role_content
-        knowledge_basename = message.doc_engine_name
-        top_k = message.top_k
-        score_threshold = message.score_threshold
-        if knowledge_basename:
-            docs = DocRetrieval.run(query, knowledge_basename, top_k, score_threshold)
-            message.db_docs = [Doc(**doc) for doc in docs]
-        return message
+    # def get_doc_retrieval(self, message: Message) -> Message:
+    #     query = message.role_content
+    #     knowledge_basename = message.doc_engine_name
+    #     top_k = message.top_k
+    #     score_threshold = message.score_threshold
+    #     if knowledge_basename:
+    #         docs = DocRetrieval.run(query, knowledge_basename, top_k, score_threshold)
+    #         message.db_docs = [Doc(**doc) for doc in docs]
+    #     return message
     
-    def get_code_retrieval(self, message: Message) -> Message:
-        # DocRetrieval.run("langchain是什么", "DSADSAD")
-        query = message.input_query
-        code_engine_name = message.code_engine_name
-        history_node_list = message.history_node_list
-        code_docs = CodeRetrieval.run(code_engine_name, query, code_limit=message.top_k, history_node_list=history_node_list)
-        message.code_docs = [CodeDoc(**doc) for doc in code_docs]
-        return message
+    # def get_code_retrieval(self, message: Message) -> Message:
+    #     # DocRetrieval.run("langchain是什么", "DSADSAD")
+    #     query = message.input_query
+    #     code_engine_name = message.code_engine_name
+    #     history_node_list = message.history_node_list
+    #     code_docs = CodeRetrieval.run(code_engine_name, query, code_limit=message.top_k, history_node_list=history_node_list)
+    #     message.code_docs = [CodeDoc(**doc) for doc in code_docs]
+    #     return message
 
-    def get_tool_retrieval(self, message: Message) -> Message:
-        return message
+    # def get_tool_retrieval(self, message: Message) -> Message:
+    #     return message
     
     def update(self) -> Memory:
         pass
@@ -205,7 +227,7 @@ class BasePhase:
             )
     
     def get_memory_str(self, do_all_memory=True, content_key="role_content") -> str:
-        memory = self.global_message if do_all_memory else self.phase_memory
+        memory = self.global_memory if do_all_memory else self.phase_memory
         return "\n".join([": ".join(i) for i in memory.to_tuple_messages(content_key=content_key)])
     
     def get_chains_memory(self, content_key="role_content") -> List[Tuple]:

dev_opsgpt/connector/schema/__init__.py

@@ -0,0 +1,9 @@
+from .memory import Memory
+from .general_schema import *
+from .message import Message
+
+__all__ = [
+    "Memory", "ActionStatus", "Doc", "CodeDoc", "Task",
+    "Env", "Role", "ChainConfig", "AgentConfig", "PhaseConfig", "Message",
+    "load_role_configs", "load_chain_configs", "load_phase_configs"
+]

dev_opsgpt/connector/schema/general_schema.py

@@ -16,11 +16,50 @@ class ActionStatus(Enum):
     EXECUTING_CODE = "executing_code"
     EXECUTING_TOOL = "executing_tool"
     DEFAUILT = "default"
+    CODING2FILE = "coding2file"
 
     def __eq__(self, other):
         if isinstance(other, str):
             return self.value == other
         return super().__eq__(other)
+    
+
+class RoleTypeEnums(Enum):
+    SYSTEM = "system"
+    USER = "user"
+    ASSISTANT = "assistant"
+    FUNCTION = "function"
+    OBSERVATION = "observation"
+
+    def __eq__(self, other):
+        if isinstance(other, str):
+            return self.value == other
+        return super().__eq__(other)
+
+
+class InputKeyEnums(Enum):
+    # Origin Query is ui's user question
+    ORIGIN_QUERY = "origin_query"
+    # agent's input from last agent
+    CURRENT_QUESTION = "current_question"
+    # ui memory contaisn (user and assistants)
+    UI_MEMORY = "ui_memory"
+    # agent's memory
+    SELF_MEMORY = "self_memory"
+    # chain memory
+    CHAIN_MEMORY = "chain_memory"
+    # agent's memory
+    SELF_ONE_MEMORY = "self_one_memory"
+    # chain memory
+    CHAIN_ONE_MEMORY = "chain_one_memory"
+    # Doc Infomations contains (Doc\Code\Search)
+    DOC_INFOS = "doc_infos"
+
+    def __eq__(self, other):
+        if isinstance(other, str):
+            return self.value == other
+        return super().__eq__(other)
+
 
 class Doc(BaseModel):
     title: str
@@ -117,10 +156,13 @@ class ChainConfig(BaseModel):
 
 class AgentConfig(BaseModel):
     role: Role
+    stop: str = None
     chat_turn: int = 1
     do_search: bool =  False
     do_doc_retrieval: bool = False
     do_tool_retrieval: bool = False
+    focus_agents: List = []
+    focus_message_keys: List = []
 
 
 class PhaseConfig(BaseModel):
@@ -133,92 +175,6 @@ class PhaseConfig(BaseModel):
     do_code_retrieval: bool = False
     do_tool_retrieval: bool = False
 
-class Message(BaseModel):
-    role_name: str
-    role_type: str
-    role_prompt: str = None
-    input_query: str = None
-
-    # 模型最终返回
-    role_content: str = None
-    role_contents: List[str] = []
-    step_content: str = None
-    step_contents: List[str] = []
-    chain_content: str = None
-    chain_contents: List[str] = []
-
-    # 模型结果解析
-    plans: List[str] = None
-    code_content: str = None
-    code_filename: str = None
-    tool_params: str = None
-    tool_name: str = None
-
-    # 执行结果
-    action_status: str = ActionStatus.DEFAUILT
-    code_answer: str = None
-    tool_answer: str = None
-    observation: str = None
-    figures: Dict[str, str] = {}
-
-    # 辅助信息
-    tools: List[BaseTool] = []
-    task: Task = None
-    db_docs: List['Doc'] = []
-    code_docs: List['CodeDoc'] = []
-    search_docs: List['Doc'] = []
-
-    # 执行输入
-    phase_name: str = None
-    chain_name: str = None
-    do_search: bool = False
-    doc_engine_name: str = None
-    code_engine_name: str = None
-    search_engine_name: str = None 
-    top_k: int = 3
-    score_threshold: float = 1.0
-    do_doc_retrieval: bool = False
-    do_code_retrieval: bool = False
-    do_tool_retrieval: bool = False
-    history_node_list: List[str] = []
-
-
-    def to_tuple_message(self, return_all: bool = False, content_key="role_content"):
-        if content_key == "role_content":
-            role_content = self.role_content
-        elif content_key == "step_content":
-            role_content  = self.step_content or self.role_content
-        else:
-            role_content =self.role_content
-
-        if return_all:
-            return (self.role_name, self.role_type, role_content)
-        else:
-            return (self.role_name, role_content)
-            return (self.role_type, re.sub("}", "}}", re.sub("{", "{{", str(self.role_content))))
-    
-    def to_dict_message(self, return_all: bool = False, content_key="role_content"):
-        if content_key == "role_content":
-            role_content =self.role_content
-        elif content_key == "step_content":
-            role_content  = self.step_content or self.role_content
-        else:
-            role_content =self.role_content
-
-        if return_all:
-            return vars(self)
-        else:
-            return {"role": self.role_name, "content": role_content}
-    
-    def is_system_role(self,):
-        return self.role_type == "system"
-    
-    def __str__(self) -> str:
-        # key_str = '\n'.join([k for k, v in vars(self).items()])
-        # logger.debug(f"{key_str}")
-        return "\n".join([": ".join([k, str(v)]) for k, v in vars(self).items()])
-    
-
 
 def load_role_configs(config) -> Dict[str, AgentConfig]:
     if isinstance(config, str):

dev_opsgpt/connector/schema/memory.py

@@ -1,17 +1,18 @@
 from pydantic import BaseModel
-from typing import List
+from typing import List, Union
 from loguru import logger
 
-from dev_opsgpt.connector.connector_schema import Message
+from .message import Message
 from dev_opsgpt.utils.common_utils import (
     save_to_jsonl_file, save_to_json_file, read_json_file, read_jsonl_file
 )
 
 
-class Memory:
-    
-    def __init__(self, messages: List[Message] = []):
-        self.messages = messages
+class Memory(BaseModel):
+    messages: List[Message] = []
+
+    # def __init__(self, messages: List[Message] = []):
+    #     self.messages = messages
 
     def append(self, message: Message):
         self.messages.append(message)
@@ -28,8 +29,9 @@ class Memory:
     def delete(self, ):
         pass
 
-    def get_messages(self, ) -> List[Message]:
-        return self.messages
+    def get_messages(self, k=0) -> List[Message]:
+        """Return the most recent k memories, return all when k=0"""
+        return self.messages[-k:]
 
     def save(self, file_type="jsonl", return_all=True):
         try:
@@ -57,32 +59,56 @@ class Memory:
         
         return False
     
-    def to_tuple_messages(self, return_system: bool = False, return_all: bool = False, content_key="role_content"):
+    def to_tuple_messages(self, return_all: bool = True, content_key="role_content", filter_roles=[]):
         # logger.debug(f"{[message.to_tuple_message(return_all, content_key) for message in self.messages  ]}")
         return [
             message.to_tuple_message(return_all, content_key) for message in self.messages 
-            if not message.is_system_role() |  return_system
+            if message.role_name not in filter_roles
             ]
     
-    def to_dict_messages(self, return_system: bool = False, return_all: bool = False, content_key="role_content"):
+    def to_dict_messages(self, return_all: bool = True, content_key="role_content", filter_roles=[]):
         return [
             message.to_dict_message(return_all, content_key) for message in self.messages 
-            if not message.is_system_role() |  return_system
+            if message.role_name not in filter_roles
             ]
     
-    def to_str_messages(self, return_system: bool = False, return_all: bool = False, content_key="role_content"):
+    def to_str_messages(self, return_all: bool = True, content_key="role_content", filter_roles=[]):
+        # for message in self.messages:
+        #     logger.debug(f"{message.to_tuple_message(return_all, content_key)}")
         # logger.debug(f"{[message.to_tuple_message(return_all, content_key) for message in self.messages  ]}")
-        return "\n".join([
-            ": ".join(message.to_tuple_message(return_all, content_key)) for message in self.messages 
-            if not message.is_system_role() |  return_system
+        return "\n\n".join([message.to_str_content(return_all, content_key) for message in self.messages 
+            if message.role_name not in filter_roles
             ])
     
+    def get_parserd_output(self, ):
+        return [message.parsed_output for message in self.messages]
+    
+    def get_parserd_output_list(self, ):
+        # for message in self.messages:
+        #     logger.debug(f"{message.role_name}: {message.parsed_output_list}")
+        return [parsed_output for message in self.messages for parsed_output in message.parsed_output_list[1:]]
+    
+    def get_rolenames(self, ):
+        ''''''
+        return [message.role_name for message in self.messages]
+    
     @classmethod
     def from_memory_list(cls, memorys: List['Memory']) -> 'Memory':
-        return cls([message for memory in memorys for message in memory.get_messages()])
+        return cls(messages=[message for memory in memorys for message in memory.get_messages()])
     
     def __len__(self, ):
         return len(self.messages)
 
     def __str__(self) -> str:
-        return "\n".join([":".join(i) for i in self.to_tuple_messages()])
+        return "\n".join([":".join(i) for i in self.to_tuple_messages()])
+    
+    def __add__(self, other: Union[Message, 'Memory']) -> 'Memory':
+        if isinstance(other, Message):
+            return Memory(messages=self.messages + [other])
+        elif isinstance(other, Memory):
+            return Memory(messages=self.messages + other.messages)
+        else:
+            raise ValueError(f"cant add unspecified type like as {type(other)}")
+        
+    
+    

dev_opsgpt/connector/schema/message.py

@@ -0,0 +1,98 @@
+from pydantic import BaseModel
+from loguru import logger
+
+from .general_schema import *
+
+
+class Message(BaseModel):
+    chat_index: str = None
+    role_name: str
+    role_type: str
+    role_prompt: str = None
+    input_query: str = None
+    origin_query: str = None
+
+    # 模型最终返回
+    role_content: str = None
+    role_contents: List[str] = []
+    step_content: str = None
+    step_contents: List[str] = []
+    chain_content: str = None
+    chain_contents: List[str] = []
+
+    # 模型结果解析
+    plans: List[str] = None
+    code_content: str = None
+    code_filename: str = None
+    tool_params: str = None
+    tool_name: str = None
+    parsed_output: dict = {}
+    spec_parsed_output: dict = {}
+    parsed_output_list: List[Dict] = []
+
+    # 执行结果
+    action_status: str = ActionStatus.DEFAUILT
+    agent_index: int = None
+    code_answer: str = None
+    tool_answer: str = None
+    observation: str = None
+    figures: Dict[str, str] = {}
+
+    # 辅助信息
+    tools: List[BaseTool] = []
+    task: Task = None
+    db_docs: List['Doc'] = []
+    code_docs: List['CodeDoc'] = []
+    search_docs: List['Doc'] = []
+    agents: List = []
+
+    # 执行输入
+    phase_name: str = None
+    chain_name: str = None
+    do_search: bool = False
+    doc_engine_name: str = None
+    code_engine_name: str = None
+    cb_search_type: str = None
+    search_engine_name: str = None 
+    top_k: int = 3
+    score_threshold: float = 1.0
+    do_doc_retrieval: bool = False
+    do_code_retrieval: bool = False
+    do_tool_retrieval: bool = False
+    history_node_list: List[str] = []
+
+    def to_tuple_message(self, return_all: bool = True, content_key="role_content"):
+        role_content = self.to_str_content(False, content_key)
+        if return_all:
+            return (self.role_name, role_content)
+        else:
+            return (role_content)
+    
+    def to_dict_message(self, return_all: bool = True, content_key="role_content"):
+        role_content = self.to_str_content(False, content_key)
+        if return_all:
+            return {"role": self.role_name, "content": role_content}
+        else:
+            return vars(self)
+        
+    def to_str_content(self, return_all: bool = True, content_key="role_content"):
+        if content_key == "role_content":
+            role_content = self.role_content or self.input_query
+        elif content_key == "step_content":
+            role_content  = self.step_content or self.role_content or self.input_query
+        else:
+            role_content = self.role_content or self.input_query
+
+        if return_all:
+            return f"{self.role_name}: {role_content}"
+        else:
+            return role_content
+    
+    def is_system_role(self,):
+        return self.role_type == "system"
+    
+    def __str__(self) -> str:
+        # key_str = '\n'.join([k for k, v in vars(self).items()])
+        # logger.debug(f"{key_str}")
+        return "\n".join([": ".join([k, str(v)]) for k, v in vars(self).items()])
+    

dev_opsgpt/connector/shcema/__init__.py

@@ -1,6 +0,0 @@
-from .memory import Memory
-
-
-__all__ = [
-    "Memory"
-]

dev_opsgpt/connector/utils.py

@@ -1,5 +1,30 @@
+import re
 
 
+
+def parse_section(text, section_name):
+    # Define a pattern to extract the named section along with its content
+    section_pattern = rf'#### {section_name}\n(.*?)(?=####|$)'
+    
+    # Find the specific section content
+    section_content = re.search(section_pattern, text, re.DOTALL)
+    
+    if section_content:
+        # If the section is found, extract the content
+        content = section_content.group(1)
+        
+        # Define a pattern to find segments that follow the format **xx:**
+        segments_pattern = r'\*\*([^*]+):\*\*'
+        
+        # Use findall method to extract all matches in the section content
+        segments = re.findall(segments_pattern, content)
+        
+        return segments
+    else:
+        # If the section is not found, return an empty list
+        return []
+    
+
 def prompt_cost(model_type: str, num_prompt_tokens: float, num_completion_tokens: float):
     input_cost_map = {
         "gpt-3.5-turbo": 0.0015,

dev_opsgpt/db_handler/__init__.py

@@ -0,0 +1,7 @@
+# encoding: utf-8
+'''
+@author: 温进
+@file: __init__.py.py
+@time: 2023/11/16 下午3:15
+@desc:
+'''

dev_opsgpt/db_handler/graph_db_handler/__init__.py

@@ -0,0 +1,7 @@
+# encoding: utf-8
+'''
+@author: 温进
+@file: __init__.py.py
+@time: 2023/11/20 下午3:07
+@desc:
+'''

dev_opsgpt/db_handler/graph_db_handler/nebula_handler.py

@@ -0,0 +1,263 @@
+# encoding: utf-8
+'''
+@author: 温进
+@file: nebula_handler.py
+@time: 2023/11/16 下午3:15
+@desc:
+'''
+import time
+from loguru import logger
+
+from nebula3.gclient.net import ConnectionPool
+from nebula3.Config import Config
+
+
+class NebulaHandler:
+    def __init__(self, host: str, port: int, username: str, password: str = '', space_name: str = ''):
+        '''
+        init nebula connection_pool
+        @param host: host
+        @param port: port
+        @param username: username
+        @param password: password
+        '''
+        config = Config()
+
+        self.connection_pool = ConnectionPool()
+        self.connection_pool.init([(host, port)], config)
+        self.username = username
+        self.password = password
+        self.space_name = space_name
+
+    def execute_cypher(self, cypher: str, space_name: str = ''):
+        '''
+
+        @param space_name: space_name, if provided, will execute use space_name first
+        @param cypher:
+        @return:
+        '''
+        with self.connection_pool.session_context(self.username, self.password) as session:
+            if space_name:
+                cypher = f'USE {space_name};{cypher}'
+            logger.debug(cypher)
+            resp = session.execute(cypher)
+
+        return resp
+
+    def close_connection(self):
+        self.connection_pool.close()
+
+    def create_space(self, space_name: str, vid_type: str, comment: str = ''):
+        '''
+        create space
+        @param space_name: cannot startwith number
+        @return:
+        '''
+        cypher = f'CREATE SPACE IF NOT EXISTS {space_name} (vid_type={vid_type}) comment="{comment}";'
+        resp = self.execute_cypher(cypher)
+        return resp
+
+    def show_space(self):
+        cypher = 'SHOW SPACES'
+        resp = self.execute_cypher(cypher)
+        return resp
+
+    def drop_space(self, space_name):
+        cypher = f'DROP SPACE {space_name}'
+        return self.execute_cypher(cypher)
+
+    def create_tag(self, tag_name: str, prop_dict: dict = {}):
+        '''
+        创建 tag
+        @param tag_name: tag 名称
+        @param prop_dict: 属性字典 {'prop 名字': 'prop 类型'}
+        @return:
+        '''
+        cypher = f'CREATE TAG IF NOT EXISTS {tag_name}'
+        cypher += '('
+        for k, v in prop_dict.items():
+            cypher += f'{k} {v},'
+        cypher = cypher.rstrip(',')
+        cypher += ')'
+        cypher += ';'
+
+        res = self.execute_cypher(cypher, self.space_name)
+        return res
+
+    def show_tags(self):
+        '''
+        查看 tag
+        @return:
+        '''
+        cypher = 'SHOW TAGS'
+        resp = self.execute_cypher(cypher, self.space_name)
+        return resp
+
+    def insert_vertex(self, tag_name: str, value_dict: dict):
+        '''
+        insert vertex
+        @param tag_name:
+        @param value_dict: {'properties_name': [], values: {'vid':[]}} order should be the same in properties_name and values
+        @return:
+        '''
+        cypher = f'INSERT VERTEX {tag_name} ('
+
+        properties_name = value_dict['properties_name']
+
+        for property_name in properties_name:
+            cypher += f'{property_name},'
+        cypher = cypher.rstrip(',')
+
+        cypher += ') VALUES '
+
+        for vid, properties in value_dict['values'].items():
+            cypher += f'"{vid}":('
+            for property in properties:
+                if type(property) == str:
+                    cypher += f'"{property}",'
+                else:
+                    cypher += f'{property}'
+            cypher = cypher.rstrip(',')
+            cypher += '),'
+        cypher = cypher.rstrip(',')
+        cypher += ';'
+
+        res = self.execute_cypher(cypher, self.space_name)
+        return res
+
+    def create_edge_type(self, edge_type_name: str, prop_dict: dict = {}):
+        '''
+        创建 tag
+        @param edge_type_name: tag 名称
+        @param prop_dict: 属性字典 {'prop 名字': 'prop 类型'}
+        @return:
+        '''
+        cypher = f'CREATE EDGE IF NOT EXISTS {edge_type_name}'
+
+        cypher += '('
+        for k, v in prop_dict.items():
+            cypher += f'{k} {v},'
+        cypher = cypher.rstrip(',')
+        cypher += ')'
+        cypher += ';'
+
+        res = self.execute_cypher(cypher, self.space_name)
+        return res
+
+    def show_edge_type(self):
+        '''
+        查看 tag
+        @return:
+        '''
+        cypher = 'SHOW EDGES'
+        resp = self.execute_cypher(cypher, self.space_name)
+        return resp
+
+    def drop_edge_type(self, edge_type_name: str):
+        cypher = f'DROP EDGE {edge_type_name}'
+        return self.execute_cypher(cypher, self.space_name)
+
+    def insert_edge(self, edge_type_name: str, value_dict: dict):
+        '''
+        insert edge
+        @param edge_type_name:
+        @param value_dict: value_dict: {'properties_name': [], values: {(src_vid, dst_vid):[]}} order should be the
+                            same in properties_name and values
+        @return:
+        '''
+        cypher = f'INSERT EDGE {edge_type_name} ('
+
+        properties_name = value_dict['properties_name']
+
+        for property_name in properties_name:
+            cypher += f'{property_name},'
+        cypher = cypher.rstrip(',')
+
+        cypher += ') VALUES '
+
+        for (src_vid, dst_vid), properties in value_dict['values'].items():
+            cypher += f'"{src_vid}"->"{dst_vid}":('
+            for property in properties:
+                if type(property) == str:
+                    cypher += f'"{property}",'
+                else:
+                    cypher += f'{property}'
+            cypher = cypher.rstrip(',')
+            cypher += '),'
+        cypher = cypher.rstrip(',')
+        cypher += ';'
+
+        res = self.execute_cypher(cypher, self.space_name)
+        return res
+
+    def set_space_name(self, space_name):
+        self.space_name = space_name
+
+    def add_host(self, host: str, port: str):
+        '''
+        add host
+        @return:
+        '''
+        cypher = f'ADD HOSTS {host}:{port}'
+        res = self.execute_cypher(cypher)
+        return res
+
+    def get_stat(self):
+        '''
+
+        @return:
+        '''
+        submit_cypher = 'SUBMIT JOB STATS;'
+        self.execute_cypher(cypher=submit_cypher, space_name=self.space_name)
+        time.sleep(2)
+
+        stats_cypher = 'SHOW STATS;'
+        stats_res = self.execute_cypher(cypher=stats_cypher, space_name=self.space_name)
+
+        res = {'vertices': -1, 'edges': -1}
+
+        stats_res_dict = self.result_to_dict(stats_res)
+        for idx in range(len(stats_res_dict['Type'])):
+            t = stats_res_dict['Type'][idx].as_string()
+            name = stats_res_dict['Name'][idx].as_string()
+            count = stats_res_dict['Count'][idx].as_int()
+
+            if t == 'Space' and name in res:
+                res[name] = count
+        return res
+
+    def get_vertices(self, tag_name: str = '', limit: int = 10000):
+        '''
+        get all vertices
+        @return:
+        '''
+        if tag_name:
+            cypher = f'''MATCH (v:{tag_name}) RETURN v LIMIT {limit};'''
+        else:
+            cypher = f'MATCH (v) RETURN v LIMIT {limit};'
+
+        res = self.execute_cypher(cypher, self.space_name)
+        return self.result_to_dict(res)
+
+    def result_to_dict(self, result) -> dict:
+        """
+        build list for each column, and transform to dataframe
+        """
+        logger.info(result.error_msg())
+        assert result.is_succeeded()
+        columns = result.keys()
+        d = {}
+        for col_num in range(result.col_size()):
+            col_name = columns[col_num]
+            col_list = result.column_values(col_name)
+            d[col_name] = [x for x in col_list]
+        return d
+
+
+
+
+
+
+
+
+

dev_opsgpt/db_handler/vector_db_handler/__init__.py

@@ -0,0 +1,7 @@
+# encoding: utf-8
+'''
+@author: 温进
+@file: __init__.py.py
+@time: 2023/11/20 下午3:08
+@desc:
+'''

dev_opsgpt/db_handler/vector_db_handler/chroma_handler.py

@@ -0,0 +1,140 @@
+# encoding: utf-8
+'''
+@author: 温进
+@file: chroma_handler.py
+@time: 2023/11/21 下午12:21
+@desc:
+'''
+from loguru import logger
+import chromadb
+
+
+class ChromaHandler:
+    def __init__(self, path: str, collection_name: str = ''):
+        '''
+        init client
+        @param path: path of data
+        @collection_name: name of collection
+        '''
+        self.client = chromadb.PersistentClient(path)
+        self.client.heartbeat()
+
+        if collection_name:
+            self.collection = self.client.get_or_create_collection(name=collection_name)
+
+    def create_collection(self, collection_name: str):
+        '''
+        create collection, if exists, will override
+        @return:
+        '''
+        try:
+            collection = self.client.create_collection(name=collection_name)
+        except Exception as e:
+            return {'result_code': -1, 'msg': f'fail, error={e}'}
+        return {'result_code': 0, 'msg': 'success'}
+
+    def delete_collection(self, collection_name: str):
+        '''
+
+        @param collection_name:
+        @return:
+        '''
+        try:
+            self.client.delete_collection(name=collection_name)
+        except Exception as e:
+            return {'result_code': -1, 'msg': f'fail, error={e}'}
+        return {'result_code': 0, 'msg': 'success'}
+
+    def set_collection(self, collection_name: str):
+        '''
+
+        @param collection_name:
+        @return:
+        '''
+        try:
+            self.collection = self.client.get_collection(collection_name)
+        except Exception as e:
+            return {'result_code': -1, 'msg': f'fail, error={e}'}
+        return {'result_code': 0, 'msg': 'success'}
+
+    def add_data(self, ids: list, documents: list = None, embeddings: list = None, metadatas: list = None):
+        '''
+        add data to chroma
+        @param documents: list of doc string
+        @param embeddings: list of vector
+        @param metadatas: list of metadata
+        @param ids: list of id
+        @return:
+        '''
+        try:
+            self.collection.add(
+                ids=ids,
+                embeddings=embeddings,
+                metadatas=metadatas,
+                documents=documents
+            )
+        except Exception as e:
+            return {'result_code': -1, 'msg': f'fail, error={e}'}
+        return {'result_code': 0, 'msg': 'success'}
+
+    def query(self, query_embeddings=None, query_texts=None, n_results=10, where=None, where_document=None,
+              include=["metadatas", "documents", "distances"]):
+        '''
+
+        @param query_embeddings:
+        @param query_texts:
+        @param n_results:
+        @param where:
+        @param where_document:
+        @param include:
+        @return:
+        '''
+        try:
+            query_result = self.collection.query(query_embeddings=query_embeddings, query_texts=query_texts,
+                                                 n_results=n_results, where=where, where_document=where_document,
+                                                 include=include)
+            return {'result_code': 0, 'msg': 'success', 'result': query_result}
+        except Exception as e:
+            return {'result_code': -1, 'msg': f'fail, error={e}'}
+
+    def get(self, ids=None, where=None, limit=None, offset=None, where_document=None, include=["metadatas", "documents"]):
+        '''
+        get by condition
+        @param ids:
+        @param where:
+        @param limit:
+        @param offset:
+        @param where_document:
+        @param include:
+        @return:
+            '''
+        try:
+            query_result = self.collection.get(ids=ids, where=where, where_document=where_document,
+                                               limit=limit,
+                                               offset=offset, include=include)
+            return {'result_code': 0, 'msg': 'success', 'result': query_result}
+        except Exception as e:
+            return {'result_code': -1, 'msg': f'fail, error={e}'}
+
+    def peek(self, limit: int=10):
+        '''
+        peek
+        @param limit:
+        @return:
+        '''
+        try:
+            query_result = self.collection.peek(limit)
+            return {'result_code': 0, 'msg': 'success', 'result': query_result}
+        except Exception as e:
+            return {'result_code': -1, 'msg': f'fail, error={e}'}
+
+    def count(self):
+        '''
+        count
+        @return:
+        '''
+        try:
+            query_result = self.collection.count()
+            return {'result_code': 0, 'msg': 'success', 'result': query_result}
+        except Exception as e:
+            return {'result_code': -1, 'msg': f'fail, error={e}'}

dev_opsgpt/embeddings/get_embedding.py

@@ -0,0 +1,39 @@
+# encoding: utf-8
+'''
+@author: 温进
+@file: get_embedding.py
+@time: 2023/11/22 上午11:30
+@desc:
+'''
+from loguru import logger
+
+from configs.model_config import EMBEDDING_MODEL
+from dev_opsgpt.embeddings.openai_embedding import OpenAIEmbedding
+from dev_opsgpt.embeddings.huggingface_embedding import HFEmbedding
+
+
+def get_embedding(engine: str, text_list: list):
+    '''
+    get embedding
+    @param engine: openai / hf
+    @param text_list:
+    @return:
+    '''
+    emb_res = {}
+
+    if engine == 'openai':
+        oae = OpenAIEmbedding()
+        emb_res = oae.get_emb(text_list)
+    elif engine == 'model':
+        hfe = HFEmbedding(EMBEDDING_MODEL)
+        emb_res = hfe.get_emb(text_list)
+
+    return emb_res
+
+
+if __name__ == '__main__':
+    engine = 'model'
+    text_list = ['这段代码是一个OkHttp拦截器，用于在请求头中添加授权令牌。它继承自`com.theokanning.openai.client.AuthenticationInterceptor`类，并且被标记为`@Deprecated`，意味着它已经过时了。\n\n这个拦截器的作用是在每个请求的头部添加一个名为"Authorization"的字段，值为传入的授权令牌。这样，当请求被发送到服务器时，服务器可以使用这个令牌来验证请求的合法性。\n\n这段代码的构造函数接受一个令牌作为参数，并将其传递给父类的构造函数。这个令牌应该是一个有效的授权令牌，用于访问受保护的资源。', '这段代码定义了一个接口`OpenAiApi`，并使用`@Deprecated`注解将其标记为已过时。它还扩展了`com.theokanning.openai.client.OpenAiApi`接口。\n\n`@Deprecated`注解表示该接口已经过时，不推荐使用。开发者应该使用`com.theokanning.openai.client.OpenAiApi`接口代替。\n\n注释中提到这个接口只是为了保持向后兼容性。这意味着它可能是为了与旧版本的代码兼容而保留的，但不推荐在新代码中使用。', '这段代码是一个OkHttp的拦截器，用于在请求头中添加授权令牌（authorization token）。\n\n在这个拦截器中，首先获取到传入的授权令牌（token），然后在每个请求的构建过程中，使用`newBuilder()`方法创建一个新的请求构建器，并在该构建器中添加一个名为"Authorization"的请求头，值为"Bearer " + token。最后，使用该构建器构建一个新的请求，并通过`chain.proceed(request)`方法继续处理该请求。\n\n这样，当使用OkHttp发送请求时，该拦截器会自动在请求头中添加授权令牌，以实现身份验证的功能。', '这段代码是一个Java接口，用于定义与OpenAI API进行通信的方法。它包含了各种不同类型的请求和响应方法，用于与OpenAI API的不同端点进行交互。\n\n接口中的方法包括：\n- `listModels()`：获取可用的模型列表。\n- `getModel(String modelId)`：获取指定模型的详细信息。\n- `createCompletion(CompletionRequest request)`：创建文本生成的请求。\n- `createChatCompletion(ChatCompletionRequest request)`：创建聊天式文本生成的请求。\n- `createEdit(EditRequest request)`：创建文本编辑的请求。\n- `createEmbeddings(EmbeddingRequest request)`：创建文本嵌入的请求。\n- `listFiles()`：获取已上传文件的列表。\n- `uploadFile(RequestBody purpose, MultipartBody.Part file)`：上传文件。\n- `deleteFile(String fileId)`：删除文件。\n- `retrieveFile(String fileId)`：获取文件的详细信息。\n- `retrieveFileContent(String fileId)`：获取文件的内容。\n- `createFineTuningJob(FineTuningJobRequest request)`：创建Fine-Tuning任务。\n- `listFineTuningJobs()`：获取Fine-Tuning任务的列表。\n- `retrieveFineTuningJob(String fineTuningJobId)`：获取指定Fine-Tuning任务的详细信息。\n- `cancelFineTuningJob(String fineTuningJobId)`：取消Fine-Tuning任务。\n- `listFineTuningJobEvents(String fineTuningJobId)`：获取Fine-Tuning任务的事件列表。\n- `createFineTuneCompletion(CompletionRequest request)`：创建Fine-Tuning模型的文本生成请求。\n- `createImage(CreateImageRequest request)`：创建图像生成的请求。\n- `createImageEdit(RequestBody requestBody)`：创建图像编辑的请求。\n- `createImageVariation(RequestBody requestBody)`：创建图像变体的请求。\n- `createTranscription(RequestBody requestBody)`：创建音频转录的请求。\n- `createTranslation(RequestBody requestBody)`：创建音频翻译的请求。\n- `createModeration(ModerationRequest request)`：创建内容审核的请求。\n- `getEngines()`：获取可用的引擎列表。\n- `getEngine(String engineId)`：获取指定引擎的详细信息。\n- `subscription()`：获取账户订阅信息。\n- `billingUsage(LocalDate starDate, LocalDate endDate)`：获取账户消费信息。\n\n这些方法使用不同的HTTP请求类型（GET、POST、DELETE）和路径来与OpenAI API进行交互，并返回相应的响应数据。']
+
+    res = get_embedding(engine, text_list)
+    logger.debug(res)

dev_opsgpt/embeddings/huggingface_embedding.py

@@ -0,0 +1,49 @@
+# encoding: utf-8
+'''
+@author: 温进
+@file: huggingface_embedding.py
+@time: 2023/11/30 上午11:41
+@desc:
+'''
+from loguru import logger
+from configs.model_config import EMBEDDING_DEVICE
+from dev_opsgpt.embeddings.utils import load_embeddings
+
+
+class HFEmbedding:
+    _instance = {}
+
+    def __new__(cls, *args, **kwargs):
+        instance_key = f'{args},{kwargs}'
+
+        if cls._instance.get(instance_key, None):
+            return cls._instance[instance_key]
+        else:
+            cls._instance[instance_key] = super().__new__(cls)
+            return cls._instance[instance_key]
+
+    def __init__(self, model_name):
+        self.model = load_embeddings(model=model_name, device=EMBEDDING_DEVICE)
+        logger.debug('load success')
+
+    def get_emb(self, text_list):
+        '''
+        get embedding
+        @param text_list:
+        @return:
+        '''
+        logger.info('st')
+        emb_res = self.model.embed_documents(text_list)
+        logger.info('ed')
+        res = {
+            text_list[idx]: emb_res[idx] for idx in range(len(text_list))
+        }
+        return res
+
+
+if __name__ == '__main__':
+    model_name = 'text2vec-base'
+    hfe = HFEmbedding(model_name)
+    text_list = ['这段代码是一个OkHttp拦截器，用于在请求头中添加授权令牌。它继承自`com.theokanning.openai.client.AuthenticationInterceptor`类，并且被标记为`@Deprecated`，意味着它已经过时了。\n\n这个拦截器的作用是在每个请求的头部添加一个名为"Authorization"的字段，值为传入的授权令牌。这样，当请求被发送到服务器时，服务器可以使用这个令牌来验证请求的合法性。\n\n这段代码的构造函数接受一个令牌作为参数，并将其传递给父类的构造函数。这个令牌应该是一个有效的授权令牌，用于访问受保护的资源。', '这段代码定义了一个接口`OpenAiApi`，并使用`@Deprecated`注解将其标记为已过时。它还扩展了`com.theokanning.openai.client.OpenAiApi`接口。\n\n`@Deprecated`注解表示该接口已经过时，不推荐使用。开发者应该使用`com.theokanning.openai.client.OpenAiApi`接口代替。\n\n注释中提到这个接口只是为了保持向后兼容性。这意味着它可能是为了与旧版本的代码兼容而保留的，但不推荐在新代码中使用。', '这段代码是一个OkHttp的拦截器，用于在请求头中添加授权令牌（authorization token）。\n\n在这个拦截器中，首先获取到传入的授权令牌（token），然后在每个请求的构建过程中，使用`newBuilder()`方法创建一个新的请求构建器，并在该构建器中添加一个名为"Authorization"的请求头，值为"Bearer " + token。最后，使用该构建器构建一个新的请求，并通过`chain.proceed(request)`方法继续处理该请求。\n\n这样，当使用OkHttp发送请求时，该拦截器会自动在请求头中添加授权令牌，以实现身份验证的功能。', '这段代码是一个Java接口，用于定义与OpenAI API进行通信的方法。它包含了各种不同类型的请求和响应方法，用于与OpenAI API的不同端点进行交互。\n\n接口中的方法包括：\n- `listModels()`：获取可用的模型列表。\n- `getModel(String modelId)`：获取指定模型的详细信息。\n- `createCompletion(CompletionRequest request)`：创建文本生成的请求。\n- `createChatCompletion(ChatCompletionRequest request)`：创建聊天式文本生成的请求。\n- `createEdit(EditRequest request)`：创建文本编辑的请求。\n- `createEmbeddings(EmbeddingRequest request)`：创建文本嵌入的请求。\n- `listFiles()`：获取已上传文件的列表。\n- `uploadFile(RequestBody purpose, MultipartBody.Part file)`：上传文件。\n- `deleteFile(String fileId)`：删除文件。\n- `retrieveFile(String fileId)`：获取文件的详细信息。\n- `retrieveFileContent(String fileId)`：获取文件的内容。\n- `createFineTuningJob(FineTuningJobRequest request)`：创建Fine-Tuning任务。\n- `listFineTuningJobs()`：获取Fine-Tuning任务的列表。\n- `retrieveFineTuningJob(String fineTuningJobId)`：获取指定Fine-Tuning任务的详细信息。\n- `cancelFineTuningJob(String fineTuningJobId)`：取消Fine-Tuning任务。\n- `listFineTuningJobEvents(String fineTuningJobId)`：获取Fine-Tuning任务的事件列表。\n- `createFineTuneCompletion(CompletionRequest request)`：创建Fine-Tuning模型的文本生成请求。\n- `createImage(CreateImageRequest request)`：创建图像生成的请求。\n- `createImageEdit(RequestBody requestBody)`：创建图像编辑的请求。\n- `createImageVariation(RequestBody requestBody)`：创建图像变体的请求。\n- `createTranscription(RequestBody requestBody)`：创建音频转录的请求。\n- `createTranslation(RequestBody requestBody)`：创建音频翻译的请求。\n- `createModeration(ModerationRequest request)`：创建内容审核的请求。\n- `getEngines()`：获取可用的引擎列表。\n- `getEngine(String engineId)`：获取指定引擎的详细信息。\n- `subscription()`：获取账户订阅信息。\n- `billingUsage(LocalDate starDate, LocalDate endDate)`：获取账户消费信息。\n\n这些方法使用不同的HTTP请求类型（GET、POST、DELETE）和路径来与OpenAI API进行交互，并返回相应的响应数据。']
+
+    hfe.get_emb(text_list)

dev_opsgpt/embeddings/openai_embedding.py

@@ -0,0 +1,50 @@
+# encoding: utf-8
+'''
+@author: 温进
+@file: openai_embedding.py
+@time: 2023/11/22 上午10:45
+@desc:
+'''
+import openai
+import base64
+import json
+import os
+from loguru import logger
+
+from configs.model_config import OPENAI_API_BASE
+
+
+class OpenAIEmbedding:
+    def __init__(self):
+        pass
+
+    def get_emb(self, text_list):
+        openai.api_key = os.environ["OPENAI_API_KEY"]
+        openai.api_base = os.environ["API_BASE_URL"]
+
+        # change , to ，to avoid bug
+        modified_text_list = [i.replace(',', '，') for i in text_list]
+
+        emb_all_result = openai.Embedding.create(
+            model="text-embedding-ada-002",
+            input=modified_text_list
+        )
+
+        res = {}
+        # logger.debug(emb_all_result)
+        logger.debug(f'len of result={len(emb_all_result["data"])}')
+        for emb_result in emb_all_result['data']:
+            index = emb_result['index']
+            # logger.debug(index)
+            text = text_list[index]
+            emb = emb_result['embedding']
+            res[text] = emb
+
+        return res
+
+
+if __name__ == '__main__':
+    oae = OpenAIEmbedding()
+    res = oae.get_emb(text_list=['这段代码是一个OkHttp拦截器，用于在请求头中添加授权令牌。它继承自`com.theokanning.openai.client.AuthenticationInterceptor`类，并且被标记为`@Deprecated`，意味着它已经过时了。\n\n这个拦截器的作用是在每个请求的头部添加一个名为"Authorization"的字段，值为传入的授权令牌。这样，当请求被发送到服务器时，服务器可以使用这个令牌来验证请求的合法性。\n\n这段代码的构造函数接受一个令牌作为参数，并将其传递给父类的构造函数。这个令牌应该是一个有效的授权令牌，用于访问受保护的资源。', '这段代码定义了一个接口`OpenAiApi`，并使用`@Deprecated`注解将其标记为已过时。它还扩展了`com.theokanning.openai.client.OpenAiApi`接口。\n\n`@Deprecated`注解表示该接口已经过时，不推荐使用。开发者应该使用`com.theokanning.openai.client.OpenAiApi`接口代替。\n\n注释中提到这个接口只是为了保持向后兼容性。这意味着它可能是为了与旧版本的代码兼容而保留的，但不推荐在新代码中使用。', '这段代码是一个OkHttp的拦截器，用于在请求头中添加授权令牌（authorization token）。\n\n在这个拦截器中，首先获取到传入的授权令牌（token），然后在每个请求的构建过程中，使用`newBuilder()`方法创建一个新的请求构建器，并在该构建器中添加一个名为"Authorization"的请求头，值为"Bearer " + token。最后，使用该构建器构建一个新的请求，并通过`chain.proceed(request)`方法继续处理该请求。\n\n这样，当使用OkHttp发送请求时，该拦截器会自动在请求头中添加授权令牌，以实现身份验证的功能。', '这段代码是一个Java接口，用于定义与OpenAI API进行通信的方法。它包含了各种不同类型的请求和响应方法，用于与OpenAI API的不同端点进行交互。\n\n接口中的方法包括：\n- `listModels()`：获取可用的模型列表。\n- `getModel(String modelId)`：获取指定模型的详细信息。\n- `createCompletion(CompletionRequest request)`：创建文本生成的请求。\n- `createChatCompletion(ChatCompletionRequest request)`：创建聊天式文本生成的请求。\n- `createEdit(EditRequest request)`：创建文本编辑的请求。\n- `createEmbeddings(EmbeddingRequest request)`：创建文本嵌入的请求。\n- `listFiles()`：获取已上传文件的列表。\n- `uploadFile(RequestBody purpose, MultipartBody.Part file)`：上传文件。\n- `deleteFile(String fileId)`：删除文件。\n- `retrieveFile(String fileId)`：获取文件的详细信息。\n- `retrieveFileContent(String fileId)`：获取文件的内容。\n- `createFineTuningJob(FineTuningJobRequest request)`：创建Fine-Tuning任务。\n- `listFineTuningJobs()`：获取Fine-Tuning任务的列表。\n- `retrieveFineTuningJob(String fineTuningJobId)`：获取指定Fine-Tuning任务的详细信息。\n- `cancelFineTuningJob(String fineTuningJobId)`：取消Fine-Tuning任务。\n- `listFineTuningJobEvents(String fineTuningJobId)`：获取Fine-Tuning任务的事件列表。\n- `createFineTuneCompletion(CompletionRequest request)`：创建Fine-Tuning模型的文本生成请求。\n- `createImage(CreateImageRequest request)`：创建图像生成的请求。\n- `createImageEdit(RequestBody requestBody)`：创建图像编辑的请求。\n- `createImageVariation(RequestBody requestBody)`：创建图像变体的请求。\n- `createTranscription(RequestBody requestBody)`：创建音频转录的请求。\n- `createTranslation(RequestBody requestBody)`：创建音频翻译的请求。\n- `createModeration(ModerationRequest request)`：创建内容审核的请求。\n- `getEngines()`：获取可用的引擎列表。\n- `getEngine(String engineId)`：获取指定引擎的详细信息。\n- `subscription()`：获取账户订阅信息。\n- `billingUsage(LocalDate starDate, LocalDate endDate)`：获取账户消费信息。\n\n这些方法使用不同的HTTP请求类型（GET、POST、DELETE）和路径来与OpenAI API进行交互，并返回相应的响应数据。'])
+    # res = oae.get_emb(text_list=['''test1"test2test3''', '''test4test5test6'''])
+    print(res)

dev_opsgpt/embeddings/utils.py

@@ -9,4 +9,6 @@ def load_embeddings(model: str, device: str):
     logger.info("load embedding model: {}, {}".format(model, embedding_model_dict[model]))
     embeddings = HuggingFaceEmbeddings(model_name=embedding_model_dict[model],
                                        model_kwargs={'device': device})
-    return embeddings
+    return embeddings
+
+

dev_opsgpt/orm/commands/code_base_cds.py

@@ -11,12 +11,14 @@ from dev_opsgpt.orm.schemas.base_schema import CodeBaseSchema
 
 
 @with_session
-def add_cb_to_db(session, code_name, code_path, code_graph_node_num, code_file_num):
+def add_cb_to_db(session, code_name, code_path, code_graph_node_num, code_file_num, do_interpret):
+    do_interpret = 'YES' if do_interpret else 'NO'
+
     # 增：创建知识库实例
     cb = session.query(CodeBaseSchema).filter_by(code_name=code_name).first()
     if not cb:
         cb = CodeBaseSchema(code_name=code_name, code_path=code_path, code_graph_node_num=code_graph_node_num,
-                            code_file_num=code_file_num)
+                            code_file_num=code_file_num, do_interpret=do_interpret)
         session.add(cb)
     else:
         cb.code_path = code_path
@@ -47,10 +49,10 @@ def cb_exists(session, code_name):
 def load_cb_from_db(session, code_name):
     cb = session.query(CodeBaseSchema).filter_by(code_name=code_name).first()
     if cb:
-        code_name, code_path, code_graph_node_num = cb.code_name, cb.code_path, cb.code_graph_node_num
+        code_name, code_path, code_graph_node_num, do_interpret = cb.code_name, cb.code_path, cb.code_graph_node_num, cb.do_interpret
     else:
-        code_name, code_path, code_graph_node_num = None, None, None
-    return code_name, code_path, code_graph_node_num
+        code_name, code_path, code_graph_node_num = None, None, None, None
+    return code_name, code_path, code_graph_node_num, do_interpret
 
 
 @with_session
@@ -71,7 +73,8 @@ def get_cb_detail(session, code_name: str) -> dict:
             "code_name": cb.code_name,
             "code_path": cb.code_path,
             "code_graph_node_num": cb.code_graph_node_num,
-            'code_file_num': cb.code_file_num
+            'code_file_num': cb.code_file_num,
+            'do_interpret': cb.do_interpret
         }
     else:
         return {

dev_opsgpt/orm/schemas/base_schema.py

@@ -58,6 +58,7 @@ class CodeBaseSchema(Base):
     code_path = Column(String, comment='代码本地路径')
     code_graph_node_num = Column(String, comment='代码图谱节点数')
     code_file_num = Column(String, comment='代码解析文件数')
+    do_interpret = Column(String, comment='是否代码解读，Y or N')
     create_time = Column(DateTime, default=func.now(), comment='创建时间')
 
     def __repr__(self):
@@ -65,5 +66,6 @@ class CodeBaseSchema(Base):
         code_name='{self.code_name}', 
         code_path='{self.code_path}', 
         code_graph_node_num='{self.code_graph_node_num}',
-        code_file_num='{self.code_file_num}'
+        code_file_num='{self.code_file_num}',
+        do_interpret='{self.do_interpret}',
         create_time='{self.create_time}')>"""

dev_opsgpt/sandbox/pycodebox.py

@@ -92,11 +92,11 @@ class PyCodeBox(BaseBox):
                     "metadata": {},
                     "content": {
                         "code": code_text,
-                        "silent": True,
-                        "store_history": True,
+                        "silent": False, # True，则内核会执行代码，但不会发送执行结果（如输出）
+                        "store_history": True, # True，则执行的代码会被记录在交互式环境的历史记录中
                         "user_expressions": {},
-                        "allow_stdin": False,
-                        "stop_on_error": True,
+                        "allow_stdin": False, # True，允许代码执行时接受用户输入
+                        "stop_on_error": True, # True，当执行中遇到错误时，后续代码将不会继续执行。
                     },
                     "channel": "shell",
                     "buffers": [],
@@ -163,7 +163,7 @@ class PyCodeBox(BaseBox):
                 return CodeBoxResponse(
                         code_exe_type="text",
                         code_text=code_text,
-                        code_exe_response=result or "Code run successfully (no output)，可能没有打印需要确认的变量",
+                        code_exe_response=result or "Code run successfully (no output)",
                         code_exe_status=200,
                         do_code_exe=self.do_code_exe
                     )

dev_opsgpt/service/cb_api.py

@@ -23,7 +23,9 @@ from configs.model_config import (
     CB_ROOT_PATH
 )
 
-from dev_opsgpt.codebase_handler.codebase_handler import CodeBaseHandler
+# from dev_opsgpt.codebase_handler.codebase_handler import CodeBaseHandler
+
+from dev_opsgpt.codechat.codebase_handler.codebase_handler import CodeBaseHandler
 
 from loguru import logger
 
@@ -33,10 +35,12 @@ async def list_cbs():
     return ListResponse(data=list_cbs_from_db())
 
 
-async def create_cb(cb_name: str = Body(..., examples=["samples"]),
-                    code_path: str = Body(..., examples=["samples"])
+async def create_cb(zip_file,
+                    cb_name: str = Body(..., examples=["samples"]),
+                    code_path: str = Body(..., examples=["samples"]),
+                    do_interpret: bool = Body(..., examples=["samples"])
                     ) -> BaseResponse:
-    logger.info('cb_name={}, zip_path={}'.format(cb_name, code_path))
+    logger.info('cb_name={}, zip_path={}, do_interpret={}'.format(cb_name, code_path, do_interpret))
 
     # Create selected knowledge base
     if not validate_kb_name(cb_name):
@@ -50,17 +54,16 @@ async def create_cb(cb_name: str = Body(..., examples=["samples"]),
 
     try:
         logger.info('start build code base')
-        cbh = CodeBaseHandler(cb_name, code_path, cb_root_path=CB_ROOT_PATH)
-        cbh.import_code(do_save=True)
-        code_graph_node_num = len(cbh.nh)
-        code_file_num = len(cbh.lcdh)
+        cbh = CodeBaseHandler(cb_name, code_path)
+        vertices_num, edge_num, file_num = cbh.import_code(zip_file=zip_file, do_interpret=do_interpret)
         logger.info('build code base done')
 
         # create cb to table
-        add_cb_to_db(cb_name, cbh.code_path, code_graph_node_num, code_file_num)
+        add_cb_to_db(cb_name, cbh.code_path, vertices_num, file_num, do_interpret)
         logger.info('add cb to mysql table success')
     except Exception as e:
         print(e)
+        logger.exception(e)
         return BaseResponse(code=500, msg=f"创建代码知识库出错： {e}")
 
     return BaseResponse(code=200, msg=f"已新增代码知识库 {cb_name}")
@@ -81,6 +84,11 @@ async def delete_cb(cb_name: str = Body(..., examples=["samples"])) -> BaseRespo
 
             # delete local file
             shutil.rmtree(CB_ROOT_PATH + os.sep + cb_name)
+
+            # delete from codebase
+            cbh = CodeBaseHandler(cb_name)
+            cbh.delete_codebase(codebase_name=cb_name)
+
         except Exception as e:
             print(e)
             return BaseResponse(code=500, msg=f"删除代码知识库出错： {e}")
@@ -91,24 +99,25 @@ async def delete_cb(cb_name: str = Body(..., examples=["samples"])) -> BaseRespo
 def search_code(cb_name: str = Body(..., examples=["sofaboot"]),
                 query: str = Body(..., examples=['你好']),
                 code_limit: int = Body(..., examples=['1']),
+                search_type: str = Body(..., examples=['你好']),
                 history_node_list: list = Body(...)) -> dict:
 
     logger.info('cb_name={}'.format(cb_name))
     logger.info('query={}'.format(query))
     logger.info('code_limit={}'.format(code_limit))
+    logger.info('search_type={}'.format(search_type))
     logger.info('history_node_list={}'.format(history_node_list))
 
     try:
         # load codebase
-        cbh = CodeBaseHandler(code_name=cb_name, cb_root_path=CB_ROOT_PATH)
-        cbh.import_code(do_load=True)
+        cbh = CodeBaseHandler(codebase_name=cb_name)
 
         # search code
-        related_code, related_node = cbh.search_code(query, code_limit=code_limit, history_node_list=history_node_list)
+        context, related_vertices = cbh.search_code(query, search_type=search_type, limit=code_limit)
 
         res = {
-            'related_code': related_code,
-            'related_node': related_node
+            'context': context,
+            'related_vertices': related_vertices
         }
         return res
     except Exception as e:

dev_opsgpt/service/sdfile_api.py

@@ -12,7 +12,7 @@ from fastapi.middleware.cors import CORSMiddleware
 from fastapi.responses import StreamingResponse, FileResponse
 from fastapi import File, UploadFile
 
-from dev_opsgpt.utils.server_utils import BaseResponse, ListResponse
+from dev_opsgpt.utils.server_utils import BaseResponse, ListResponse, DataResponse
 from configs.server_config import OPEN_CROSS_DOMAIN, SDFILE_API_SERVER
 from configs.model_config import (
     JUPYTER_WORK_PATH
@@ -35,7 +35,8 @@ async def sd_upload_file(file: UploadFile = File(...), work_dir: str = JUPYTER_W
 async def sd_download_file(filename: str, save_filename: str = "filename_to_download.ext", work_dir: str = JUPYTER_WORK_PATH):
     # 从服务器下载文件
     logger.debug(f"{os.path.join(work_dir, filename)}")
-    return {"data": FileResponse(os.path.join(work_dir, filename), filename=save_filename)}
+    return {"data": os.path.join(work_dir, filename), "filename": save_filename}
+    # return {"data": FileResponse(os.path.join(work_dir, filename), filename=save_filename)}
 
 
 async def sd_list_files(work_dir: str = JUPYTER_WORK_PATH):
@@ -78,7 +79,7 @@ def create_app():
     
     app.get("/sdfiles/download",
             tags=["files upload and download"],
-            response_model=BaseResponse,
+            response_model=DataResponse,
             summary="从沙盒下载文件"
             )(sd_download_file)
 

dev_opsgpt/tools/__init__.py

@@ -1,3 +1,5 @@
+import  importlib
+
 from .base_tool import toLangchainTools, get_tool_schema, BaseToolModel
 from .weather import WeatherInfo, DistrictInfo
 from .multiplier import Multiplier
@@ -7,27 +9,22 @@ from .metrics_query import MetricsQuery
 from .duckduckgo_search import DDGSTool
 from .docs_retrieval import DocRetrieval
 from .cb_query_tool import CodeRetrieval 
+from .ocr_tool import BaiduOcrTool
+from .stock_tool  import StockInfo, StockName
 
-TOOL_SETS = [
-    "WeatherInfo", "WorldTimeGetTimezoneByArea", "Multiplier", "DistrictInfo", "KSigmaDetector", "MetricsQuery", "DDGSTool",
-    "DocRetrieval", "CodeRetrieval"
-    ]
 
-TOOL_DICT = {
-    "WeatherInfo": WeatherInfo,
-    "WorldTimeGetTimezoneByArea": WorldTimeGetTimezoneByArea,
-    "Multiplier": Multiplier,
-    "DistrictInfo": DistrictInfo,
-    "KSigmaDetector": KSigmaDetector,
-    "MetricsQuery": MetricsQuery,
-    "DDGSTool": DDGSTool,
-    "DocRetrieval": DocRetrieval,
-    "CodeRetrieval": CodeRetrieval
-}
-
-__all__ = [
-    "WeatherInfo", "WorldTimeGetTimezoneByArea", "Multiplier", "DistrictInfo", "KSigmaDetector", "MetricsQuery", "DDGSTool",
-    "DocRetrieval", "CodeRetrieval",
-    "toLangchainTools", "get_tool_schema",  "tool_sets", "BaseToolModel"
+IMPORT_TOOL = [
+    WeatherInfo, DistrictInfo, Multiplier, WorldTimeGetTimezoneByArea,
+    KSigmaDetector, MetricsQuery, DDGSTool, DocRetrieval, CodeRetrieval,
+    BaiduOcrTool, StockInfo, StockName
 ]
 
+TOOL_SETS = [tool.__name__ for tool in IMPORT_TOOL]
+
+TOOL_DICT = {tool.__name__: tool for tool in IMPORT_TOOL}
+
+
+__all__ = [
+    "toLangchainTools", "get_tool_schema",  "tool_sets", "BaseToolModel"
+] + TOOL_SETS
+

dev_opsgpt/tools/cb_query_tool.py

@@ -35,13 +35,26 @@ class CodeRetrieval(BaseToolModel):
         code: str  = Field(..., description="检索代码")
 
     @classmethod
-    def run(cls, code_base_name, query, code_limit=CODE_SEARCH_TOP_K, history_node_list=[]):
+    def run(cls, code_base_name, query, code_limit=CODE_SEARCH_TOP_K, history_node_list=[], search_type="tag"):
         """excute your tool!"""
-        codes = search_code(code_base_name, query, code_limit, history_node_list=history_node_list)
-        return_codes = []
-        related_code = codes['related_code']
-        related_nodes = codes['related_node']
+        
+        search_type = {
+            '基于 cypher': 'cypher',
+            '基于标签': 'tag',
+            '基于描述': 'description',
+            'tag': 'tag',
+            'description': 'description',
+            'cypher': 'cypher'
+        }.get(search_type, 'tag')
+
+        # default
+        codes = search_code(code_base_name, query, code_limit, search_type=search_type, history_node_list=history_node_list)
+        return_codes = []
+        context = codes['context']
+        related_nodes = codes['related_vertices']
+        logger.debug(f"{code_base_name}, {query}, {code_limit}, {search_type}")
+        logger.debug(f"context: {context}, related_nodes: {related_nodes}")
+
+        return_codes.append({'index': 0, 'code': context, "related_nodes": related_nodes})
 
-        for idx, code in enumerate(related_code):
-            return_codes.append({'index': idx, 'code': code, "related_nodes": related_nodes})
         return return_codes

dev_opsgpt/tools/multiplier.py

@@ -32,7 +32,4 @@ class Multiplier(BaseToolModel):
     
     @staticmethod
     def run(a, b):
-        return a * b
-    
-def multi_run(a, b):
-    return a * b
+        return a * b

dev_opsgpt/tools/ocr_tool.py

@@ -0,0 +1,96 @@
+from pydantic import BaseModel, Field
+from typing import List, Dict
+import requests
+import base64
+import urllib
+import os
+from loguru import logger
+from .base_tool import BaseToolModel
+
+from configs.model_config import JUPYTER_WORK_PATH
+
+
+class BaiduOcrTool(BaseToolModel):
+    """
+    Tips:
+        百度ocr tool
+    
+    example：
+        API_KEY = ""
+        SECRET_KEY = ""
+        image_path = ''
+        ocr_result = BaiduOcrTool.run(API_KEY=API_KEY , SECRET_KEY=SECRET_KEY, image_path=image_path)
+    """
+
+    name: str = "Baidu_orc_tool"
+    description: str = """ 百度OCR手写字符识别调用器。 输入一张图片位置，返回图片中的文本"""
+
+    class ToolInputArgs(BaseModel):
+        """Input for Multiplier."""
+        image_name : str = Field(..., description="待提取文本信息的图片名称")
+
+    class ToolOutputArgs(BaseModel):
+        """Output for Multiplier."""
+
+        ocr_result: str = Field(..., description="OCR分析提取的自然语言文本")
+
+    @classmethod
+    def ocr_baidu_main(cls, API_KEY, SECRET_KEY, image_path):
+        '''
+        根据图片地址，返回OCR识别结果
+        OCR的结果不仅包含了文字，也包含了文字的位置。但可以根据简单的提取方法，只将文字提前取出来
+        下面是ocr的返回结果
+        '{"words_result":[{"location":{"top":17,"left":33,"width":227,"height":24},"words":"手写识别测试图片样例:"},
+                        {"location":{"top":91,"left":190,"width":713,"height":70},"words":"每一个人的生命中,都应该有一次,"},
+                        {"location":{"top":177,"left":87,"width":831,"height":65},"words":"为了某个人而忘了自己,不求有结果."},
+                        {"location":{"top":263,"left":80,"width":842,"height":76},"words":"不求同行,不求曾经拥有,甚至不求"}],
+                        "words_result_num":4,"log_id":1722502064951792680}'
+        '''
+        url = "https://aip.baidubce.com/rest/2.0/ocr/v1/handwriting?access_token=" + BaiduOcrTool.get_access_token(API_KEY, SECRET_KEY)
+        
+        # image 可以通过 get_file_content_as_base64("C:\fakepath\ocr_input_example.png",True) 方法获取
+        image = BaiduOcrTool.get_file_content_as_base64(image_path, True)
+        payload = 'image=' + image + '&detect_direction=false&probability=false'
+        headers = {
+            'Content-Type': 'application/x-www-form-urlencoded',
+            'Accept': 'application/json'
+        }
+        
+        response = requests.request("POST", url, headers=headers, data=payload)
+        s = ""
+        try:
+            for word_result in response.json()["words_result"]:
+                s += "\n" + word_result["words"]
+        except Exception as e:
+            logger.exception(e)
+            s = "无法识别图片内容"
+        return s
+        
+    @classmethod
+    def get_file_content_as_base64(cls, image_path, urlencoded=False):
+        """
+        获取文件base64编码
+        :param path: 文件路径
+        :param urlencoded: 是否对结果进行urlencoded 
+        :return: base64编码信息
+        """
+        with open(image_path, "rb") as f:
+            content = base64.b64encode(f.read()).decode("utf8")
+            if urlencoded:
+                content = urllib.parse.quote_plus(content)
+        return content
+    
+    @classmethod
+    def get_access_token(cls, API_KEY, SECRET_KEY):
+        """
+        使用 AK，SK 生成鉴权签名（Access Token）
+        :return: access_token，或是None(如果错误)
+        """
+        url = "https://aip.baidubce.com/oauth/2.0/token"
+        params = {"grant_type": "client_credentials", "client_id": API_KEY, "client_secret": SECRET_KEY}
+        return str(requests.post(url, params=params).json().get("access_token"))
+    
+    @classmethod
+    def run(cls, image_name, image_path=JUPYTER_WORK_PATH, API_KEY=os.environ.get("BAIDU_OCR_API_KEY"), SECRET_KEY=os.environ.get("BAIDU_OCR_SECRET_KEY")):
+        image_file = os.path.join(image_path, image_name)
+        return cls.ocr_baidu_main(API_KEY, SECRET_KEY, image_file)

dev_opsgpt/tools/stock_tool.py

@@ -0,0 +1,189 @@
+
+import json
+import os
+import re
+from pydantic import BaseModel, Field
+from typing import List, Dict, Optional
+import requests
+import numpy as np
+from loguru import logger
+
+from .base_tool import BaseToolModel
+from dev_opsgpt.utils.common_utils import read_json_file
+
+
+cur_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+stock_infos = read_json_file(os.path.join(cur_dir, "../sources/tool_datas/stock.json"))
+stock_dict = {i["mc"]: i["jys"]+i["dm"] for i in stock_infos}
+
+
+class StockName(BaseToolModel):
+    """
+    Tips
+    """
+    name: str = "StockName"
+    description: str = "通过股票名称查询股票代码"
+
+    class ToolInputArgs(BaseModel):
+        """Input for StockName"""
+        stock_name: int = Field(..., description="股票名称")
+
+    class ToolOutputArgs(BaseModel):
+        """Output for StockName"""
+        stock_code: str = Field(..., description="股票代码")
+    
+    @staticmethod
+    def run(stock_name: str):
+        return stock_dict.get(stock_name, "no stock_code")
+    
+
+
+class StockInfo(BaseToolModel):
+    """
+    用于查询股票市场数据的StockInfo工具。
+    """
+
+    name: str = "StockInfo"
+    description: str = "根据提供的股票代码、日期范围和数据频率提供股票市场数据。"
+
+    class ToolInputArgs(BaseModel):
+        """StockInfo的输入参数。"""
+        code: str = Field(..., description="要查询的股票代码，格式为'marketcode'")
+        end_date: Optional[str] = Field(default="", description="数据查询的结束日期。留空则为当前日期。如果没有提供结束日期，就留空")
+        count: int = Field(default=10, description="返回数据点的数量。")
+        frequency: str = Field(default='1d', description="数据点的频率，例如，'1d'表示每日，'1w'表示每周，'1M'表示每月，'1m'表示每分钟等。")
+
+    class ToolOutputArgs(BaseModel):
+        """StockInfo的输出参数。"""
+        data: dict = Field(default=None, description="查询到的股票市场数据。")
+
+    @staticmethod
+    def run(code: str, count: int, frequency: str, end_date: Optional[str]="") -> "ToolOutputArgs":
+        """执行股票数据查询工具。"""
+        # 该方法封装了调用底层股票数据API的逻辑，并将结果格式化为pandas DataFrame。
+        try:
+            df = get_price(code, end_date=end_date, count=count, frequency=frequency)
+            # 将DataFrame转换为输出的字典格式
+            data = df.reset_index().to_dict(orient='list')  # 将dataframe转换为字典列表
+            return StockInfo.ToolOutputArgs(data=data)
+        except Exception as e:
+            logger.exception("获取股票数据时发生错误。")
+            return e
+
+#-*- coding:utf-8 -*-    --------------Ashare 股票行情数据双核心版( https://github.com/mpquant/Ashare ) 
+
+import json,requests,datetime
+import pandas as pd  #
+
+#腾讯日线
+
+def get_price_day_tx(code, end_date='', count=10, frequency='1d'):     #日线获取  
+
+    unit='week' if frequency in '1w' else 'month' if frequency in '1M' else 'day'     #判断日线，周线，月线
+
+    if end_date:  end_date=end_date.strftime('%Y-%m-%d') if isinstance(end_date,datetime.date) else end_date.split(' ')[0]
+
+    end_date='' if end_date==datetime.datetime.now().strftime('%Y-%m-%d') else end_date   #如果日期今天就变成空    
+
+    URL=f'http://web.ifzq.gtimg.cn/appstock/app/fqkline/get?param={code},{unit},,{end_date},{count},qfq'     
+
+    st= json.loads(requests.get(URL).content);    ms='qfq'+unit;      stk=st['data'][code]   
+
+    buf=stk[ms] if ms in stk else stk[unit]       #指数返回不是qfqday,是day
+
+    df=pd.DataFrame(buf,columns=['time','open','close','high','low','volume'],dtype='float')     
+
+    df.time=pd.to_datetime(df.time);    df.set_index(['time'], inplace=True);   df.index.name=''          #处理索引 
+
+    return df
+
+
+#腾讯分钟线
+
+def get_price_min_tx(code, end_date=None, count=10, frequency='1d'):    #分钟线获取 
+
+    ts=int(frequency[:-1]) if frequency[:-1].isdigit() else 1           #解析K线周期数
+
+    if end_date: end_date=end_date.strftime('%Y-%m-%d') if isinstance(end_date,datetime.date) else end_date.split(' ')[0]        
+
+    URL=f'http://ifzq.gtimg.cn/appstock/app/kline/mkline?param={code},m{ts},,{count}' 
+
+    st= json.loads(requests.get(URL).content);       buf=st[ 'data'][code]['m'+str(ts)] 
+
+    df=pd.DataFrame(buf,columns=['time','open','close','high','low','volume','n1','n2'])   
+
+    df=df[['time','open','close','high','low','volume']]    
+
+    df[['open','close','high','low','volume']]=df[['open','close','high','low','volume']].astype('float')
+
+    df.time=pd.to_datetime(df.time);   df.set_index(['time'], inplace=True);   df.index.name=''          #处理索引     
+
+    df['close'][-1]=float(st['data'][code]['qt'][code][3])                #最新基金数据是3位的
+
+    return df
+
+
+#sina新浪全周期获取函数，分钟线 5m,15m,30m,60m  日线1d=240m   周线1w=1200m  1月=7200m
+
+def get_price_sina(code, end_date='', count=10, frequency='60m'):    #新浪全周期获取函数    
+
+    frequency=frequency.replace('1d','240m').replace('1w','1200m').replace('1M','7200m');   mcount=count
+
+    ts=int(frequency[:-1]) if frequency[:-1].isdigit() else 1       #解析K线周期数
+
+    if (end_date!='') & (frequency in ['240m','1200m','7200m']): 
+
+        end_date=pd.to_datetime(end_date) if not isinstance(end_date,datetime.date) else end_date    #转换成datetime
+        unit=4 if frequency=='1200m' else 29 if frequency=='7200m' else 1    #4,29多几个数据不影响速度
+
+        count=count+(datetime.datetime.now()-end_date).days//unit            #结束时间到今天有多少天自然日(肯定 >交易日)        
+
+        #print(code,end_date,count)    
+
+    URL=f'http://money.finance.sina.com.cn/quotes_service/api/json_v2.php/CN_MarketData.getKLineData?symbol={code}&scale={ts}&ma=5&datalen={count}' 
+
+    dstr= json.loads(requests.get(URL).content);       
+
+    #df=pd.DataFrame(dstr,columns=['day','open','high','low','close','volume'],dtype='float') 
+
+    df= pd.DataFrame(dstr,columns=['day','open','high','low','close','volume'])
+
+    df['open'] = df['open'].astype(float); df['high'] = df['high'].astype(float);                          #转换数据类型
+    df['low'] = df['low'].astype(float);   df['close'] = df['close'].astype(float);  df['volume'] = df['volume'].astype(float)    
+
+    df.day=pd.to_datetime(df.day);    
+
+    df.set_index(['day'], inplace=True);     
+
+    df.index.name=''            #处理索引                 
+
+    if (end_date!='') & (frequency in ['240m','1200m','7200m']): 
+        return df[df.index<=end_date][-mcount:]   #日线带结束时间先返回              
+
+    return df
+
+
+def get_price(code, end_date='',count=10, frequency='1d', fields=[]):        #对外暴露只有唯一函数，这样对用户才是最友好的  
+
+    xcode= code.replace('.XSHG','').replace('.XSHE','')                      #证券代码编码兼容处理 
+    xcode='sh'+xcode if ('XSHG' in code)  else  'sz'+xcode  if ('XSHE' in code)  else code     
+
+    if  frequency in ['1d','1w','1M']:   #1d日线  1w周线  1M月线
+        try:    
+            return get_price_sina( xcode, end_date=end_date,count=count,frequency=frequency)   #主力
+        except: 
+            return get_price_day_tx(xcode,end_date=end_date,count=count,frequency=frequency)   #备用                    
+
+    if  frequency in ['1m','5m','15m','30m','60m']:  #分钟线 ,1m只有腾讯接口  5分钟5m   60分钟60m
+        if frequency in '1m': 
+            return get_price_min_tx(xcode,end_date=end_date,count=count,frequency=frequency)
+        try:
+            return get_price_sina(xcode,end_date=end_date,count=count,frequency=frequency)   #主力   
+        except: 
+            return get_price_min_tx(xcode,end_date=end_date,count=count,frequency=frequency)   #备用
+
+
+if __name__ == "__main__":
+    tool = StockInfo()
+    output = tool.run(code='sh600519', end_date='', count=10, frequency='15m')
+    print(output.json(indent=2))

dev_opsgpt/utils/common_utils.py

@@ -73,14 +73,14 @@ def save_to_json_file(data, filename):
 
 
 def file_normalize(file: Union[str, Path, bytes], filename=None):
-    logger.debug(f"{file}")
+    # logger.debug(f"{file}")
     if isinstance(file, bytes): # raw bytes
         file = BytesIO(file)
     elif hasattr(file, "read"): # a file io like object
         filename = filename or file.name
     else: # a local path
         file = Path(file).absolute().open("rb")
-        logger.debug(file)
+        # logger.debug(file)
         filename = filename or file.name
     return file, filename