codefuse-chatbot/sources/readme_docs/start-en.md

If you need to deploy a privatized model, please install the NVIDIA driver yourself.

Preparation of Python environment

• It is recommended to use conda to manage the python environment (optional)

# Prepare conda environment
conda create --name Codefusegpt python=3.9
conda activate Codefusegpt

• Install related dependencies

cd Codefuse-ChatBot
pip install -r requirements.txt

Sandbox Environment Preparation

• Windows Docker installation: Docker Desktop for Windows supports 64-bit versions of Windows 10 Pro with Hyper-V enabled (Hyper-V is not required for versions v1903 and above), or 64-bit versions of Windows 10 Home v1903 and above.

  • 【全面详细】Windows10 Docker安装详细教程
  • Docker 从入门到实践
  • Handling 'Docker Desktop requires the Server service to be enabled'
  • 安装wsl或者等报错提示

• Linux Docker installation： Linux installation is relatively simple, please search Baidu/Google for installation guides.

• Mac Docker installation

  • Docker 从入门到实践

# Build the image for the sandbox environment, see above for notebook version issues
bash docker_build.sh

Model Download (Optional)

If you need to use open-source LLM and Embedding models, you can download them from HuggingFace. Here we take THUDM/chatglm2-6b and text2vec-base-chinese as examples:

# install git-lfs
git lfs install

# install LLM-model
git lfs clone https://huggingface.co/THUDM/chatglm2-6b
cp ~/THUDM/chatglm2-6b ~/codefuse-chatbot/llm_models/

# install Embedding-model
git lfs clone https://huggingface.co/shibing624/text2vec-base-chinese
cp ~/shibing624/text2vec-base-chinese ~/codefuse-chatbot/embedding_models/

Basic Configuration

# Modify the basic configuration for service startup
cd configs
cp model_config.py.example model_config.py
cp server_config.py.example server_config.py

# model_config#11~12 If you need to use the OpenAI interface, the OpenAI interface key
os.environ["OPENAI_API_KEY"] = "sk-xxx"
# Replace with the api_base_url you need
os.environ["API_BASE_URL"] = "https://api.openai.com/v1"

# vi model_config#LLM_MODEL The language model you need to choose
LLM_MODEL = "gpt-3.5-turbo"
LLM_MODELs = ["gpt-3.5-turbo"]

# vi model_config#EMBEDDING_MODEL The private vector model you need to choose
EMBEDDING_ENGINE = 'model'
EMBEDDING_MODEL = "text2vec-base"

# Example of vector model access, modify model_config#embedding_model_dict
# If the model directory is:
model_dir: ~/codefuse-chatbot/embedding_models/shibing624/text2vec-base-chinese
# Configure as follows
"text2vec-base": "shibing624/text2vec-base-chinese"


# vi server_config#8~14, It's recommended to use a container to start the service to prevent environment conflicts when installing other dependencies using the codeInterpreter feature
DOCKER_SERVICE = True
# Whether to use a container sandbox
SANDBOX_DO_REMOTE = True

Starting the Service

By default, only the webui-related services are started, and fastchat is not started (optional).

# If you need to support the codellama-34b-int4 model, you need to patch fastchat
# cp examples/gptq.py ~/site-packages/fastchat/modules/gptq.py
# Modify examples/llm_api.py#258 to kwargs={"gptq_wbits": 4},

# start llm-service (optional)
python examples/llm_api.py

For more LLM integration methods, seemore details...

# After completing the server_config.py configuration, you can start with one click
cd examples
python start.py