3.9 KiB
If you need to deploy a privatized model, please install the NVIDIA driver yourself.
Preparation of Python environment
- It is recommended to use conda to manage the python environment (optional)
# Prepare conda environment
conda create --name Codefusegpt python=3.9
conda activate Codefusegpt
- Install related dependencies
cd Codefuse-ChatBot
pip install -r requirements.txt
Sandbox Environment Preparation
-
Windows Docker installation: Docker Desktop for Windows supports 64-bit versions of Windows 10 Pro with Hyper-V enabled (Hyper-V is not required for versions v1903 and above), or 64-bit versions of Windows 10 Home v1903 and above.
-
Linux Docker installation： Linux installation is relatively simple, please search Baidu/Google for installation guides.
-
Mac Docker installation
# Build the image for the sandbox environment, see above for notebook version issues
bash docker_build.sh
Model Download (Optional)
If you need to use open-source LLM and Embedding models, you can download them from HuggingFace. Here we take THUDM/chatglm2-6b and text2vec-base-chinese as examples:
# install git-lfs
git lfs install
# install LLM-model
git lfs clone https://huggingface.co/THUDM/chatglm2-6b
cp ~/THUDM/chatglm2-6b ~/codefuse-chatbot/llm_models/
# install Embedding-model
git lfs clone https://huggingface.co/shibing624/text2vec-base-chinese
cp ~/shibing624/text2vec-base-chinese ~/codefuse-chatbot/embedding_models/
Basic Configuration
# Modify the basic configuration for service startup
cd configs
cp model_config.py.example model_config.py
cp server_config.py.example server_config.py
# model_config#11~12 If you need to use the OpenAI interface, the OpenAI interface key
os.environ["OPENAI_API_KEY"] = "sk-xxx"
# Replace with the api_base_url you need
os.environ["API_BASE_URL"] = "https://api.openai.com/v1"
# vi model_config#LLM_MODEL The language model you need to choose
LLM_MODEL = "gpt-3.5-turbo"
LLM_MODELs = ["gpt-3.5-turbo"]
# vi model_config#EMBEDDING_MODEL The private vector model you need to choose
EMBEDDING_ENGINE = 'model'
EMBEDDING_MODEL = "text2vec-base"
# Example of vector model access, modify model_config#embedding_model_dict
# If the model directory is:
model_dir: ~/codefuse-chatbot/embedding_models/shibing624/text2vec-base-chinese
# Configure as follows
"text2vec-base": "shibing624/text2vec-base-chinese"
# vi server_config#8~14, It's recommended to use a container to start the service to prevent environment conflicts when installing other dependencies using the codeInterpreter feature
DOCKER_SERVICE = True
# Whether to use a container sandbox
SANDBOX_DO_REMOTE = True
Starting the Service
By default, only the webui-related services are started, and fastchat is not started (optional).
# If you need to support the codellama-34b-int4 model, you need to patch fastchat
# cp examples/gptq.py ~/site-packages/fastchat/modules/gptq.py
# Modify examples/llm_api.py#258 to kwargs={"gptq_wbits": 4},
# start llm-service (optional)
python examples/llm_api.py
For more LLM integration methods, seemore details...
# After completing the server_config.py configuration, you can start with one click
cd examples
python start.py