112 lines
3.9 KiB
Markdown
112 lines
3.9 KiB
Markdown
|
||
If you need to deploy a privatized model, please install the NVIDIA driver yourself.
|
||
|
||
### Preparation of Python environment
|
||
- It is recommended to use conda to manage the python environment (optional)
|
||
```bash
|
||
# Prepare conda environment
|
||
conda create --name Codefusegpt python=3.9
|
||
conda activate Codefusegpt
|
||
```
|
||
|
||
- Install related dependencies
|
||
```bash
|
||
cd Codefuse-ChatBot
|
||
pip install -r requirements.txt
|
||
```
|
||
|
||
### Sandbox Environment Preparation
|
||
- Windows Docker installation:
|
||
[Docker Desktop for Windows](https://docs.docker.com/desktop/install/windows-install/) supports 64-bit versions of Windows 10 Pro with Hyper-V enabled (Hyper-V is not required for versions v1903 and above), or 64-bit versions of Windows 10 Home v1903 and above.
|
||
- [【全面详细】Windows10 Docker安装详细教程](https://zhuanlan.zhihu.com/p/441965046)
|
||
- [Docker 从入门到实践](https://yeasy.gitbook.io/docker_practice/install/windows)
|
||
- [Handling 'Docker Desktop requires the Server service to be enabled'](https://blog.csdn.net/sunhy_csdn/article/details/106526991)
|
||
- [安装wsl或者等报错提示](https://learn.microsoft.com/zh-cn/windows/wsl/install)
|
||
|
||
- Linux Docker installation：
|
||
Linux installation is relatively simple, please search Baidu/Google for installation guides.
|
||
|
||
- Mac Docker installation
|
||
- [Docker 从入门到实践](https://yeasy.gitbook.io/docker_practice/install/mac)
|
||
|
||
```bash
|
||
# Build the image for the sandbox environment, see above for notebook version issues
|
||
bash docker_build.sh
|
||
```
|
||
|
||
### Model Download (Optional)
|
||
|
||
If you need to use open-source LLM and Embedding models, you can download them from HuggingFace.
|
||
Here we take THUDM/chatglm2-6b and text2vec-base-chinese as examples:
|
||
|
||
```
|
||
# install git-lfs
|
||
git lfs install
|
||
|
||
# install LLM-model
|
||
git lfs clone https://huggingface.co/THUDM/chatglm2-6b
|
||
cp ~/THUDM/chatglm2-6b ~/codefuse-chatbot/llm_models/
|
||
|
||
# install Embedding-model
|
||
git lfs clone https://huggingface.co/shibing624/text2vec-base-chinese
|
||
cp ~/shibing624/text2vec-base-chinese ~/codefuse-chatbot/embedding_models/
|
||
```
|
||
|
||
|
||
|
||
### Basic Configuration
|
||
|
||
```bash
|
||
# Modify the basic configuration for service startup
|
||
cd configs
|
||
cp model_config.py.example model_config.py
|
||
cp server_config.py.example server_config.py
|
||
|
||
# model_config#11~12 If you need to use the OpenAI interface, the OpenAI interface key
|
||
os.environ["OPENAI_API_KEY"] = "sk-xxx"
|
||
# Replace with the api_base_url you need
|
||
os.environ["API_BASE_URL"] = "https://api.openai.com/v1"
|
||
|
||
# vi model_config#LLM_MODEL The language model you need to choose
|
||
LLM_MODEL = "gpt-3.5-turbo"
|
||
LLM_MODELs = ["gpt-3.5-turbo"]
|
||
|
||
# vi model_config#EMBEDDING_MODEL The private vector model you need to choose
|
||
EMBEDDING_ENGINE = 'model'
|
||
EMBEDDING_MODEL = "text2vec-base"
|
||
|
||
# Example of vector model access, modify model_config#embedding_model_dict
|
||
# If the model directory is:
|
||
model_dir: ~/codefuse-chatbot/embedding_models/shibing624/text2vec-base-chinese
|
||
# Configure as follows
|
||
"text2vec-base": "shibing624/text2vec-base-chinese"
|
||
|
||
|
||
# vi server_config#8~14, It's recommended to use a container to start the service to prevent environment conflicts when installing other dependencies using the codeInterpreter feature
|
||
DOCKER_SERVICE = True
|
||
# Whether to use a container sandbox
|
||
SANDBOX_DO_REMOTE = True
|
||
```
|
||
|
||
|
||
|
||
### Starting the Service
|
||
|
||
By default, only the webui-related services are started, and fastchat is not started (optional).
|
||
|
||
```bash
|
||
# If you need to support the codellama-34b-int4 model, you need to patch fastchat
|
||
# cp examples/gptq.py ~/site-packages/fastchat/modules/gptq.py
|
||
# Modify examples/llm_api.py#258 to kwargs={"gptq_wbits": 4},
|
||
|
||
# start llm-service (optional)
|
||
python examples/llm_api.py
|
||
```
|
||
For more LLM integration methods, see[more details...](sources/readme_docs/fastchat-en.md)
|
||
<br>
|
||
|
||
```bash
|
||
# After completing the server_config.py configuration, you can start with one click
|
||
cd examples
|
||
python start.py
|
||
``` |