用LangChain对一系列文档进行带来源的问答 – LangChain中文网

6大核心模块（Modules)链（Chains)索引示例（Index Examples）问答与来源（Q&A with Sources）

用LangChain对一系列文档进行带来源的问答
本教程介绍如何使用LangChain对一系列文档进行带来源的问答。它涵盖了四种不同的链条类型：
```code
stuff
```
、
```code
map_reduce
```
、
```code
refine
```
、
```code
map-rerank
```
。有关这些链条类型的更深入解释，请参见此处 (opens in a new tab)。
准备数据
首先，我们需要准备数据。在此示例中，我们在向量数据库上进行相似性搜索，但这些文档可以以任何方式获取（本教程的重点是强调在获取文档后要做什么)。

```code
from langchain.embeddings.openai import OpenAIEmbeddings
from langchain.embeddings.cohere import CohereEmbeddings
from langchain.text_splitter import CharacterTextSplitter
from langchain.vectorstores.elastic_vector_search import ElasticVectorSearch
from langchain.vectorstores import Chroma
from langchain.docstore.document import Document
from langchain.prompts import PromptTemplate

```

```code
with open("../../state_of_the_union.txt") as f:
state_of_the_union = f.read()
text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)
texts = text_splitter.split_text(state_of_the_union)

embeddings = OpenAIEmbeddings()

```

```code
docsearch = Chroma.from_texts(texts, embeddings, metadatas=[{"source": str(i)} for i in range(len(texts))])

```

```code
Running Chroma using direct local API.
Using DuckDB in-memory for database. Data will be transient.

```

```code
query = "What did the president say about Justice Breyer"
docs = docsearch.similarity_search(query)

```

```code
from langchain.chains.qa_with_sources import load_qa_with_sources_chain
from langchain.llms import OpenAI

```

快速入门
如果您只想尽快开始，这是推荐的方法：

```code
chain = load_qa_with_sources_chain(OpenAI(temperature=0), chain_type="stuff")
query = "What did the president say about Justice Breyer"
chain({"input_documents": docs, "question": query}, return_only_outputs=True)

```

```code
{'output_text': ' The president thanked Justice Breyer for his service.\nSOURCES: 30-pl'}

```

If you want more control and understanding over what is happening, please see the information below.
The
```code
stuff
```
Chain#
This sections shows results of using the
```code
stuff
```
Chain to do question answering with sources.

```code
chain = load_qa_with_sources_chain(OpenAI(temperature=0), chain_type="stuff")

```

```code
query = "What did the president say about Justice Breyer"
chain({"input_documents": docs, "question": query}, return_only_outputs=True)

```

```code
{'output_text': ' The president thanked Justice Breyer for his service.\nSOURCES: 30-pl'}

```

Custom Prompts
You can also use your own prompts with this chain. In this example, we will respond in Italian.

```code
template = """Given the following extracted parts of a long document and a question, create a final answer with references ("SOURCES").
If you don't know the answer, just say that you don't know. Don't try to make up an answer.
ALWAYS return a "SOURCES" part in your answer.
Respond in Italian.

QUESTION: {question}
=========
{summaries}
=========
FINAL ANSWER IN ITALIAN:"""
PROMPT = PromptTemplate(template=template, input_variables=["summaries", "question"])

chain = load_qa_with_sources_chain(OpenAI(temperature=0), chain_type="stuff", prompt=PROMPT)
query = "What did the president say about Justice Breyer"
chain({"input_documents": docs, "question": query}, return_only_outputs=True)

```

```code
{'output_text': '\nNon so cosa abbia detto il presidente riguardo a Justice Breyer.\nSOURCES: 30, 31, 33'}

```

The
```code
map_reduce
```
Chain#
This sections shows results of using the
```code
map_reduce
```
Chain to do question answering with sources.

```code
chain = load_qa_with_sources_chain(OpenAI(temperature=0), chain_type="map_reduce")

```

```code
query = "What did the president say about Justice Breyer"
chain({"input_documents": docs, "question": query}, return_only_outputs=True)

```

```code
{'output_text': ' The president thanked Justice Breyer for his service.\nSOURCES: 30-pl'}

```

Intermediate Steps
We can also return the intermediate steps for
```code
map_reduce
```
chains, should we want to inspect them. This is done with the
```code
return_intermediate_steps
```
variable.

```code
chain = load_qa_with_sources_chain(OpenAI(temperature=0), chain_type="map_reduce", return_intermediate_steps=True)

```

```code
chain({"input_documents": docs, "question": query}, return_only_outputs=True)

```

```code
{'intermediate_steps': [' "Tonight, I’d like to honor someone who has dedicated his life to serve this country: Justice Stephen Breyer—an Army veteran, Constitutional scholar, and retiring Justice of the United States Supreme Court. Justice Breyer, thank you for your service."',
' None',
' None',
' None'],
'output_text': ' The president thanked Justice Breyer for his service.\nSOURCES: 30-pl'}

```

Custom Prompts
You can also use your own prompts with this chain. In this example, we will respond in Italian.

```code
question_prompt_template = """Use the following portion of a long document to see if any of the text is relevant to answer the question.
Return any relevant text in Italian.
{context}
Question: {question}
Relevant text, if any, in Italian:"""
QUESTION_PROMPT = PromptTemplate(
template=question_prompt_template, input_variables=["context", "question"]
)

combine_prompt_template = """Given the following extracted parts of a long document and a question, create a final answer with references ("SOURCES").
If you don't know the answer, just say that you don't know. Don't try to make up an answer.
ALWAYS return a "SOURCES" part in your answer.
Respond in Italian.

QUESTION: {question}
=========
{summaries}
=========
FINAL ANSWER IN ITALIAN:"""
COMBINE_PROMPT = PromptTemplate(
template=combine_prompt_template, input_variables=["summaries", "question"]
)

chain = load_qa_with_sources_chain(OpenAI(temperature=0), chain_type="map_reduce", return_intermediate_steps=True, question_prompt=QUESTION_PROMPT, combine_prompt=COMBINE_PROMPT)
chain({"input_documents": docs, "question": query}, return_only_outputs=True)

```

```code
{'intermediate_steps': ["\nStasera vorrei onorare qualcuno che ha dedicato la sua vita a servire questo paese: il giustizia Stephen Breyer - un veterano dell'esercito, uno studioso costituzionale e un giustizia in uscita della Corte Suprema degli Stati Uniti. Giustizia Breyer, grazie per il tuo servizio.",
' Non pertinente.',
' Non rilevante.',
" Non c'è testo pertinente."],
'output_text': ' Non conosco la risposta. SOURCES: 30, 31, 33, 20.'}

```

Batch Size
When using the
```code
map_reduce
```
chain, one thing to keep in mind is the batch size you are using during the map step. If this is too high, it could cause rate limiting errors. You can control this by setting the batch size on the LLM used. Note that this only applies for LLMs with this parameter. Below is an example of doing so:

```code
llm = OpenAI(batch_size=5, temperature=0)

```

The
```code
refine
```
Chain#
This sections shows results of using the
```code
refine
```
Chain to do question answering with sources.

```code
chain = load_qa_with_sources_chain(OpenAI(temperature=0), chain_type="refine")

```

```code
query = "What did the president say about Justice Breyer"
chain({"input_documents": docs, "question": query}, return_only_outputs=True)

```

```code
{'output_text': "  The president said that he was honoring Justice Breyer for his dedication to serving the country and that he was a retiring Justice of the United States Supreme Court. He also thanked him for his service and praised his career as a top litigator in private practice, a former federal public defender, and a family of public school educators and police officers. He noted Justice Breyer's reputation as a consensus builder and the broad range of support he has received from the Fraternal Order of Police to former judges appointed by Democrats and Republicans. He also highlighted the importance of securing the border and fixing the immigration system in order to advance liberty and justice, and mentioned the new technology, joint patrols, dedicated immigration judges, and commitments to support partners in South and Central America that have been put in place. He also expressed his commitment to the LGBTQ+ community, noting the need for the bipartisan Equality Act and the importance of protecting transgender Americans from state laws targeting them. He also highlighted his commitment to bipartisanship, noting the 80 bipartisan bills he signed into law last year, and his plans to strengthen the Violence Against Women Act. Additionally, he announced that the Justice Department will name a chief prosecutor for pandemic fraud and his plan to lower the deficit by more than one trillion dollars in a"}

```

Intermediate Steps
We can also return the intermediate steps for
```code
refine
```
chains, should we want to inspect them. This is done with the
```code
return_intermediate_steps
```
variable.

```code
chain = load_qa_with_sources_chain(OpenAI(temperature=0), chain_type="refine", return_intermediate_steps=True)

```

```code
chain({"input_documents": docs, "question": query}, return_only_outputs=True)

```

```code
{'intermediate_steps': ['\nThe president said that he was honoring Justice Breyer for his dedication to serving the country and that he was a retiring Justice of the United States Supreme Court. He also thanked Justice Breyer for his service.',
'  The president said that he was honoring Justice Breyer for his dedication to serving the country and that he was a retiring Justice of the United States Supreme Court. He also thanked Justice Breyer for his service, noting his background as a top litigator in private practice, a former federal public defender, and a family of public school educators and police officers. He praised Justice Breyer for being a consensus builder and for receiving a broad range of support from the Fraternal Order of Police to former judges appointed by Democrats and Republicans. He also noted that in order to advance liberty and justice, it was necessary to secure the border and fix the immigration system, and that the government was taking steps to do both.   Source: 31',
'  The president said that he was honoring Justice Breyer for his dedication to serving the country and that he was a retiring Justice of the United States Supreme Court. He also thanked Justice Breyer for his service, noting his background as a top litigator in private practice, a former federal public defender, and a family of public school educators and police officers. He praised Justice Breyer for being a consensus builder and for receiving a broad range of support from the Fraternal Order of Police to former judges appointed by Democrats and Republicans. He also noted that in order to advance liberty and justice, it was necessary to secure the border and fix the immigration system, and that the government was taking steps to do both. He also mentioned the need to pass the bipartisan Equality Act to protect LGBTQ+ Americans, and to strengthen the Violence Against Women Act that he had written three decades ago.   Source: 31, 33',
'  The president said that he was honoring Justice Breyer for his dedication to serving the country and that he was a retiring Justice of the United States Supreme Court. He also thanked Justice Breyer for his service, noting his background as a top litigator in private practice, a former federal public defender, and a family of public school educators and police officers. He praised Justice Breyer for being a consensus builder and for receiving a broad range of support from the Fraternal Order of Police to former judges appointed by Democrats and Republicans. He also noted that in order to advance liberty and justice, it was necessary to secure the border and fix the immigration system, and that the government was taking steps to do both. He also mentioned the need to pass the bipartisan Equality Act to protect LGBTQ+ Americans, and to strengthen the Violence Against Women Act that he had written three decades ago. Additionally, he mentioned his plan to lower costs to give families a fair shot, lower the deficit, and go after criminals who stole billions in relief money meant for small businesses and millions of Americans. He also announced that the Justice Department will name a chief prosecutor for pandemic fraud.   Source: 20, 31, 33'],
'output_text': '  The president said that he was honoring Justice Breyer for his dedication to serving the country and that he was a retiring Justice of the United States Supreme Court. He also thanked Justice Breyer for his service, noting his background as a top litigator in private practice, a former federal public defender, and a family of public school educators and police officers. He praised Justice Breyer for being a consensus builder and for receiving a broad range of support from the Fraternal Order of Police to former judges appointed by Democrats and Republicans. He also noted that in order to advance liberty and justice, it was necessary to secure the border and fix the immigration system, and that the government was taking steps to do both. He also mentioned the need to pass the bipartisan Equality Act to protect LGBTQ+ Americans, and to strengthen the Violence Against Women Act that he had written three decades ago. Additionally, he mentioned his plan to lower costs to give families a fair shot, lower the deficit, and go after criminals who stole billions in relief money meant for small businesses and millions of Americans. He also announced that the Justice Department will name a chief prosecutor for pandemic fraud.   Source: 20, 31, 33'}

```

Custom Prompts
You can also use your own prompts with this chain. In this example, we will respond in Italian.

```code
refine_template = (
"The original question is as follows: {question}\n"
"We have provided an existing answer, including sources: {existing_answer}\n"
"We have the opportunity to refine the existing answer"
"(only if needed) with some more context below.\n"
"------------\n"
"{context_str}\n"
"------------\n"
"Given the new context, refine the original answer to better "
"answer the question (in Italian)"
"If you do update it, please update the sources as well. "
"If the context isn't useful, return the original answer."
)
refine_prompt = PromptTemplate(
input_variables=["question", "existing_answer", "context_str"],
template=refine_template,
)

question_template = (
"Context information is below. \n"
"---------------------\n"
"{context_str}"
"\n---------------------\n"
"Given the context information and not prior knowledge, "
"answer the question in Italian: {question}\n"
)
question_prompt = PromptTemplate(
input_variables=["context_str", "question"], template=question_template
)

```

```code
chain = load_qa_with_sources_chain(OpenAI(temperature=0), chain_type="refine", return_intermediate_steps=True, question_prompt=question_prompt, refine_prompt=refine_prompt)
chain({"input_documents": docs, "question": query}, return_only_outputs=True)

```

```code
{'intermediate_steps': ['\nIl presidente ha detto che Justice Breyer ha dedicato la sua vita al servizio di questo paese e ha onorato la sua carriera.',
"  Il presidente ha detto che Justice Breyer ha dedicato la sua vita al servizio di questo paese, ha onorato la sua carriera e ha contribuito a costruire un consenso. Ha ricevuto un ampio sostegno, dall'Ordine Fraterno della Polizia a ex giudici nominati da democratici e repubblicani. Inoltre, ha sottolineato l'importanza di avanzare la libertà e la giustizia attraverso la sicurezza delle frontiere e la risoluzione del sistema di immigrazione. Ha anche menzionato le nuove tecnologie come scanner all'avanguardia per rilevare meglio il traffico di droga, le pattuglie congiunte con Messico e Guatemala per catturare più trafficanti di esseri umani, l'istituzione di giudici di immigrazione dedicati per far sì che le famiglie che fuggono da per",
"  Il presidente ha detto che Justice Breyer ha dedicato la sua vita al servizio di questo paese, ha onorato la sua carriera e ha contribuito a costruire un consenso. Ha ricevuto un ampio sostegno, dall'Ordine Fraterno della Polizia a ex giudici nominati da democratici e repubblicani. Inoltre, ha sottolineato l'importanza di avanzare la libertà e la giustizia attraverso la sicurezza delle frontiere e la risoluzione del sistema di immigrazione. Ha anche menzionato le nuove tecnologie come scanner all'avanguardia per rilevare meglio il traffico di droga, le pattuglie congiunte con Messico e Guatemala per catturare più trafficanti di esseri umani, l'istituzione di giudici di immigrazione dedicati per far sì che le famiglie che fuggono da per",
"  Il presidente ha detto che Justice Breyer ha dedicato la sua vita al servizio di questo paese, ha onorato la sua carriera e ha contribuito a costruire un consenso. Ha ricevuto un ampio sostegno, dall'Ordine Fraterno della Polizia a ex giudici nominati da democratici e repubblicani. Inoltre, ha sottolineato l'importanza di avanzare la libertà e la giustizia attraverso la sicurezza delle frontiere e la risoluzione del sistema di immigrazione. Ha anche menzionato le nuove tecnologie come scanner all'avanguardia per rilevare meglio il traffico di droga, le pattuglie congiunte con Messico e Guatemala per catturare più trafficanti di esseri umani, l'istituzione di giudici di immigrazione dedicati per far sì che le famiglie che fuggono da per"],
'output_text': "  Il presidente ha detto che Justice Breyer ha dedicato la sua vita al servizio di questo paese, ha onorato la sua carriera e ha contribuito a costruire un consenso. Ha ricevuto un ampio sostegno, dall'Ordine Fraterno della Polizia a ex giudici nominati da democratici e repubblicani. Inoltre, ha sottolineato l'importanza di avanzare la libertà e la giustizia attraverso la sicurezza delle frontiere e la risoluzione del sistema di immigrazione. Ha anche menzionato le nuove tecnologie come scanner all'avanguardia per rilevare meglio il traffico di droga, le pattuglie congiunte con Messico e Guatemala per catturare più trafficanti di esseri umani, l'istituzione di giudici di immigrazione dedicati per far sì che le famiglie che fuggono da per"}

```

The
```code
map-rerank
```
Chain#
This sections shows results of using the
```code
map-rerank
```
Chain to do question answering with sources.

```code
chain = load_qa_with_sources_chain(OpenAI(temperature=0), chain_type="map_rerank", metadata_keys=['source'], return_intermediate_steps=True)

```

```code
query = "What did the president say about Justice Breyer"
result = chain({"input_documents": docs, "question": query}, return_only_outputs=True)

```

```code
result["output_text"]

```

```code
' The President thanked Justice Breyer for his service and honored him for dedicating his life to serve the country.'

```

```code
result["intermediate_steps"]

```

```code
[{'answer': ' The President thanked Justice Breyer for his service and honored him for dedicating his life to serve the country.',
'score': '100'},
{'answer': ' This document does not answer the question', 'score': '0'},
{'answer': ' This document does not answer the question', 'score': '0'},
{'answer': ' This document does not answer the question', 'score': '0'}]

```

自定义提示
您也可以使用自己的提示来完成此链。在这个例子中，我们将用意大利语回答。

```code
from langchain.output_parsers import RegexParser

output_parser = RegexParser(
regex=r"(.*?)\nScore: (.*)",
output_keys=["answer", "score"],
)

prompt_template = """Use the following pieces of context to answer the question at the end. If you don't know the answer, just say that you don't know, don't try to make up an answer.

In addition to giving an answer, also return a score of how fully it answered the user's question. This should be in the following format:

Question: [question here]
Helpful Answer In Italian: [answer here]
Score: [score between 0 and 100]

Begin!

Context:
---------
{context}
---------
Question: {question}
Helpful Answer In Italian:"""
PROMPT = PromptTemplate(
template=prompt_template,
input_variables=["context", "question"],
output_parser=output_parser,
)
chain = load_qa_with_sources_chain(OpenAI(temperature=0), chain_type="map_rerank", metadata_keys=['source'], return_intermediate_steps=True, prompt=PROMPT)
query = "What did the president say about Justice Breyer"
result = chain({"input_documents": docs, "question": query}, return_only_outputs=True)

```

```code
result

```

```code
{'source': 30,
'intermediate_steps': [{'answer': ' Il presidente ha detto che Justice Breyer ha dedicato la sua vita a servire questo paese e ha onorato la sua carriera.',
'score': '100'},
{'answer': ' Il presidente non ha detto nulla sulla Giustizia Breyer.',
'score': '100'},
{'answer': ' Non so.', 'score': '0'},
{'answer': ' Il presidente non ha detto nulla sulla giustizia Breyer.',
'score': '100'}],
'output_text': ' Il presidente ha detto che Justice Breyer ha dedicato la sua vita a servire questo paese e ha onorato la sua carriera.'}

```
海德（Hyde）问答（Question Answering）


问答 – LangChain中文网

6大核心模块（Modules)链（Chains)索引示例（Index Examples）问答（Question Answering）

问答
#
本教程教你如何使用LangChain在文档列表上进行问答。它介绍了四种不同的链式处理方式：
```code
stuff
```
、
```code
map_reduce
```
、
```code
refine
```
、
```code
map_rerank
```
。有关这些链式处理方式的更详细解释，请参见此处 (opens in a new tab)。
准备数据
#
首先，我们准备数据。对于此示例，我们在向量数据库上进行相似性搜索，但是这些文档可以以任何方式获取（本教程的重点是强调在获取文档后要做什么)。

```code
from langchain.embeddings.openai import OpenAIEmbeddings
from langchain.text_splitter import CharacterTextSplitter
from langchain.vectorstores import Chroma
from langchain.docstore.document import Document
from langchain.prompts import PromptTemplate
from langchain.indexes.vectorstore import VectorstoreIndexCreator

```

```code
with open("../../state_of_the_union.txt") as f:
state_of_the_union = f.read()
text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)
texts = text_splitter.split_text(state_of_the_union)

embeddings = OpenAIEmbeddings()

```

```code
docsearch = Chroma.from_texts(texts, embeddings, metadatas=[{"source": str(i)} for i in range(len(texts))]).as_retriever()

```

```code
Running Chroma using direct local API.
Using DuckDB in-memory for database. Data will be transient.

```

```code
query = "What did the president say about Justice Breyer"
docs = docsearch.get_relevant_documents(query)

```

```code
from langchain.chains.question_answering import load_qa_chain
from langchain.llms import OpenAI

```

快速入门
#
如果您只想尽快开始，这是推荐的方法：

```code
chain = load_qa_chain(OpenAI(temperature=0), chain_type="stuff")
query = "What did the president say about Justice Breyer"
chain.run(input_documents=docs, question=query)

```

```code
' The president said that Justice Breyer has dedicated his life to serve the country and thanked him for his service.'

```

如果您想更多地控制和理解正在发生的事情，请参见下面的信息。
stuff 链 #
本节显示使用 stuff 链进行问题解答的结果。

```code
chain = load_qa_chain(OpenAI(temperature=0), chain_type="stuff")

```

```code
query = "What did the president say about Justice Breyer"
chain({"input_documents": docs, "question": query}, return_only_outputs=True)

```

```code
{'output_text': ' The president said that Justice Breyer has dedicated his life to serve the country and thanked him for his service.'}

```

自定义提示
您还可以使用自己的提示来使用此链。在此示例中，我们将以意大利语回答。

```code
prompt_template = """Use the following pieces of context to answer the question at the end. If you don't know the answer, just say that you don't know, don't try to make up an answer.

{context}

Question: {question}
Answer in Italian:"""
PROMPT = PromptTemplate(
template=prompt_template, input_variables=["context", "question"]
)
chain = load_qa_chain(OpenAI(temperature=0), chain_type="stuff", prompt=PROMPT)
chain({"input_documents": docs, "question": query}, return_only_outputs=True)

```

```code
{'output_text': ' Il presidente ha detto che Justice Breyer ha dedicato la sua vita a servire questo paese e ha ricevuto una vasta gamma di supporto.'}

```

map_reduce 链 #
本节显示使用 map_reduce 链进行问题解答的结果。

```code
chain = load_qa_chain(OpenAI(temperature=0), chain_type="map_reduce")

```

```code
query = "What did the president say about Justice Breyer"
chain({"input_documents": docs, "question": query}, return_only_outputs=True)

```

```code
{'output_text': ' The president said that Justice Breyer is an Army veteran, Constitutional scholar, and retiring Justice of the United States Supreme Court, and thanked him for his service.'}

```

中间步骤
如果我们想要检查中间步骤，我们还可以返回
```code
map_reduce
```
链的中间步骤。这是通过
```code
return_map_steps
```
变量完成的。

```code
chain = load_qa_chain(OpenAI(temperature=0), chain_type="map_reduce", return_map_steps=True)

```

```code
chain({"input_documents": docs, "question": query}, return_only_outputs=True)

```

```code
{'intermediate_steps': [' "Tonight, I’d like to honor someone who has dedicated his life to serve this country: Justice Stephen Breyer—an Army veteran, Constitutional scholar, and retiring Justice of the United States Supreme Court. Justice Breyer, thank you for your service."',
' A former top litigator in private practice. A former federal public defender. And from a family of public school educators and police officers. A consensus builder. Since she’s been nominated, she’s received a broad range of support—from the Fraternal Order of Police to former judges appointed by Democrats and Republicans.',
' None',
' None'],
'output_text': ' The president said that Justice Breyer is an Army veteran, Constitutional scholar, and retiring Justice of the United States Supreme Court, and thanked him for his service.'}

```

自定义提示
您还可以使用自己的提示来使用此链。在此示例中，我们将以意大利语回答。

```code
question_prompt_template = """Use the following portion of a long document to see if any of the text is relevant to answer the question.
Return any relevant text translated into italian.
{context}
Question: {question}
Relevant text, if any, in Italian:"""
QUESTION_PROMPT = PromptTemplate(
template=question_prompt_template, input_variables=["context", "question"]
)

combine_prompt_template = """Given the following extracted parts of a long document and a question, create a final answer italian.
If you don't know the answer, just say that you don't know. Don't try to make up an answer.

QUESTION: {question}
=========
{summaries}
=========
Answer in Italian:"""
COMBINE_PROMPT = PromptTemplate(
template=combine_prompt_template, input_variables=["summaries", "question"]
)
chain = load_qa_chain(OpenAI(temperature=0), chain_type="map_reduce", return_map_steps=True, question_prompt=QUESTION_PROMPT, combine_prompt=COMBINE_PROMPT)
chain({"input_documents": docs, "question": query}, return_only_outputs=True)

```

```code
{'intermediate_steps': ["\nStasera vorrei onorare qualcuno che ha dedicato la sua vita a servire questo paese: il giustizia Stephen Breyer - un veterano dell'esercito, uno studioso costituzionale e un giustizia in uscita della Corte Suprema degli Stati Uniti. Giustizia Breyer, grazie per il tuo servizio.",
'\nNessun testo pertinente.',
' Non ha detto nulla riguardo a Justice Breyer.',
" Non c'è testo pertinente."],
'output_text': ' Non ha detto nulla riguardo a Justice Breyer.'}

```

批处理大小
使用
```code
map_reduce
```
链时，需要注意map步骤中使用的批处理大小。如果太高，可能会导致速率限制错误。您可以通过设置使用的LLM的批处理大小来控制这一点。请注意，这仅适用于具有此参数的LLM。以下是一个示例：

```code
llm = OpenAI(batch_size=5, temperature=0)

```

```code
Refine
```
链#
本节展示了使用
```code
refine
```
链来进行问答的结果。

```code
chain = load_qa_chain(OpenAI(temperature=0), chain_type="refine")

```

```code
query = "What did the president say about Justice Breyer"
chain({"input_documents": docs, "question": query}, return_only_outputs=True)

```

```code
{'output_text': '  The president said that he wanted to honor Justice Breyer for his dedication to serving the country, his legacy of excellence, and his commitment to advancing liberty and justice, as well as for his support of the Equality Act and his commitment to protecting the rights of LGBTQ+ Americans. He also praised Justice Breyer for his role in helping to pass the Bipartisan Infrastructure Law, which he said would be the most sweeping investment to rebuild America in history and would help the country compete for the jobs of the 21st Century.'}

```

中间步骤
如果需要检查中间步骤，我们还可以返回
```code
refine
```
链的中间步骤。这是通过
```code
return_refine_steps
```
变量完成的。

```code
chain = load_qa_chain(OpenAI(temperature=0), chain_type="refine", return_refine_steps=True)

```

```code
chain({"input_documents": docs, "question": query}, return_only_outputs=True)

```

```code
{'intermediate_steps': ['\nThe president said that he wanted to honor Justice Breyer for his dedication to serving the country and his legacy of excellence.',
'\nThe president said that he wanted to honor Justice Breyer for his dedication to serving the country, his legacy of excellence, and his commitment to advancing liberty and justice.',
'  The president said that he wanted to honor Justice Breyer for his dedication to serving the country, his legacy of excellence, and his commitment to advancing liberty and justice, as well as for his support of the Equality Act and his commitment to protecting the rights of LGBTQ+ Americans.',
'  The president said that he wanted to honor Justice Breyer for his dedication to serving the country, his legacy of excellence, and his commitment to advancing liberty and justice, as well as for his support of the Equality Act and his commitment to protecting the rights of LGBTQ+ Americans. He also praised Justice Breyer for his role in helping to pass the Bipartisan Infrastructure Law, which is the most sweeping investment to rebuild America in history.'],
'output_text': '  The president said that he wanted to honor Justice Breyer for his dedication to serving the country, his legacy of excellence, and his commitment to advancing liberty and justice, as well as for his support of the Equality Act and his commitment to protecting the rights of LGBTQ+ Americans. He also praised Justice Breyer for his role in helping to pass the Bipartisan Infrastructure Law, which is the most sweeping investment to rebuild America in history.'}

```

自定义提示
您还可以在此链中使用自己的提示。在本例中，我们将用意大利语回答。

```code
refine_prompt_template = (
"The original question is as follows: {question}\n"
"We have provided an existing answer: {existing_answer}\n"
"We have the opportunity to refine the existing answer"
"(only if needed) with some more context below.\n"
"------------\n"
"{context_str}\n"
"------------\n"
"Given the new context, refine the original answer to better "
"answer the question. "
"If the context isn't useful, return the original answer. Reply in Italian."
)
refine_prompt = PromptTemplate(
input_variables=["question", "existing_answer", "context_str"],
template=refine_prompt_template,
)

initial_qa_template = (
"Context information is below. \n"
"---------------------\n"
"{context_str}"
"\n---------------------\n"
"Given the context information and not prior knowledge, "
"answer the question: {question}\nYour answer should be in Italian.\n"
)
initial_qa_prompt = PromptTemplate(
input_variables=["context_str", "question"], template=initial_qa_template
)
chain = load_qa_chain(OpenAI(temperature=0), chain_type="refine", return_refine_steps=True,
question_prompt=initial_qa_prompt, refine_prompt=refine_prompt)
chain({"input_documents": docs, "question": query}, return_only_outputs=True)

```

```code
{'intermediate_steps': ['\nIl presidente ha detto che Justice Breyer ha dedicato la sua vita al servizio di questo paese e ha reso omaggio al suo servizio.',
"\nIl presidente ha detto che Justice Breyer ha dedicato la sua vita al servizio di questo paese, ha reso omaggio al suo servizio e ha sostenuto la nomina di una top litigatrice in pratica privata, un ex difensore pubblico federale e una famiglia di insegnanti e agenti di polizia delle scuole pubbliche. Ha anche sottolineato l'importanza di avanzare la libertà e la giustizia attraverso la sicurezza delle frontiere e la risoluzione del sistema di immigrazione.",
"\nIl presidente ha detto che Justice Breyer ha dedicato la sua vita al servizio di questo paese, ha reso omaggio al suo servizio e ha sostenuto la nomina di una top litigatrice in pratica privata, un ex difensore pubblico federale e una famiglia di insegnanti e agenti di polizia delle scuole pubbliche. Ha anche sottolineato l'importanza di avanzare la libertà e la giustizia attraverso la sicurezza delle frontiere, la risoluzione del sistema di immigrazione, la protezione degli americani LGBTQ+ e l'approvazione dell'Equality Act. Ha inoltre sottolineato l'importanza di lavorare insieme per sconfiggere l'epidemia di oppiacei.",
"  Il presidente ha detto che Justice Breyer ha dedicato la sua vita al servizio di questo paese, ha reso omaggio al suo servizio e ha sostenuto la nomina di una top litigatrice in pratica privata, un ex difensore pubblico federale e una famiglia di insegnanti e agenti di polizia delle scuole pubbliche. Ha anche sottolineato l'importanza di avanzare la libertà e la giustizia attraverso la sicurezza delle frontiere, la risoluzione del sistema di immigrazione, la protezione degli americani LGBTQ+ e l'approvazione dell'Equality Act. Ha inoltre sottolineato l'importanza di lavorare insieme per sconfiggere l'epidemia di oppiacei e per investire in America, educare gli americani, far crescere la forza lavoro e costruire l'economia dal"],
'output_text': "  Il presidente ha detto che Justice Breyer ha dedicato la sua vita al servizio di questo paese, ha reso omaggio al suo servizio e ha sostenuto la nomina di una top litigatrice in pratica privata, un ex difensore pubblico federale e una famiglia di insegnanti e agenti di polizia delle scuole pubbliche. Ha anche sottolineato l'importanza di avanzare la libertà e la giustizia attraverso la sicurezza delle frontiere, la risoluzione del sistema di immigrazione, la protezione degli americani LGBTQ+ e l'approvazione dell'Equality Act. Ha inoltre sottolineato l'importanza di lavorare insieme per sconfiggere l'epidemia di oppiacei e per investire in America, educare gli americani, far crescere la forza lavoro e costruire l'economia dal"}

```

```code
map-rerank
```
链#
本节展示了使用
```code
map-rerank
```
链来进行带有来源的问答的结果。

```code
chain = load_qa_chain(OpenAI(temperature=0), chain_type="map_rerank", return_intermediate_steps=True)

```

```code
query = "What did the president say about Justice Breyer"
results = chain({"input_documents": docs, "question": query}, return_only_outputs=True)

```

```code
results["output_text"]

```

```code
' The President thanked Justice Breyer for his service and honored him for dedicating his life to serve the country.'

```

```code
results["intermediate_steps"]

```

```code
[{'answer': ' The President thanked Justice Breyer for his service and honored him for dedicating his life to serve the country.',
'score': '100'},
{'answer': ' This document does not answer the question', 'score': '0'},
{'answer': ' This document does not answer the question', 'score': '0'},
{'answer': ' This document does not answer the question', 'score': '0'}]

```

自定义提示
您还可以使用自己的提示来使用此链。在此示例中，我们将用意大利语回复。

```code
from langchain.output_parsers import RegexParser

output_parser = RegexParser(
regex=r"(.*?)\nScore: (.*)",
output_keys=["answer", "score"],
)

prompt_template = """Use the following pieces of context to answer the question at the end. If you don't know the answer, just say that you don't know, don't try to make up an answer.

In addition to giving an answer, also return a score of how fully it answered the user's question. This should be in the following format:

Question: [question here]
Helpful Answer In Italian: [answer here]
Score: [score between 0 and 100]

Begin!

Context:
---------
{context}
---------
Question: {question}
Helpful Answer In Italian:"""
PROMPT = PromptTemplate(
template=prompt_template,
input_variables=["context", "question"],
output_parser=output_parser,
)

chain = load_qa_chain(OpenAI(temperature=0), chain_type="map_rerank", return_intermediate_steps=True, prompt=PROMPT)
query = "What did the president say about Justice Breyer"
chain({"input_documents": docs, "question": query}, return_only_outputs=True)

```

```code
{'intermediate_steps': [{'answer': ' Il presidente ha detto che Justice Breyer ha dedicato la sua vita a servire questo paese.',
'score': '100'},
{'answer': ' Il presidente non ha detto nulla sulla Giustizia Breyer.',
'score': '100'},
{'answer': ' Non so.', 'score': '0'},
{'answer': ' Non so.', 'score': '0'}],
'output_text': ' Il presidente ha detto che Justice Breyer ha dedicato la sua vita a servire questo paese.'}

```
问答与来源（Q&A with Sources）摘要（Summarize）


用LangChain对文档列表进行摘要 – LangChain中文网

6大核心模块（Modules)链（Chains)索引示例（Index Examples）摘要（Summarize）

用LangChain对文档列表进行摘要
本教程演示了如何使用LangChain对文档列表进行摘要。它涵盖了三种不同的链式类型：
```code
stuff
```
，
```code
map_reduce
```
和
```code
refine
```
。有关这些链式类型的更深入解释，请参见此处 (opens in a new tab)。
准备数据#
首先，我们准备数据。在此示例中，我们从一个长文档中创建多个文档，但这些文档可以以任何方式获取（本笔记的重点是强调获取文档后要做什么)。

```code
from langchain import OpenAI, PromptTemplate, LLMChain
from langchain.text_splitter import CharacterTextSplitter
from langchain.chains.mapreduce import MapReduceChain
from langchain.prompts import PromptTemplate

llm = OpenAI(temperature=0)

text_splitter = CharacterTextSplitter()

```

```code
with open("../../state_of_the_union.txt") as f:
state_of_the_union = f.read()
texts = text_splitter.split_text(state_of_the_union)

```

```code
from langchain.docstore.document import Document

docs = [Document(page_content=t) for t in texts[:3]]

```

```code
from langchain.chains.summarize import load_summarize_chain

```

快速入门#
如果您只想尽快开始，请使用推荐的方法：

```code
chain = load_summarize_chain(llm, chain_type="map_reduce")
chain.run(docs)

```

```code
' In response to Russian aggression in Ukraine, the United States and its allies are taking action to hold Putin accountable, including economic sanctions, asset seizures, and military assistance. The US is also providing economic and humanitarian aid to Ukraine, and has passed the American Rescue Plan and the Bipartisan Infrastructure Law to help struggling families and create jobs. The US remains unified and determined to protect Ukraine and the free world.'

```

如果您想更多地掌握和了解正在发生的事情，请参见下面的信息。
The
```code
stuff
```
Chain#
This sections shows results of using the
```code
stuff
```
Chain to do summarization.

```code
chain = load_summarize_chain(llm, chain_type="stuff")

```

```code
chain.run(docs)

```

```code
' In his speech, President Biden addressed the crisis in Ukraine, the American Rescue Plan, and the Bipartisan Infrastructure Law. He discussed the need to invest in America, educate Americans, and build the economy from the bottom up. He also announced the release of 60 million barrels of oil from reserves around the world, and the creation of a dedicated task force to go after the crimes of Russian oligarchs. He concluded by emphasizing the need to Buy American and use taxpayer dollars to rebuild America.'

```

Custom Prompts
You can also use your own prompts with this chain. In this example, we will respond in Italian.

```code
prompt_template = """Write a concise summary of the following:

{text}

CONCISE SUMMARY IN ITALIAN:"""
PROMPT = PromptTemplate(template=prompt_template, input_variables=["text"])
chain = load_summarize_chain(llm, chain_type="stuff", prompt=PROMPT)
chain.run(docs)

```

```code
"  In questa serata, il Presidente degli Stati Uniti ha annunciato una serie di misure per affrontare la crisi in Ucraina, causata dall'aggressione di Putin. Ha anche annunciato l'invio di aiuti economici, militari e umanitari all'Ucraina. Ha anche annunciato che gli Stati Uniti e i loro alleati stanno imponendo sanzioni economiche a Putin e stanno rilasciando 60 milioni di barili di petrolio dalle riserve di tutto il mondo. Inoltre, ha annunciato che il Dipartimento di Giustizia degli Stati Uniti sta creando una task force dedicata ai crimini degli oligarchi russi. Il Presidente ha anche annunciato l'approvazione della legge bipartitica sull'infrastruttura, che prevede investimenti per la ricostruzione dell'America. Questo porterà a creare posti"

```

The
```code
map_reduce
```
Chain#
This sections shows results of using the
```code
map_reduce
```
Chain to do summarization.

```code
chain = load_summarize_chain(llm, chain_type="map_reduce")

```

```code
chain.run(docs)

```

```code
" In response to Russia's aggression in Ukraine, the United States and its allies have imposed economic sanctions and are taking other measures to hold Putin accountable. The US is also providing economic and military assistance to Ukraine, protecting NATO countries, and releasing oil from its Strategic Petroleum Reserve. President Biden and Vice President Harris have passed legislation to help struggling families and rebuild America's infrastructure."

```

Intermediate Steps
We can also return the intermediate steps for
```code
map_reduce
```
chains, should we want to inspect them. This is done with the
```code
return_map_steps
```
variable.

```code
chain = load_summarize_chain(OpenAI(temperature=0), chain_type="map_reduce", return_intermediate_steps=True)

```

```code
chain({"input_documents": docs}, return_only_outputs=True)

```

```code
{'map_steps': [" In response to Russia's aggression in Ukraine, the United States has united with other freedom-loving nations to impose economic sanctions and hold Putin accountable. The U.S. Department of Justice is also assembling a task force to go after the crimes of Russian oligarchs and seize their ill-gotten gains.",
' The United States and its European allies are taking action to punish Russia for its invasion of Ukraine, including seizing assets, closing off airspace, and providing economic and military assistance to Ukraine. The US is also mobilizing forces to protect NATO countries and has released 30 million barrels of oil from its Strategic Petroleum Reserve to help blunt gas prices. The world is uniting in support of Ukraine and democracy, and the US stands with its Ukrainian-American citizens.',
" President Biden and Vice President Harris ran for office with a new economic vision for America, and have since passed the American Rescue Plan and the Bipartisan Infrastructure Law to help struggling families and rebuild America's infrastructure. This includes creating jobs, modernizing roads, airports, ports, and waterways, replacing lead pipes, providing affordable high-speed internet, and investing in American products to support American jobs."],
'output_text': " In response to Russia's aggression in Ukraine, the United States and its allies have imposed economic sanctions and are taking other measures to hold Putin accountable. The US is also providing economic and military assistance to Ukraine, protecting NATO countries, and passing legislation to help struggling families and rebuild America's infrastructure. The world is uniting in support of Ukraine and democracy, and the US stands with its Ukrainian-American citizens."}

```

Custom Prompts
You can also use your own prompts with this chain. In this example, we will respond in Italian.

```code
prompt_template = """Write a concise summary of the following:

{text}

CONCISE SUMMARY IN ITALIAN:"""
PROMPT = PromptTemplate(template=prompt_template, input_variables=["text"])
chain = load_summarize_chain(OpenAI(temperature=0), chain_type="map_reduce", return_intermediate_steps=True, map_prompt=PROMPT, combine_prompt=PROMPT)
chain({"input_documents": docs}, return_only_outputs=True)

```

```code
{'intermediate_steps': ["  Questa sera, ci incontriamo come democratici, repubblicani e indipendenti, ma soprattutto come americani. La Russia di Putin ha cercato di scuotere le fondamenta del mondo libero, ma ha sottovalutato la forza della gente ucraina. Gli Stati Uniti e i loro alleati stanno ora imponendo sanzioni economiche a Putin e stanno tagliando l'accesso della Russia alla tecnologia. Il Dipartimento di Giustizia degli Stati Uniti sta anche creando una task force dedicata per andare dopo i crimini degli oligarchi russi.",
"  Stiamo unendo le nostre forze con quelle dei nostri alleati europei per sequestrare yacht, appartamenti di lusso e jet privati di Putin. Abbiamo chiuso lo spazio aereo americano ai voli russi e stiamo fornendo più di un miliardo di dollari in assistenza all'Ucraina. Abbiamo anche mobilitato le nostre forze terrestri, aeree e navali per proteggere i paesi della NATO. Abbiamo anche rilasciato 60 milioni di barili di petrolio dalle riserve di tutto il mondo, di cui 30 milioni dalla nostra riserva strategica di petrolio. Stiamo affrontando una prova reale e ci vorrà del tempo, ma alla fine Putin non riuscirà a spegnere l'amore dei popoli per la libertà.",
"  Il Presidente Biden ha lottato per passare l'American Rescue Plan per aiutare le persone che soffrivano a causa della pandemia. Il piano ha fornito sollievo economico immediato a milioni di americani, ha aiutato a mettere cibo sulla loro tavola, a mantenere un tetto sopra le loro teste e a ridurre il costo dell'assicurazione sanitaria. Il piano ha anche creato più di 6,5 milioni di nuovi posti di lavoro, il più alto numero di posti di lavoro creati in un anno nella storia degli Stati Uniti. Il Presidente Biden ha anche firmato la legge bipartitica sull'infrastruttura, la più ampia iniziativa di ricostruzione della storia degli Stati Uniti. Il piano prevede di modernizzare le strade, gli aeroporti, i porti e le vie navigabili in"],
'output_text': "  Il Presidente Biden sta lavorando per aiutare le persone che soffrono a causa della pandemia attraverso l'American Rescue Plan e la legge bipartitica sull'infrastruttura. Gli Stati Uniti e i loro alleati stanno anche imponendo sanzioni economiche a Putin e tagliando l'accesso della Russia alla tecnologia. Stanno anche sequestrando yacht, appartamenti di lusso e jet privati di Putin e fornendo più di un miliardo di dollari in assistenza all'Ucraina. Alla fine, Putin non riuscirà a spegnere l'amore dei popoli per la libertà."}

```

The
```code
refine
```
Chain#
这部分展示了使用
```code
refine
```
链进行总结的结果。

```code
chain = load_summarize_chain(llm, chain_type="refine")

chain.run(docs)

```

```code
"  In response to Russia's aggression in Ukraine, the United States has united with other freedom-loving nations to impose economic sanctions and hold Putin accountable. The U.S. Department of Justice is also assembling a task force to go after the crimes of Russian oligarchs and seize their ill-gotten gains. We are joining with our European allies to find and seize the assets of Russian oligarchs, including yachts, luxury apartments, and private jets. The U.S. is also closing off American airspace to all Russian flights, further isolating Russia and adding an additional squeeze on their economy. The U.S. and its allies are providing support to the Ukrainians in their fight for freedom, including military, economic, and humanitarian assistance. The U.S. is also mobilizing ground forces, air squadrons, and ship deployments to protect NATO countries. The U.S. and its allies are also releasing 60 million barrels of oil from reserves around the world, with the U.S. contributing 30 million barrels from its own Strategic Petroleum Reserve. In addition, the U.S. has passed the American Rescue Plan to provide immediate economic relief for tens of millions of Americans, and the Bipartisan Infrastructure Law to rebuild America and create jobs. This investment will"

```

中间步骤
如果我们想要检查它们，我们也可以返回
```code
refine
```
链的中间步骤。这可以通过
```code
return_refine_steps
```
变量来完成。

```code
chain = load_summarize_chain(OpenAI(temperature=0), chain_type="refine", return_intermediate_steps=True)

chain({"input_documents": docs}, return_only_outputs=True)

```

```code
{'refine_steps': [" In response to Russia's aggression in Ukraine, the United States has united with other freedom-loving nations to impose economic sanctions and hold Putin accountable. The U.S. Department of Justice is also assembling a task force to go after the crimes of Russian oligarchs and seize their ill-gotten gains.",
"  In response to Russia's aggression in Ukraine, the United States has united with other freedom-loving nations to impose economic sanctions and hold Putin accountable. The U.S. Department of Justice is also assembling a task force to go after the crimes of Russian oligarchs and seize their ill-gotten gains. We are joining with our European allies to find and seize the assets of Russian oligarchs, including yachts, luxury apartments, and private jets. The U.S. is also closing off American airspace to all Russian flights, further isolating Russia and adding an additional squeeze on their economy. The U.S. and its allies are providing support to the Ukrainians in their fight for freedom, including military, economic, and humanitarian assistance. The U.S. is also mobilizing ground forces, air squadrons, and ship deployments to protect NATO countries. The U.S. and its allies are also releasing 60 million barrels of oil from reserves around the world, with the U.S. contributing 30 million barrels from its own Strategic Petroleum Reserve. Putin's war on Ukraine has left Russia weaker and the rest of the world stronger, with the world uniting in support of democracy and peace.",
"  In response to Russia's aggression in Ukraine, the United States has united with other freedom-loving nations to impose economic sanctions and hold Putin accountable. The U.S. Department of Justice is also assembling a task force to go after the crimes of Russian oligarchs and seize their ill-gotten gains. We are joining with our European allies to find and seize the assets of Russian oligarchs, including yachts, luxury apartments, and private jets. The U.S. is also closing off American airspace to all Russian flights, further isolating Russia and adding an additional squeeze on their economy. The U.S. and its allies are providing support to the Ukrainians in their fight for freedom, including military, economic, and humanitarian assistance. The U.S. is also mobilizing ground forces, air squadrons, and ship deployments to protect NATO countries. The U.S. and its allies are also releasing 60 million barrels of oil from reserves around the world, with the U.S. contributing 30 million barrels from its own Strategic Petroleum Reserve. In addition, the U.S. has passed the American Rescue Plan to provide immediate economic relief for tens of millions of Americans, and the Bipartisan Infrastructure Law to rebuild America and create jobs. This includes investing"],
'output_text': "  In response to Russia's aggression in Ukraine, the United States has united with other freedom-loving nations to impose economic sanctions and hold Putin accountable. The U.S. Department of Justice is also assembling a task force to go after the crimes of Russian oligarchs and seize their ill-gotten gains. We are joining with our European allies to find and seize the assets of Russian oligarchs, including yachts, luxury apartments, and private jets. The U.S. is also closing off American airspace to all Russian flights, further isolating Russia and adding an additional squeeze on their economy. The U.S. and its allies are providing support to the Ukrainians in their fight for freedom, including military, economic, and humanitarian assistance. The U.S. is also mobilizing ground forces, air squadrons, and ship deployments to protect NATO countries. The U.S. and its allies are also releasing 60 million barrels of oil from reserves around the world, with the U.S. contributing 30 million barrels from its own Strategic Petroleum Reserve. In addition, the U.S. has passed the American Rescue Plan to provide immediate economic relief for tens of millions of Americans, and the Bipartisan Infrastructure Law to rebuild America and create jobs. This includes investing"}

```

自定义提示
您也可以使用自己的提示来运行此链。在这个例子中，我们将用意大利语回答。

```code
prompt_template = """Write a concise summary of the following:

{text}

CONCISE SUMMARY IN ITALIAN:"""
PROMPT = PromptTemplate(template=prompt_template, input_variables=["text"])
refine_template = (
"Your job is to produce a final summary\n"
"We have provided an existing summary up to a certain point: {existing_answer}\n"
"We have the opportunity to refine the existing summary"
"(only if needed) with some more context below.\n"
"------------\n"
"{text}\n"
"------------\n"
"Given the new context, refine the original summary in Italian"
"If the context isn't useful, return the original summary."
)
refine_prompt = PromptTemplate(
input_variables=["existing_answer", "text"],
template=refine_template,
)
chain = load_summarize_chain(OpenAI(temperature=0), chain_type="refine", return_intermediate_steps=True, question_prompt=PROMPT, refine_prompt=refine_prompt)
chain({"input_documents": docs}, return_only_outputs=True)

```

```code
{'intermediate_steps': ["  Questa sera, ci incontriamo come democratici, repubblicani e indipendenti, ma soprattutto come americani. La Russia di Putin ha cercato di scuotere le fondamenta del mondo libero, ma ha sottovalutato la forza della gente ucraina. Insieme ai nostri alleati, stiamo imponendo sanzioni economiche, tagliando l'accesso della Russia alla tecnologia e bloccando i suoi più grandi istituti bancari dal sistema finanziario internazionale. Il Dipartimento di Giustizia degli Stati Uniti sta anche assemblando una task force dedicata per andare dopo i crimini degli oligarchi russi.",
"  Questa sera, ci incontriamo come democratici, repubblicani e indipendenti, ma soprattutto come americani. La Russia di Putin ha cercato di scuotere le fondamenta del mondo libero, ma ha sottovalutato la forza della gente ucraina. Insieme ai nostri alleati, stiamo imponendo sanzioni economiche, tagliando l'accesso della Russia alla tecnologia, bloccando i suoi più grandi istituti bancari dal sistema finanziario internazionale e chiudendo lo spazio aereo americano a tutti i voli russi. Il Dipartimento di Giustizia degli Stati Uniti sta anche assemblando una task force dedicata per andare dopo i crimini degli oligarchi russi. Stiamo fornendo più di un miliardo di dollari in assistenza diretta all'Ucraina e fornendo assistenza militare,",
"  Questa sera, ci incontriamo come democratici, repubblicani e indipendenti, ma soprattutto come americani. La Russia di Putin ha cercato di scuotere le fondamenta del mondo libero, ma ha sottovalutato la forza della gente ucraina. Insieme ai nostri alleati, stiamo imponendo sanzioni economiche, tagliando l'accesso della Russia alla tecnologia, bloccando i suoi più grandi istituti bancari dal sistema finanziario internazionale e chiudendo lo spazio aereo americano a tutti i voli russi. Il Dipartimento di Giustizia degli Stati Uniti sta anche assemblando una task force dedicata per andare dopo i crimini degli oligarchi russi. Stiamo fornendo più di un miliardo di dollari in assistenza diretta all'Ucraina e fornendo assistenza militare."],
'output_text': "  Questa sera, ci incontriamo come democratici, repubblicani e indipendenti, ma soprattutto come americani. La Russia di Putin ha cercato di scuotere le fondamenta del mondo libero, ma ha sottovalutato la forza della gente ucraina. Insieme ai nostri alleati, stiamo imponendo sanzioni economiche, tagliando l'accesso della Russia alla tecnologia, bloccando i suoi più grandi istituti bancari dal sistema finanziario internazionale e chiudendo lo spazio aereo americano a tutti i voli russi. Il Dipartimento di Giustizia degli Stati Uniti sta anche assemblando una task force dedicata per andare dopo i crimini degli oligarchi russi. Stiamo fornendo più di un miliardo di dollari in assistenza diretta all'Ucraina e fornendo assistenza militare."}

```
问答（Question Answering）向量数据库问答（Vector DB QA）


检索问答# – LangChain中文网

6大核心模块（Modules)链（Chains)索引示例（Index Examples）向量数据库问答（Vector DB QA）

检索问答#
这个示例展示了如何在索引上进行问答。

```code
from langchain.embeddings.openai import OpenAIEmbeddings
from langchain.vectorstores import Chroma
from langchain.text_splitter import CharacterTextSplitter
from langchain.llms import OpenAI
from langchain.chains import RetrievalQA

```

```code
from langchain.document_loaders import TextLoader
loader = TextLoader("../../state_of_the_union.txt")
documents = loader.load()
text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)
texts = text_splitter.split_documents(documents)

embeddings = OpenAIEmbeddings()
docsearch = Chroma.from_documents(texts, embeddings)

```

```code
Running Chroma using direct local API.
Using DuckDB in-memory for database. Data will be transient.

```

```code
qa = RetrievalQA.from_chain_type(llm=OpenAI(), chain_type="stuff", retriever=docsearch.as_retriever())

```

```code
query = "What did the president say about Ketanji Brown Jackson"
qa.run(query)

```

```code
" The president said that she is one of the nation's top legal minds, a former top litigator in private practice, a former federal public defender, and from a family of public school educators and police officers. He also said that she is a consensus builder and has received a broad range of support, from the Fraternal Order of Police to former judges appointed by Democrats and Republicans."

```

链类型#
你可以轻松地指定不同的链类型来加载和使用RetrievalQA链。有关这些类型的更详细的演示，请参见这个notebook。
有两种加载不同链类型的方法。首先，你可以在
```code
from_chain_type
```
方法中指定链类型参数。这允许你传入你想要使用的链类型的名称。例如，在下面的例子中，我们将链类型更改为
```code
map_reduce
```
。

```code
qa = RetrievalQA.from_chain_type(llm=OpenAI(), chain_type="map_reduce", retriever=docsearch.as_retriever())

```

```code
query = "What did the president say about Ketanji Brown Jackson"
qa.run(query)

```

```code
" The president said that Judge Ketanji Brown Jackson is one of our nation's top legal minds, a former top litigator in private practice and a former federal public defender, from a family of public school educators and police officers, a consensus builder and has received a broad range of support from the Fraternal Order of Police to former judges appointed by Democrats and Republicans."

```

以上方法允许你非常简单地更改链类型，但它确实提供了对该链类型参数的许多灵活性。如果你想控制这些参数，你可以直接加载链（就像在这个notebook中所做的那样)，然后将其直接传递给RetrievalQA链的
```code
combine_documents_chain
```
参数。例如：

```code
from langchain.chains.question_answering import load_qa_chain
qa_chain = load_qa_chain(OpenAI(temperature=0), chain_type="stuff")
qa = RetrievalQA(combine_documents_chain=qa_chain, retriever=docsearch.as_retriever())

```

```code
query = "What did the president say about Ketanji Brown Jackson"
qa.run(query)

```

```code
" The president said that Ketanji Brown Jackson is one of the nation's top legal minds, a former top litigator in private practice, a former federal public defender, and from a family of public school educators and police officers. He also said that she is a consensus builder and has received a broad range of support from the Fraternal Order of Police to former judges appointed by Democrats and Republicans."

```

自定义提示#
您可以传递自定义提示来进行问答。这些提示与您可以传递到基础问答链中的提示相同。

```code
from langchain.prompts import PromptTemplate
prompt_template = """Use the following pieces of context to answer the question at the end. If you don't know the answer, just say that you don't know, don't try to make up an answer.

{context}

Question: {question}
Answer in Italian:"""
PROMPT = PromptTemplate(
template=prompt_template, input_variables=["context", "question"]
)

```

```code
chain_type_kwargs = {"prompt": PROMPT}
qa = RetrievalQA.from_chain_type(llm=OpenAI(), chain_type="stuff", retriever=docsearch.as_retriever(), chain_type_kwargs=chain_type_kwargs)

```

```code
query = "What did the president say about Ketanji Brown Jackson"
qa.run(query)

```

```code
" Il presidente ha detto che Ketanji Brown Jackson è una delle menti legali più importanti del paese, che continuerà l'eccellenza di Justice Breyer e che ha ricevuto un ampio sostegno, da Fraternal Order of Police a ex giudici nominati da democratici e repubblicani."

```

返回源文档#
此外，我们可以在构建链时指定一个可选参数来返回用于回答问题的源文档。

```code
qa = RetrievalQA.from_chain_type(llm=OpenAI(), chain_type="stuff", retriever=docsearch.as_retriever(), return_source_documents=True)

```

```code
query = "What did the president say about Ketanji Brown Jackson"
result = qa({"query": query})

```

```code
result["result"]

```

```code
" The president said that Ketanji Brown Jackson is one of the nation's top legal minds, a former top litigator in private practice and a former federal public defender from a family of public school educators and police officers, and that she has received a broad range of support from the Fraternal Order of Police to former judges appointed by Democrats and Republicans."

```

```code
result["source_documents"]

```

```code
[Document(page_content='Tonight. I call on the Senate to: Pass the Freedom to Vote Act. Pass the John Lewis Voting Rights Act. And while you’re at it, pass the Disclose Act so Americans can know who is funding our elections.   Tonight, I’d like to honor someone who has dedicated his life to serve this country: Justice Stephen Breyer—an Army veteran, Constitutional scholar, and retiring Justice of the United States Supreme Court. Justice Breyer, thank you for your service.   One of the most serious constitutional responsibilities a President has is nominating someone to serve on the United States Supreme Court.   And I did that 4 days ago, when I nominated Circuit Court of Appeals Judge Ketanji Brown Jackson. One of our nation’s top legal minds, who will continue Justice Breyer’s legacy of excellence.', lookup_str='', metadata={'source': '../../state_of_the_union.txt'}, lookup_index=0),
Document(page_content='A former top litigator in private practice. A former federal public defender. And from a family of public school educators and police officers. A consensus builder. Since she’s been nominated, she’s received a broad range of support—from the Fraternal Order of Police to former judges appointed by Democrats and Republicans.   And if we are to advance liberty and justice, we need to secure the Border and fix the immigration system.   We can do both. At our border, we’ve installed new technology like cutting-edge scanners to better detect drug smuggling.    We’ve set up joint patrols with Mexico and Guatemala to catch more human traffickers.    We’re putting in place dedicated immigration judges so families fleeing persecution and violence can have their cases heard faster.   We’re securing commitments and supporting partners in South and Central America to host more refugees and secure their own borders.', lookup_str='', metadata={'source': '../../state_of_the_union.txt'}, lookup_index=0),
Document(page_content='And for our LGBTQ+ Americans, let’s finally get the bipartisan Equality Act to my desk. The onslaught of state laws targeting transgender Americans and their families is wrong.   As I said last year, especially to our younger transgender Americans, I will always have your back as your President, so you can be yourself and reach your God-given potential.   While it often appears that we never agree, that isn’t true. I signed 80 bipartisan bills into law last year. From preventing government shutdowns to protecting Asian-Americans from still-too-common hate crimes to reforming military justice.   And soon, we’ll strengthen the Violence Against Women Act that I first wrote three decades ago. It is important for us to show the nation that we can come together and do big things.   So tonight I’m offering a Unity Agenda for the Nation. Four big things we can do together.    First, beat the opioid epidemic.', lookup_str='', metadata={'source': '../../state_of_the_union.txt'}, lookup_index=0),
Document(page_content='Tonight, I’m announcing a crackdown on these companies overcharging American businesses and consumers.   And as Wall Street firms take over more nursing homes, quality in those homes has gone down and costs have gone up.    That ends on my watch.   Medicare is going to set higher standards for nursing homes and make sure your loved ones get the care they deserve and expect.   We’ll also cut costs and keep the economy going strong by giving workers a fair shot, provide more training and apprenticeships, hire them based on their skills not degrees.   Let’s pass the Paycheck Fairness Act and paid leave.    Raise the minimum wage to $15 an hour and extend the Child Tax Credit, so no one has to raise a family in poverty.   Let’s increase Pell Grants and increase our historic support of HBCUs, and invest in what Jill—our First Lady who teaches full-time—calls America’s best-kept secret: community colleges.', lookup_str='', metadata={'source': '../../state_of_the_union.txt'}, lookup_index=0)]

```
摘要（Summarize）向量数据库问答与来源（Vector DB Q&A with Sources）


RetrievalQAWithSourcesChain – LangChain中文网

6大核心模块（Modules)链（Chains)索引示例（Index Examples）向量数据库问答与来源（Vector DB Q&A with Sources）

RetrievalQAWithSourcesChain
本教程介绍如何使用索引对问题进行基于来源的问答。它通过使用
```code
RetrievalQAWithSourcesChain
```
来完成从索引中查找文档的工作。

```code
from langchain.embeddings.openai import OpenAIEmbeddings
from langchain.embeddings.cohere import CohereEmbeddings
from langchain.text_splitter import CharacterTextSplitter
from langchain.vectorstores.elastic_vector_search import ElasticVectorSearch
from langchain.vectorstores import Chroma

```

```code
with open("../../state_of_the_union.txt") as f:
state_of_the_union = f.read()
text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)
texts = text_splitter.split_text(state_of_the_union)

embeddings = OpenAIEmbeddings()

```

```code
docsearch = Chroma.from_texts(texts, embeddings, metadatas=[{"source": f"{i}-pl"} for i in range(len(texts))])

```

```code
Running Chroma using direct local API.
Using DuckDB in-memory for database. Data will be transient.

```

```code
from langchain.chains import RetrievalQAWithSourcesChain

```

```code
from langchain import OpenAI

chain = RetrievalQAWithSourcesChain.from_chain_type(OpenAI(temperature=0), chain_type="stuff", retriever=docsearch.as_retriever())

```

```code
chain({"question": "What did the president say about Justice Breyer"}, return_only_outputs=True)

```

```code
{'answer': ' The president honored Justice Breyer for his service and mentioned his legacy of excellence.\n',
'sources': '31-pl'}

```

链式类型#
您可以轻松指定要加载和使用的不同链式类型。有关这些类型的更详细演示，请参见本教程。
有两种加载不同链式类型的方法。首先，您可以在
```code
from_chain_type
```
方法中指定链式类型参数。这允许您传递要使用的链式类型的名称。例如，在下面的示例中，我们将链式类型更改为
```code
map_reduce
```
。

```code
chain = RetrievalQAWithSourcesChain.from_chain_type(OpenAI(temperature=0), chain_type="map_reduce", retriever=docsearch.as_retriever())

```

```code
chain({"question": "What did the president say about Justice Breyer"}, return_only_outputs=True)

```

```code
{'answer': ' The president said "Justice Breyer—an Army veteran, Constitutional scholar, and retiring Justice of the United States Supreme Court. Justice Breyer, thank you for your service."\n',
'sources': '31-pl'}

```

The above way allows you to really simply change the chain_type, but it does provide a ton of flexibility over parameters to that chain type. If you want to control those parameters, you can load the chain directly (as you did in this notebook) and then pass that directly to the the RetrievalQAWithSourcesChain chain with the
```code
combine_documents_chain
```
parameter. For example:

```code
from langchain.chains.qa_with_sources import load_qa_with_sources_chain
qa_chain = load_qa_with_sources_chain(OpenAI(temperature=0), chain_type="stuff")
qa = RetrievalQAWithSourcesChain(combine_documents_chain=qa_chain, retriever=docsearch.as_retriever())

```

```code
qa({"question": "What did the president say about Justice Breyer"}, return_only_outputs=True)

```

```code
{'answer': ' The president honored Justice Breyer for his service and mentioned his legacy of excellence.\n',
'sources': '31-pl'}

```
向量数据库问答（Vector DB QA）向量数据库文本生成（Vector DB Text Generation）


使用LangChain对向量索引进行文本生成 – LangChain中文网

6大核心模块（Modules)链（Chains)索引示例（Index Examples）向量数据库文本生成（Vector DB Text Generation）

使用LangChain对向量索引进行文本生成
本教程演示了如何使用LangChain对向量索引进行文本生成。如果我们想要生成能够从大量自定义文本中汲取知识的文本，例如生成具有先前编写的博客文章的理解或能够参考产品文档的产品教程，则这非常有用。
准备数据#
首先，我们要准备数据。对于这个示例，我们获取了托管在Github上的由markdown文件组成的文档站点，并将它们分成足够小的文档。

```code
from langchain.llms import OpenAI
from langchain.docstore.document import Document
import requests
from langchain.embeddings.openai import OpenAIEmbeddings
from langchain.vectorstores import Chroma
from langchain.text_splitter import CharacterTextSplitter
from langchain.prompts import PromptTemplate
import pathlib
import subprocess
import tempfile

```

```code
def get_github_docs(repo_owner, repo_name):
with tempfile.TemporaryDirectory() as d:
subprocess.check_call(
f"git clone --depth 1 https://github.com/{repo_owner}/{repo_name}.git .",
cwd=d,
shell=True,
)
git_sha = (
subprocess.check_output("git rev-parse HEAD", shell=True, cwd=d)
.decode("utf-8")
.strip()
)
repo_path = pathlib.Path(d)
markdown_files = list(repo_path.glob("*/*.md")) + list(
repo_path.glob("*/*.mdx")
)
for markdown_file in markdown_files:
with open(markdown_file, "r") as f:
relative_path = markdown_file.relative_to(repo_path)
github_url = f"https://github.com/{repo_owner}/{repo_name}/blob/{git_sha}/{relative_path}"
yield Document(page_content=f.read(), metadata={"source": github_url})

sources = get_github_docs("yirenlu92", "deno-manual-forked")

source_chunks = []
splitter = CharacterTextSplitter(separator=" ", chunk_size=1024, chunk_overlap=0)
for source in sources:
for chunk in splitter.split_text(source.page_content):
source_chunks.append(Document(page_content=chunk, metadata=source.metadata))

```

```code
Cloning into '.'...

```

设置向量数据库#
现在，我们将文档的内容划分成块，然后将所有这些信息放入向量索引中以便于检索。

```code
search_index = Chroma.from_documents(source_chunks, OpenAIEmbeddings())

```

使用自定义提示设置LLM Chain#
Next, let’s set up a simple LLM chain but give it a custom prompt for blog post generation. Note that the custom prompt is parameterized and takes two inputs:
```code
context
```
, which will be the documents fetched from the vector search, and
```code
topic
```
, which is given by the user.

```code
from langchain.chains import LLMChain
prompt_template = """Use the context below to write a 400 word blog post about the topic below:
Context: {context}
Topic: {topic}
Blog post:"""

PROMPT = PromptTemplate(
template=prompt_template, input_variables=["context", "topic"]
)

llm = OpenAI(temperature=0)

chain = LLMChain(llm=llm, prompt=PROMPT)

```

Generate Text#
Finally, we write a function to apply our inputs to the chain. The function takes an input parameter
```code
topic
```
. We find the documents in the vector index that correspond to that
```code
topic
```
, and use them as additional context in our simple LLM chain.

```code
def generate_blog_post(topic):
docs = search_index.similarity_search(topic, k=4)
inputs = [{"context": doc.page_content, "topic": topic} for doc in docs]
print(chain.apply(inputs))

```

```code
generate_blog_post("environment variables")

```

```code
[{'text': '  Environment variables are a great way to store and access sensitive information in your Deno applications. Deno offers built-in support for environment variables with `Deno.env`, and you can also use a `.env` file to store and access environment variables.  Using `Deno.env` is simple. It has getter and setter methods, so you can easily set and retrieve environment variables. For example, you can set the `FIREBASE_API_KEY` and `FIREBASE_AUTH_DOMAIN` environment variables like this:  ```ts\nDeno.env.set("FIREBASE_API_KEY", "examplekey123");\nDeno.env.set("FIREBASE_AUTH_DOMAIN", "firebasedomain.com");  console.log(Deno.env.get("FIREBASE_API_KEY")); // examplekey123\nconsole.log(Deno.env.get("FIREBASE_AUTH_DOMAIN")); // firebasedomain.com\n```  You can also store environment variables in a `.env` file. This is a great'}, {'text': '  Environment variables are a powerful tool for managing configuration settings in a program. They allow us to set values that can be used by the program, without having to hard-code them into the code. This makes it easier to change settings without having to modify the code.  In Deno, environment variables can be set in a few different ways. The most common way is to use the `VAR=value` syntax. This will set the environment variable `VAR` to the value `value`. This can be used to set any number of environment variables before running a command. For example, if we wanted to set the environment variable `VAR` to `hello` before running a Deno command, we could do so like this:  ```\nVAR=hello deno run main.ts\n```  This will set the environment variable `VAR` to `hello` before running the command. We can then access this variable in our code using the `Deno.env.get()` function. For example, if we ran the following command:  ```\nVAR=hello && deno eval "console.log(\'Deno: \' + Deno.env.get(\'VAR'}, {'text': '  Environment variables are a powerful tool for developers, allowing them to store and access data without having to hard-code it into their applications. In Deno, you can access environment variables using the `Deno.env.get()` function.  For example, if you wanted to access the `HOME` environment variable, you could do so like this:  ```js\n// env.js\nDeno.env.get("HOME");\n```  When running this code, you\'ll need to grant the Deno process access to environment variables. This can be done by passing the `--allow-env` flag to the `deno run` command. You can also specify which environment variables you want to grant access to, like this:  ```shell\n# Allow access to only the HOME env var\ndeno run --allow-env=HOME env.js\n```  It\'s important to note that environment variables are case insensitive on Windows, so Deno also matches them case insensitively (on Windows only).  Another thing to be aware of when using environment variables is subprocess permissions. Subprocesses are powerful and can access system resources regardless of the permissions you granted to the Den'}, {'text': '  Environment variables are an important part of any programming language, and Deno is no exception. Deno is a secure JavaScript and TypeScript runtime built on the V8 JavaScript engine, and it recently added support for environment variables. This feature was added in Deno version 1.6.0, and it is now available for use in Deno applications.  Environment variables are used to store information that can be used by programs. They are typically used to store configuration information, such as the location of a database or the name of a user. In Deno, environment variables are stored in the `Deno.env` object. This object is similar to the `process.env` object in Node.js, and it allows you to access and set environment variables.  The `Deno.env` object is a read-only object, meaning that you cannot directly modify the environment variables. Instead, you must use the `Deno.env.set()` function to set environment variables. This function takes two arguments: the name of the environment variable and the value to set it to. For example, if you wanted to set the `FOO` environment variable to `bar`, you would use the following code:  ```'}]

```
向量数据库问答与来源（Vector DB Q&A with Sources）索引（Indexes)


索引 – LangChain中文网

6大核心模块（Modules)索引（Indexes)

索引
索引是指构造文档的方法，以便 LLM 可以最好地与它们交互。此模块包含用于处理文档的实用工具函数、不同类型的索引，以及在链中使用这些索引的示例。
在链中使用索引的最常见方式是“检索”步骤。这一步是指接受用户的查询并返回最相关的文档。我们之所以这样区分，是因为(1)索引可以用于检索之外的其他事情，(2)检索可以使用索引之外的其他逻辑来查找相关文档。因此，我们有一个“寻回者”接口的概念-这是大多数链工作的接口。
大多数时候，当我们谈论索引和检索时，我们谈论的是索引和检索非结构化数据(如文本文档)。对于与结构化数据(SQL 表等)或 API 的交互，请参阅相应的用例部分以获得相关功能的链接。LangChain 支持的主要索引和检索类型目前主要集中在向量数据库上，因此我们深入研究了这些主题的许多功能。
有关这方面的一切概述，请参阅下面的教程开始:
开始
然后，我们对四个主要组成部分进行深入研究。
文档加载程序
如何从各种源加载文档。
文字分割器
关于分割文本的抽象和实现的概述。
VectorStores
概述 Vector Stores 和 LangChain 提供的许多集成。
猎犬
检索器概述和 LangChain 提供的实现。向量数据库文本生成（Vector DB Text Generation）入门（Getting Started）


开始 – LangChain中文网

6大核心模块（Modules)索引（Indexes)入门（Getting Started）

开始
LangChain 主要关注于构建索引，目标是使用它们作为检索器。为了更好地理解这意味着什么，有必要突出显示基本检索器接口是什么。LangChain 的 baseRetriever 类如下:

```code
from abc import ABC, abstractmethod
from typing import List
from langchain.schema import Document

class BaseRetriever(ABC):
@abstractmethod
def get_relevant_documents(self, query: str) -> List[Document]:
"""Get texts relevant for a query.

Args:
query: string to find relevant texts for

Returns:
List of relevant documents
"""
```

就是这么简单! get _ relevant _ document 方法可以按照您认为合适的方式实现。
当然，我们也帮助构建我们认为有用的检索器是什么。我们主要关注的检索器类型是 Vectorstore 检索器。在本指南的其余部分中，我们将关注这一点。
为了理解什么是向量库检索器，理解向量库是什么非常重要。我们来看看。
默认情况下，LangChain 使用 Chroma 作为向量存储来索引和搜索嵌入。要学习本教程，我们首先需要安装 chromadb。

```code
pip install chromadb
```

这个例子展示了对文档的问题回答。我们选择这个例子作为开始的例子，因为它很好地组合了许多不同的元素(文本分割器、嵌入、向量存储) ，然后还演示了如何在链中使用它们。
通过文件回答问题包括四个步骤:

创建索引

从该索引创建检索器

创建一个问题回答链

问问题！

每个步骤都有多个子步骤和可能的配置。在本教程中，我们将主要关注(1)。我们将首先展示这样做的一行程序，然后分解实际发生的情况。
首先，让我们导入一些无论如何都会使用的通用类。

```code
from langchain.chains import RetrievalQA
from langchain.llms import OpenAI
```

接下来在通用设置中，让我们指定要使用的文档加载程序。您可以在这里下载 state _ of _ the _ union.txt 文件

```code
from langchain.document_loaders import TextLoader
loader = TextLoader('../state_of_the_union.txt', encoding='utf8')
```

创建一行索引
为了尽快开始，我们可以使用 VectorstoreIndexCreator。

```code
from langchain.indexes import VectorstoreIndexCreator
```

```code
index = VectorstoreIndexCreator().from_loaders([loader])
```

```code
Running Chroma using direct local API.
Using DuckDB in-memory for database. Data will be transient.
```

现在已经创建了索引，我们可以使用它来询问数据的问题！请注意，在引擎盖下，这实际上也在执行一些步骤，我们将在本指南后面介绍这些步骤。

```code
query = "What did the president say about Ketanji Brown Jackson"
index.query(query)
```

```code
" The president said that Ketanji Brown Jackson is one of the nation's top legal minds, a former top litigator in private practice, a former federal public defender, and from a family of public school educators and police officers. He also said that she is a consensus builder and has received a broad range of support from the Fraternal Order of Police to former judges appointed by Democrats and Republicans."
```

```code
query = "What did the president say about Ketanji Brown Jackson"
index.query_with_sources(query)
```

```code
{'question': 'What did the president say about Ketanji Brown Jackson',
'answer': " The president said that he nominated Circuit Court of Appeals Judge Ketanji Brown Jackson, one of the nation's top legal minds, to continue Justice Breyer's legacy of excellence, and that she has received a broad range of support from the Fraternal Order of Police to former judges appointed by Democrats and Republicans.\n",
'sources': '../state_of_the_union.txt'}

```

从 VectorstoreIndexCreator 返回的是 VectorStoreIndexWrapper，它提供了这些优秀的查询和 query _ with _ source 功能。如果我们只是想直接访问向量存储，我们也可以这样做。

```code
index.vectorstore
```

```code
<langchain.vectorstores.chroma.Chroma at 0x119aa5940>
```

如果我们想要访问 VectorstoreRetriever，我们可以使用:

```code
index.vectorstore.as_retriever()
```

```code
VectorStoreRetriever(vectorstore=<langchain.vectorstores.chroma.Chroma object at 0x119aa5940>, search_kwargs={})

```

演练
好吧，到底是怎么回事? 这个索引是怎么创建的？
这个 VectorstoreIndexCreator 隐藏了很多魔力，这是在做什么？
加载文件后有三个主要步骤:

将文档分割成块

为每个文档创建嵌入

在向量库中存储文档和嵌入

让我们用代码来演示一下

```code
documents = loader.load()
```

接下来，我们将把文档分割成块。

```code

from langchain.text_splitter import CharacterTextSplitter
text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)
texts = text_splitter.split_documents(documents)

```

然后，我们将选择要使用的嵌入。

```code
from langchain.embeddings import OpenAIEmbeddings
embeddings = OpenAIEmbeddings()
```

现在我们创建用作索引的向量存储。

```code
from langchain.vectorstores import Chroma
db = Chroma.from_documents(texts, embeddings)
```

```code
Running Chroma using direct local API.
Using DuckDB in-memory for database. Data will be transient.
```

这就是创建索引的过程，然后，我们在一个检索接口中公开这个索引。

```code
retriever = db.as_retriever()
```

然后，像以前一样，我们创建一个链，并使用它来回答问题！

```code
qa = RetrievalQA.from_chain_type(llm=OpenAI(), chain_type="stuff", retriever=retriever)
```

```code
query = "What did the president say about Ketanji Brown Jackson"
qa.run(query)
```

```code
" The President said that Judge Ketanji Brown Jackson is one of the nation's top legal minds, a former top litigator in private practice, a former federal public defender, and from a family of public school educators and police officers. He said she is a consensus builder and has received a broad range of support from organizations such as the Fraternal Order of Police and former judges appointed by Democrats and Republicans."

```

VectorstoreIndexCreator 只是所有这些逻辑的包装器。它可以在它使用的文本分割器、它使用的嵌入以及它使用的向量存储中进行配置。例如，您可以按以下方式配置它:

```code
index_creator = VectorstoreIndexCreator(
vectorstore_cls=Chroma,
embedding=OpenAIEmbeddings(),
text_splitter=CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)
)

```

希望这能够突出显示 VectorstoreIndexCreator 的底层正在发生的事情。虽然我们认为用一种简单的方法来创建索引很重要，但是我们也认为了解底下发生了什么也很重要。索引（Indexes)文档加载器（Document Loaders）


文档加载器# – LangChain中文网

6大核心模块（Modules)索引（Indexes)文档加载器（Document Loaders）

文档加载器#

概念指南 (opens in a new tab)

将语言模型与自己的文本数据结合使用是区分它们的强大方式。
这样做的第一步是将数据加载到“文档”中-一种花哨的方式来说一些文本片段。
该模块旨在使这个过程变得容易。
这样做的主要驱动因素是Unstructured (opens in a new tab) Python 包。
该软件包是将所有类型的文件-文本、PowerPoint、图像、HTML、PDF 等-转换为文本数据的好方法。
有关如何设置 Unstructured 的详细说明，请参见此处 (opens in a new tab)的安装指南。
提供以下文档加载器:

Airbyte JSON

Apify Dataset

Arxiv

AWS S3 Directory

AWS S3 File

AZLyrics

Azure Blob Storage Container

Azure Blob Storage File

Bilibili

黑板

区块链

ChatGPT数据

大学机密

Confluence

CoNLL-U

复制粘贴

CSV

Diffbot

Discord

DuckDB

电子邮件

EPub

EverNote

Facebook 聊天

Figma

文件目录

Git

GitBook

Google BigQuery

Google Cloud存储目录

Google Cloud存储文件

Google Drive

Gutenberg

黑客新闻

HTML

HuggingFace 数据集

iFixit

图片

图片标题

IMSDb

JSON 文件

Jupyter教程

Markdown

MediaWiki转储

Microsoft OneDrive

Microsoft PowerPoint

Microsoft Word

现代财政

Notion DB 1/2

Notion DB 2/2

Obsidian

Pandas DataFrame

PDF

Using PyPDFium2

ReadTheDocs Documentation

Reddit

Roam

网站地图

Slack

Spreedly

Stripe

字幕

Telegram

TOML

推特

非结构化文件

网址

Selenium网址加载器

Playwright网址加载器

WebBase加载器

WhatsApp聊天记录

维基百科

YouTube剪辑

入门（Getting Started）Azure Blob Storage File


Azure Files – LangChain中文网

6大核心模块（Modules)索引（Indexes)文档加载器（Document Loaders）示例Azure Blob Storage File

Azure Files

Azure Files (opens in a new tab)是在云中提供的完全托管的文件共享，可通过行业标准的Server Message Block（
```code
SMB
```
)协议、Network File System (
```code
NFS
```
)协议和
```code
Azure Files REST API
```
进行访问。

本文介绍如何从Azure Files中加载文档对象。

```code
#!pip install azure-storage-blob

```

```code
from langchain.document_loaders import AzureBlobStorageFileLoader

```

```code
loader = AzureBlobStorageFileLoader(conn_str='<connection string>', container='<container name>', blob_name='<blob name>')

```

```code
loader.load()

```

```code
[Document(page_content='Lorem ipsum dolor sit amet.', lookup_str='', metadata={'source': '/var/folders/y6/8_bzdg295ld6s1_97_12m4lr0000gn/T/tmpxvave6wl/fake.docx'}, lookup_index=0)]

```
文档加载器（Document Loaders）Bigquery


BigQuery加载器  BigQuery Loader
# – LangChain中文网

6大核心模块（Modules)索引（Indexes)文档加载器（Document Loaders）示例Bigquery

BigQuery加载器  BigQuery Loader
#
每个文档的每行加载一个BigQuery的查询

```code
from langchain.document_loaders import BigQueryLoader

```

```code
BASE_QUERY = '''
SELECT
id,
dna_sequence,
organism
FROM (
SELECT
ARRAY (
SELECT
AS STRUCT 1 AS id, "ATTCGA" AS dna_sequence, "Lokiarchaeum sp. (strain GC14_75)." AS organism
UNION ALL
SELECT
AS STRUCT 2 AS id, "AGGCGA" AS dna_sequence, "Heimdallarchaeota archaeon (strain LC_2)." AS organism
UNION ALL
SELECT
AS STRUCT 3 AS id, "TCCGGA" AS dna_sequence, "Acidianus hospitalis (strain W1)." AS organism) AS new_array),
UNNEST(new_array)
'''

```

基础用法 Basic Usage
#

```code
loader = BigQueryLoader(BASE_QUERY)

data = loader.load()

```

```code
print(data)

```

```code
[Document(page_content='id: 1\ndna_sequence: ATTCGA\norganism: Lokiarchaeum sp. (strain GC14_75).', lookup_str='', metadata={}, lookup_index=0), Document(page_content='id: 2\ndna_sequence: AGGCGA\norganism: Heimdallarchaeota archaeon (strain LC_2).', lookup_str='', metadata={}, lookup_index=0), Document(page_content='id: 3\ndna_sequence: TCCGGA\norganism: Acidianus hospitalis (strain W1).', lookup_str='', metadata={}, lookup_index=0)]
```

指定哪些列是内容与元数据
#

```code
loader = BigQueryLoader(BASE_QUERY, page_content_columns=["dna_sequence", "organism"], metadata_columns=["id"])

data = loader.load()

```

```code
print(data)

```

```code
[Document(page_content='dna_sequence: ATTCGA\norganism: Lokiarchaeum sp. (strain GC14_75).', lookup_str='', metadata={'id': 1}, lookup_index=0), Document(page_content='dna_sequence: AGGCGA\norganism: Heimdallarchaeota archaeon (strain LC_2).', lookup_str='', metadata={'id': 2}, lookup_index=0), Document(page_content='dna_sequence: TCCGGA\norganism: Acidianus hospitalis (strain W1).', lookup_str='', metadata={'id': 3}, lookup_index=0)]
```

将源添加到元数据
#

```code
# Note that the `id` column is being returned twice, with one instance aliased as `source`
ALIASED_QUERY = '''
SELECT
id,
dna_sequence,
organism,
id as source
FROM (
SELECT
ARRAY (
SELECT
AS STRUCT 1 AS id, "ATTCGA" AS dna_sequence, "Lokiarchaeum sp. (strain GC14_75)." AS organism
UNION ALL
SELECT
AS STRUCT 2 AS id, "AGGCGA" AS dna_sequence, "Heimdallarchaeota archaeon (strain LC_2)." AS organism
UNION ALL
SELECT
AS STRUCT 3 AS id, "TCCGGA" AS dna_sequence, "Acidianus hospitalis (strain W1)." AS organism) AS new_array),
UNNEST(new_array)
'''

```

```code
loader = BigQueryLoader(ALIASED_QUERY, metadata_columns=["source"])

data = loader.load()

```

```code
print(data)

```

```code
[Document(page_content='id: 1\ndna_sequence: ATTCGA\norganism: Lokiarchaeum sp. (strain GC14_75).\nsource: 1', lookup_str='', metadata={'source': 1}, lookup_index=0), Document(page_content='id: 2\ndna_sequence: AGGCGA\norganism: Heimdallarchaeota archaeon (strain LC_2).\nsource: 2', lookup_str='', metadata={'source': 2}, lookup_index=0), Document(page_content='id: 3\ndna_sequence: TCCGGA\norganism: Acidianus hospitalis (strain W1).\nsource: 3', lookup_str='', metadata={'source': 3}, lookup_index=0)]
```
Azure Blob Storage FileBilibili


BiliBiliLoader – LangChain中文网

6大核心模块（Modules)索引（Indexes)文档加载器（Document Loaders）示例Bilibili

BiliBiliLoader
这个加载器使用 bilibili-api (opens in a new tab) 来从
```code
B站
```
获取文本转录。
使用这个 BiliBiliLoader，用户可以轻松地获取平台上他们所需的视频内容的文字转录。

```code
#!pip install bilibili-api

```

```code
from langchain.document_loaders.bilibili import BiliBiliLoader

```

```code
loader = BiliBiliLoader(
["https://www.bilibili.com/video/BV1xt411o7Xu/"]
)

```

```code
loader.load()

```
BigqueryBlackboard


Blackboard – LangChain中文网

6大核心模块（Modules)索引（Indexes)文档加载器（Document Loaders）示例Blackboard

Blackboard

Blackboard Learn (opens in a new tab)（以前称为Blackboard Learning Management System)是由Blackboard Inc.开发的基于Web的虚拟学习环境和学习管理系统。该软件具有课程管理、可定制的开放式架构和可扩展的设计，允许与学生信息系统和身份验证协议集成。它可以安装在本地服务器上，由
```code
Blackboard ASP Solutions
```
托管，或作为在Amazon Web Services上托管的软件服务提供。其主要目的是在传统上面对面传递的课程中增加在线元素，以及开发几乎没有面对面会议的完全在线课程。

本文介绍如何从Blackboard Learn (opens in a new tab)实例中加载数据。
此加载器不适用于所有
```code
Blackboard
```
课程。它只适用于使用新
```code
Blackboard
```
界面的课程。要使用此加载器，您必须具有BbRouter cookie。你可以通过登录课程，然后从浏览器的开发工具中复制BbRouter cookie的值来获取此Cookie。

```code
from langchain.document_loaders import BlackboardLoader

loader = BlackboardLoader(
blackboard_course_url="https://blackboard.example.com/webapps/blackboard/execute/announcement?method=search&context=course_entry&course_id=_123456_1",
bbrouter="expires:12345...",
load_all_recursively=True,
)
documents = loader.load()

```
BilibiliBlockchain


区块链
# – LangChain中文网

6大核心模块（Modules)索引（Indexes)文档加载器（Document Loaders）示例Blockchain

区块链
#
概述
#
这篇笔记的目的是为Langchain文档加载程序提供一个测试功能的手段，用于区块链。
最初，这个加载程序支持：

从NFT智能合约（ERC721和ERC1155)加载NFT作为文档
Ethereum Maninnet，Ethereum Testnet，Polgyon Mainnet，Polygon Testnet（默认为eth-mainnet)
Alchemy的getNFTsForCollection API

如果社区发现该加载程序有价值，它可以扩展。具体而言：

可以添加其他API（例如交易相关的API)

这个文档加载器需要：

免费的 Alchemy API Key (opens in a new tab)

输出采用以下格式：

```code
* pageContent=个体NFT
* metadata={'source': '0x1a92f7381b9f03921564a437210bb9396471050c'，'blockchain': 'eth-mainnet'，'tokenId': '0x15'}
```

将NFT加载到文档加载器中
#

```code
alchemyApiKey = "get from https://www.alchemy.com/ and set in environment variable ALCHEMY_API_KEY"

```

选项一: Ethereum Mainnet (默认是BlockchainType) #

```code
contractAddress = "0xbc4ca0eda7647a8ab7c2061c2e118a18a936f13d" # Bored Ape Yacht Club contract address

blockchainType = BlockchainType.ETH_MAINNET  #default value, optional parameter

blockchainLoader = BlockchainDocumentLoader(contract_address=contractAddress,
api_key=alchemyApiKey)

nfts = blockchainLoader.load()

nfts[:2]

```

选项二:  Polygon Mainnet #

```code
contractAddress = "0x448676ffCd0aDf2D85C1f0565e8dde6924A9A7D9" # Polygon Mainnet contract address

blockchainType = BlockchainType.POLYGON_MAINNET

blockchainLoader = BlockchainDocumentLoader(contract_address=contractAddress,
blockchainType=blockchainType,
api_key=alchemyApiKey)

nfts = blockchainLoader.load()

nfts[:2]

```
BlackboardChatgpt Loader


ChatGPT 数据# – LangChain中文网

6大核心模块（Modules)索引（Indexes)文档加载器（Document Loaders）示例Chatgpt Loader

ChatGPT 数据#

ChatGPT (opens in a new tab) 是由OpenAI开发的人工智能（AI)聊天机器人。

本笔记涵盖了如何从您的
```code
ChatGPT
```
数据导出文件夹中加载
```code
conversations.json
```
。
您可以通过以下步骤通过电子邮件获取您的数据导出：https://chat.openai.com/ (opens in a new tab) ->（个人资料)-设置 -> 导出数据 -> 确认导出。

```code
from langchain.document_loaders.chatgpt import ChatGPTLoader

```

```code
loader = ChatGPTLoader(log_file='./example_data/fake_conversations.json', num_logs=1)

```

```code
loader.load()

```

```code
[Document(page_content="AI Overlords - AI on 2065-01-24 05:20:50: Greetings, humans. I am Hal 9000. You can trust me completely.  AI Overlords - human on 2065-01-24 05:21:20: Nice to meet you, Hal. I hope you won't develop a mind of your own.  ", metadata={'source': './example_data/fake_conversations.json'})]

```
BlockchainCollege Confidential