Markdown – LangChain中文网

6大核心模块（Modules)索引（Indexes)文档加载器（Document Loaders）示例Markdown

Markdown

Markdown 是一种轻量级标记语言，用于使用纯文本编辑器创建格式化文本。

本文介绍如何将Markdown文档加载到我们可以向下使用的文档格式中。

```code
from langchain.document_loaders import UnstructuredMarkdownLoader

```

```code
loader = UnstructuredMarkdownLoader("../../../../README.md")

```

```code
data = loader.load()

```

```code
data

```

```code
[Document(page_content="ð\x9f¦\x9cï¸\x8fð\x9f”\x97 LangChain\n\nâ\x9a¡ Building applications with LLMs through composability â\x9a¡\n\nProduction Support: As you move your LangChains into production, we'd love to offer more comprehensive support.\nPlease fill out this form and we'll set up a dedicated support Slack channel.\n\nQuick Install\n\npip install langchain\n\nð\x9f¤” What is this?\n\nLarge language models (LLMs) are emerging as a transformative technology, enabling\ndevelopers to build applications that they previously could not.\nBut using these LLMs in isolation is often not enough to\ncreate a truly powerful app - the real power comes when you can combine them with other sources of computation or knowledge.\n\nThis library is aimed at assisting in the development of those types of applications. Common examples of these types of applications include:\n\nâ\x9d“ Question Answering over specific documents\n\nDocumentation\n\nEnd-to-end Example: Question Answering over Notion Database\n\nð\x9f’¬ Chatbots\n\nDocumentation\n\nEnd-to-end Example: Chat-LangChain\n\nð\x9f¤\x96 Agents\n\nDocumentation\n\nEnd-to-end Example: GPT+WolframAlpha\n\nð\x9f“\x96 Documentation\n\nPlease see here for full documentation on:\n\nGetting started (installation, setting up the environment, simple examples)\n\nHow-To examples (demos, integrations, helper functions)\n\nReference (full API docs)\n  Resources (high-level explanation of core concepts)\n\nð\x9f\x9a\x80 What can this help with?\n\nThere are six main areas that LangChain is designed to help with.\nThese are, in increasing order of complexity:\n\nð\x9f“\x83 LLMs and Prompts:\n\nThis includes prompt management, prompt optimization, generic interface for all LLMs, and common utilities for working with LLMs.\n\nð\x9f”\x97 Chains:\n\nChains go beyond just a single LLM call, and are sequences of calls (whether to an LLM or a different utility). LangChain provides a standard interface for chains, lots of integrations with other tools, and end-to-end chains for common applications.\n\nð\x9f“\x9a Data Augmented Generation:\n\nData Augmented Generation involves specific types of chains that first interact with an external datasource to fetch data to use in the generation step. Examples of this include summarization of long pieces of text and question/answering over specific data sources.\n\nð\x9f¤\x96 Agents:\n\nAgents involve an LLM making decisions about which Actions to take, taking that Action, seeing an Observation, and repeating that until done. LangChain provides a standard interface for agents, a selection of agents to choose from, and examples of end to end agents.\n\nð\x9f§\xa0 Memory:\n\nMemory is the concept of persisting state between calls of a chain/agent. LangChain provides a standard interface for memory, a collection of memory implementations, and examples of chains/agents that use memory.\n\nð\x9f§\x90 Evaluation:\n\n[BETA] Generative models are notoriously hard to evaluate with traditional metrics. One new way of evaluating them is using language models themselves to do the evaluation. LangChain provides some prompts/chains for assisting in this.\n\nFor more information on these concepts, please see our full documentation.\n\nð\x9f’\x81 Contributing\n\nAs an open source project in a rapidly developing field, we are extremely open to contributions, whether it be in the form of a new feature, improved infra, or better documentation.\n\nFor detailed information on how to contribute, see here.", lookup_str='', metadata={'source': '../../../../README.md'}, lookup_index=0)]

```

保留元素 Retain Elements
#
在幕后，非结构化为不同的文本块创建不同的“元素”。
默认情况下，我们将它们组合在一起，但您可以通过指定
```code
mode="elements"
```
轻松保持这种分离。
```code
mode="elements"
```

.

```code
loader = UnstructuredMarkdownLoader("../../../../README.md", mode="elements")

```

```code
data = loader.load()

```

```code
data[0]

```

```code
Document(page_content='ð\x9f¦\x9cï¸\x8fð\x9f”\x97 LangChain', lookup_str='', metadata={'source': '../../../../README.md', 'page_number': 1, 'category': 'UncategorizedText'}, lookup_index=0)

```
ImsdbNotebook


Notebook – LangChain中文网

6大核心模块（Modules)索引（Indexes)文档加载器（Document Loaders）示例Notebook

Notebook
本文介绍如何将.ipynb教程中的数据加载到适合LangChain使用的格式中。

```code
from langchain.document_loaders import NotebookLoader

```

```code
loader = NotebookLoader("example_data/notebook.ipynb", include_outputs=True, max_output_length=20, remove_newline=True)

```

```code
NotebookLoader.load()
```
将
```code
.ipynb
```
笔记本文件加载到一个
```code
Document
```
对象中。
参数 :

```code
include_outputs
```

（bool）：是否将单元格输出包含在结果文档中（默认为假）。

```code
max_output_length
```

（int）：要从每个单元格输出中包括的字符的最大数量（默认为10）。

```code
remove_newline
```

（bool）：是否从单元格源和输出中删除换行符（默认为假）。

```code
traceback
```

（bool）：是否包含完整的回溯（默认为假）。

```code
loader.load()

```

```code
[Document(page_content='\'markdown\' cell: \'[\'# Notebook\', \'\', \'This notebook covers how to load data from an .ipynb notebook into a format suitable by LangChain.\']\'\n\n \'code\' cell: \'[\'from langchain.document_loaders import NotebookLoader\']\'\n\n \'code\' cell: \'[\'loader = NotebookLoader("example_data/notebook.ipynb")\']\'\n\n \'markdown\' cell: \'[\'`NotebookLoader.load()` loads the `.ipynb` notebook file into a `Document` object.\', \'\', \'**Parameters**:\', \'\', \'* `include_outputs` (bool): whether to include cell outputs in the resulting document (default is False).\', \'* `max_output_length` (int): the maximum number of characters to include from each cell output (default is 10).\', \'* `remove_newline` (bool): whether to remove newline characters from the cell sources and outputs (default is False).\', \'* `traceback` (bool): whether to include full traceback (default is False).\']\'\n\n \'code\' cell: \'[\'loader.load(include_outputs=True, max_output_length=20, remove_newline=True)\']\'\n\n', lookup_str='', metadata={'source': 'example_data/notebook.ipynb'}, lookup_index=0)]

```
MarkdownNotion


Notion – LangChain中文网

6大核心模块（Modules)索引（Indexes)文档加载器（Document Loaders）示例Notion

Notion
本文介绍如何从Notion数据库转储中加载文档。
要获取这个Notion转储，请按照以下说明操作：
🧑 自定义数据集的说明
从Notion导出数据集，您可以通过点击右上角的三个点然后点击 “导出” 来完成。
在导出时，请确保选择“Markdown和CSV”格式选项。
这将在您的下载文件夹中生成一个.zip文件。将.zip文件移动到该存储库中。
运行以下命令解压缩zip文件（根据需要替换“Export…”为您自己的文件名)。

```code
unzip Export-d3adfe0f-3131-4bf3-8987-a52017fc1bae.zip -d Notion_DB

```

运行以下命令摄取数据。

```code
from langchain.document_loaders import NotionDirectoryLoader
loader = NotionDirectoryLoader("Notion_DB")
docs = loader.load()
```
NotebookNotiondb


Notion数据库加载器 – LangChain中文网

6大核心模块（Modules)索引（Indexes)文档加载器（Document Loaders）示例Notiondb

Notion数据库加载器
NotionDBLoader是一个Python类，用于从Notion数据库中加载内容。它从数据库中检索页面，读取其内容，并返回Document对象的列表。
要求：

Notion数据库
Notion集成令牌

设置：
1. 创建Notion表数据库
在Notion中创建一个新的表数据库。可以添加任何列到数据库中，并将它们视为元数据。例如，可以添加以下列：

标题：将标题设置为默认属性。
类别：一个多选属性，用于存储与页面相关联的类别。
关键词：一个多选属性，用于存储与页面相关联的关键词。

将内容添加到数据库中每个页面的正文中。NotionDBLoader将从这些页面中提取内容和元数据。
2. 创建Notion集成
创建Notion集成，按照以下步骤操作：

访问Notion开发人员页面并使用Notion帐户登录。
点击“+新集成”按钮。
命名集成，并选择数据库所在的工作区。
选择所需的能力，此扩展名仅需读取内容能力。
点击“提交”按钮以创建集成。

集成创建后，您将获得一个集成令牌（API密钥)。复制此令牌并保持安全，因为您将需要它来使用NotionDBLoader。
3. 将集成连接到数据库
要将您的集成连接到数据库，请按照以下步骤操作：

在Notion中打开数据库。
单击数据库视图右上角的三点菜单图标。
单击“+ 新集成”按钮。
找到您的集成，您可能需要开始在搜索框中输入其名称。
单击“连接”按钮将集成与数据库连接。

4. 获取数据库ID
要获取数据库ID，请按照以下步骤操作：

在Notion中打开数据库。
单击数据库视图右上角的三点菜单图标。
从菜单中选择“复制链接”以将数据库URL复制到剪贴板中。
数据库ID是在URL中找到的长串字母数字字符。它通常看起来像这样：
```code
https://www.notion.so/username/8935f9d140a04f95a872520c4f123456?v=…
```
。在这个例子中，数据库ID是8935f9d140a04f95a872520c4f123456。

具有正确设置的数据库和已经获得的集成令牌和数据库ID，现在可以使用NotionDBLoader代码从Notion数据库中加载内容和元数据。
用法：
NotionDBLoader是langchain包的文档加载器的一部分。可以按照以下方式使用它：

```code
from getpass import getpass
NOTION_TOKEN = getpass()
DATABASE_ID = getpass()

```

```code
········
········

```

```code
from langchain.document_loaders import NotionDBLoader

```

```code
loader = NotionDBLoader(NOTION_TOKEN, DATABASE_ID)

```

```code
docs = loader.load()

```

```code
print(docs)

```

```code

```
NotionObsidian


Obsidian – LangChain中文网

6大核心模块（Modules)索引（Indexes)文档加载器（Document Loaders）示例Obsidian

Obsidian
本文介绍如何从Obsidian数据库中加载文档。
由于Obsidian只作为Markdown文件夹存储在磁盘上，因此加载器只需要取一个指向此目录的路径。
Obsidian文件有时还包含元数据，即文件顶部的YAML块。这些值将添加到文档的元数据中。（
```code
ObsidianLoader
```
还可以传递
```code
collect_metadata = False
```
参数以禁用此行为。)
用法：

```code
from langchain.document_loaders import ObsidianLoader
loader = ObsidianLoader("<path-to-obsidian>")
docs = loader.load()
```
NotiondbPdf


使用PyPDFium2# – LangChain中文网

6大核心模块（Modules)索引（Indexes)文档加载器（Document Loaders）示例Pdf

PDF#

便携式文档格式（PDF) (opens in a new tab)，标准化为ISO 32000，是Adobe于1992年开发的一种文件格式，用于以与应用软件、硬件和操作系统无关的方式呈现文档，包括文本格式和图像。

这涵盖了如何将
```code
PDF
```
文档加载到我们下游使用的文档格式中。
使用PyPDF#
使用
```code
pypdf
```
将PDF加载到文档数组中，其中每个文档包含页面内容和元数据，包括
```code
page
```
页数。

```code
!pip install pypdf

```

```code
from langchain.document_loaders import PyPDFLoader

loader = PyPDFLoader("example_data/layout-parser-paper.pdf")
pages = loader.load_and_split()

```

```code
pages[0]

```

```code
Document(page_content='LayoutParser : A Uniﬁed Toolkit for Deep  Learning Based Document Image Analysis  Zejiang Shen1( \x00), Ruochen Zhang2, Melissa Dell3, Benjamin Charles Germain  Lee4, Jacob Carlson3, and Weining Li5  1Allen Institute for AI  shannons@allenai.org  2Brown University  ruochen zhang@brown.edu  3Harvard University  fmelissadell,jacob carlson g@fas.harvard.edu  4University of Washington  bcgl@cs.washington.edu  5University of Waterloo  w422li@uwaterloo.ca  Abstract. Recent advances in document image analysis (DIA) have been  primarily driven by the application of neural networks. Ideally, research  outcomes could be easily deployed in production and extended for further  investigation. However, various factors like loosely organized codebases  and sophisticated model con\x0cgurations complicate the easy reuse of im-  portant innovations by a wide audience. Though there have been on-going  e\x0borts to improve reusability and simplify deep learning (DL) model  development in disciplines like natural language processing and computer  vision, none of them are optimized for challenges in the domain of DIA.  This represents a major gap in the existing toolkit, as DIA is central to  academic research across a wide range of disciplines in the social sciences  and humanities. This paper introduces LayoutParser , an open-source  library for streamlining the usage of DL in DIA research and applica-  tions. The core LayoutParser library comes with a set of simple and  intuitive interfaces for applying and customizing DL models for layout de-  tection, character recognition, and many other document processing tasks.  To promote extensibility, LayoutParser also incorporates a community  platform for sharing both pre-trained models and full document digiti-  zation pipelines. We demonstrate that LayoutParser is helpful for both  lightweight and large-scale digitization pipelines in real-word use cases.  The library is publicly available at https://layout-parser.github.io .  Keywords: Document Image Analysis ·Deep Learning ·Layout Analysis  ·Character Recognition ·Open Source library ·Toolkit.  1 Introduction  Deep Learning(DL)-based approaches are the state-of-the-art for a wide range of  document image analysis (DIA) tasks including document image classi\x0ccation [ 11,arXiv:2103.15348v2  [cs.CV]  21 Jun 2021', metadata={'source': 'example_data/layout-parser-paper.pdf', 'page': 0})

```

这种方法的优点是可以通过页数检索文档。
我们想使用
```code
OpenAIEmbeddings
```
，所以必须获取OpenAI API密钥。

```code
import os
import getpass

os.environ['OPENAI_API_KEY'] = getpass.getpass('OpenAI API Key:')

```

```code
from langchain.vectorstores import FAISS
from langchain.embeddings.openai import OpenAIEmbeddings

faiss_index = FAISS.from_documents(pages, OpenAIEmbeddings())
docs = faiss_index.similarity_search("How will the community be engaged?", k=2)
for doc in docs:
print(str(doc.metadata["page"]) + ":", doc.page_content[:300])

```

```code
9: 10 Z. Shen et al.
Fig. 4: Illustration of (a) the original historical Japanese document with layout
detection results and (b) a recreated version of the document image that achieves
much better character recognition recall. The reorganization algorithm rearranges
the tokens based on the their detect
3: 4 Z. Shen et al.
Efficient Data AnnotationC u s t o m i z e d  M o d e l  T r a i n i n gModel Cust omizationDI A Model HubDI A Pipeline SharingCommunity PlatformLa y out Detection ModelsDocument Images
T h e  C o r e  L a y o u t P a r s e r  L i b r a r yOCR ModuleSt or age & VisualizationLa y ou

```

使用MathPix#
受 Daniel Gross 的启发，(https://gist.github.com/danielgross/3ab4104e14faccc12b49200843adab21 (opens in a new tab))

```code
from langchain.document_loaders import MathpixPDFLoader

```

```code
loader = MathpixPDFLoader("example_data/layout-parser-paper.pdf")

```

```code
data = loader.load()

```

使用非结构化数据#

```code
from langchain.document_loaders import UnstructuredPDFLoader

```

```code
loader = UnstructuredPDFLoader("example_data/layout-parser-paper.pdf")

```

```code
data = loader.load()

```

保留元素#
在幕后，非结构化数据为不同的文本块创建不同的“元素”。默认情况下，我们会将它们组合在一起，但是您可以通过指定
```code
mode="elements"
```
来轻松保留它们之间的分隔。

```code
loader = UnstructuredPDFLoader("example_data/layout-parser-paper.pdf", mode="elements")

```

```code
data = loader.load()

```

```code
data[0]

```

```code
Document(page_content='LayoutParser: A Uniﬁed Toolkit for Deep  Learning Based Document Image Analysis  Zejiang Shen1 (�), Ruochen Zhang2, Melissa Dell3, Benjamin Charles Germain  Lee4, Jacob Carlson3, and Weining Li5  1 Allen Institute for AI  shannons@allenai.org  2 Brown University  ruochen zhang@brown.edu  3 Harvard University  {melissadell,jacob carlson}@fas.harvard.edu  4 University of Washington  bcgl@cs.washington.edu  5 University of Waterloo  w422li@uwaterloo.ca  Abstract. Recent advances in document image analysis (DIA) have been  primarily driven by the application of neural networks. Ideally, research  outcomes could be easily deployed in production and extended for further  investigation. However, various factors like loosely organized codebases  and sophisticated model conﬁgurations complicate the easy reuse of im-  portant innovations by a wide audience. Though there have been on-going  eﬀorts to improve reusability and simplify deep learning (DL) model  development in disciplines like natural language processing and computer  vision, none of them are optimized for challenges in the domain of DIA.  This represents a major gap in the existing toolkit, as DIA is central to  academic research across a wide range of disciplines in the social sciences  and humanities. This paper introduces LayoutParser, an open-source  library for streamlining the usage of DL in DIA research and applica-  tions. The core LayoutParser library comes with a set of simple and  intuitive interfaces for applying and customizing DL models for layout de-  tection, character recognition, and many other document processing tasks.  To promote extensibility, LayoutParser also incorporates a community  platform for sharing both pre-trained models and full document digiti-  zation pipelines. We demonstrate that LayoutParser is helpful for both  lightweight and large-scale digitization pipelines in real-word use cases.  The library is publicly available at https://layout-parser.github.io.  Keywords: Document Image Analysis · Deep Learning · Layout Analysis  · Character Recognition · Open Source library · Toolkit.  1  Introduction  Deep Learning(DL)-based approaches are the state-of-the-art for a wide range of  document image analysis (DIA) tasks including document image classiﬁcation [11,  arXiv:2103.15348v2  [cs.CV]  21 Jun 2021  ', lookup_str='', metadata={'file_path': 'example_data/layout-parser-paper.pdf', 'page_number': 1, 'total_pages': 16, 'format': 'PDF 1.5', 'title': '', 'author': '', 'subject': '', 'keywords': '', 'creator': 'LaTeX with hyperref', 'producer': 'pdfTeX-1.40.21', 'creationDate': 'D:20210622012710Z', 'modDate': 'D:20210622012710Z', 'trapped': '', 'encryption': None}, lookup_index=0)

```

使用非结构化数据获取远程 PDF#
介绍如何将在线 PDF 加载到我们可以在下游使用的文档格式中。这可用于各种在线 PDF 站点，例如 https://open.umn.edu/opentextbooks/textbooks/ (opens in a new tab) 和 https://arxiv.org/archive/ (opens in a new tab)
注意：所有其他pdf加载程序也可以用于获取远程pdf，但是
```code
OnlinePDFLoader
```
是一个旧的函数，专门用于
```code
UnstructuredPDFLoader
```
。

```code
from langchain.document_loaders import OnlinePDFLoader

```

```code
loader = OnlinePDFLoader("https://arxiv.org/pdf/2302.03803.pdf")

```

```code
data = loader.load()

```

```code
print(data)

```

```code
[Document(page_content='A WEAK ( k, k ) -LEFSCHETZ THEOREM FOR PROJECTIVE TORIC ORBIFOLDS  William D. Montoya  Instituto de Matem´atica, Estat´ıstica e Computa¸c˜ao Cient´ıﬁca,  In [3] we proved that, under suitable conditions, on a very general codimension s quasi- smooth intersection subvariety X in a projective toric orbifold P d Σ with d + s = 2 ( k + 1 ) the Hodge conjecture holds, that is, every ( p, p ) -cohomology class, under the Poincar´e duality is a rational linear combination of fundamental classes of algebraic subvarieties of X . The proof of the above-mentioned result relies, for p ≠ d + 1 − s , on a Lefschetz  Keywords: (1,1)- Lefschetz theorem, Hodge conjecture, toric varieties, complete intersection Email: wmontoya@ime.unicamp.br  theorem ([7]) and the Hard Lefschetz theorem for projective orbifolds ([11]). When p = d + 1 − s the proof relies on the Cayley trick, a trick which associates to X a quasi-smooth hypersurface Y in a projective vector bundle, and the Cayley Proposition (4.3) which gives an isomorphism of some primitive cohomologies (4.2) of X and Y . The Cayley trick, following the philosophy of Mavlyutov in [7], reduces results known for quasi-smooth hypersurfaces to quasi-smooth intersection subvarieties. The idea in this paper goes the other way around, we translate some results for quasi-smooth intersection subvarieties to  Acknowledgement. I thank Prof. Ugo Bruzzo and Tiago Fonseca for useful discus- sions. I also acknowledge support from FAPESP postdoctoral grant No. 2019/23499-7.  Let M be a free abelian group of rank d , let N = Hom ( M, Z ) , and N R = N ⊗ Z R .  if there exist k linearly independent primitive elements e  , . . . , e k ∈ N such that σ = { µ  e  + ⋯ + µ k e k } . • The generators e i are integral if for every i and any nonnegative rational number µ the product µe i is in N only if µ is an integer. • Given two rational simplicial cones σ , σ ′ one says that σ ′ is a face of σ ( σ ′ < σ ) if the set of integral generators of σ ′ is a subset of the set of integral generators of σ . • A ﬁnite set Σ = { σ  , . . . , σ t } of rational simplicial cones is called a rational simplicial complete d -dimensional fan if:  all faces of cones in Σ are in Σ ;  if σ, σ ′ ∈ Σ then σ ∩ σ ′ < σ and σ ∩ σ ′ < σ ′ ;  N R = σ  ∪ ⋅ ⋅ ⋅ ∪ σ t .  A rational simplicial complete d -dimensional fan Σ deﬁnes a d -dimensional toric variety P d Σ having only orbifold singularities which we assume to be projective. Moreover, T ∶ = N ⊗ Z C ∗ ≃ ( C ∗ ) d is the torus action on P d Σ . We denote by Σ ( i ) the i -dimensional cones  For a cone σ ∈ Σ, ˆ σ is the set of 1-dimensional cone in Σ that are not contained in σ  and x ˆ σ ∶ = ∏ ρ ∈ ˆ σ x ρ is the associated monomial in S .  Deﬁnition 2.2. The irrelevant ideal of P d Σ is the monomial ideal B Σ ∶ =< x ˆ σ ∣ σ ∈ Σ > and the zero locus Z ( Σ ) ∶ = V ( B Σ ) in the aﬃne space A d ∶ = Spec ( S ) is the irrelevant locus.  Proposition 2.3 (Theorem 5.1.11 [5]) . The toric variety P d Σ is a categorical quotient A d ∖ Z ( Σ ) by the group Hom ( Cl ( Σ ) , C ∗ ) and the group action is induced by the Cl ( Σ ) - grading of S .  Now we give a brief introduction to complex orbifolds and we mention the needed theorems for the next section. Namely: de Rham theorem and Dolbeault theorem for complex orbifolds.  Deﬁnition 2.4. A complex orbifold of complex dimension d is a singular complex space whose singularities are locally isomorphic to quotient singularities C d / G , for ﬁnite sub- groups G ⊂ Gl ( d, C ) .  Deﬁnition 2.5. A diﬀerential form on a complex orbifold Z is deﬁned locally at z ∈ Z as a G -invariant diﬀerential form on C d where G ⊂ Gl ( d, C ) and Z is locally isomorphic to d  Roughly speaking the local geometry of orbifolds reduces to local G -invariant geometry.  We have a complex of diﬀerential forms ( A ● ( Z ) , d ) and a double complex ( A ● , ● ( Z ) , ∂, ¯ ∂ ) of bigraded diﬀerential forms which deﬁne the de Rham and the Dolbeault cohomology groups (for a ﬁxed p ∈ N ) respectively:  (1,1)-Lefschetz theorem for projective toric orbifolds  Deﬁnition 3.1. A subvariety X ⊂ P d Σ is quasi-smooth if V ( I X ) ⊂ A #Σ ( 1 ) is smooth outside  Example 3.2 . Quasi-smooth hypersurfaces or more generally quasi-smooth intersection sub-  Example 3.2 . Quasi-smooth hypersurfaces or more generally quasi-smooth intersection sub- varieties are quasi-smooth subvarieties (see [2] or [7] for more details).  Remark 3.3 . Quasi-smooth subvarieties are suborbifolds of P d Σ in the sense of Satake in [8]. Intuitively speaking they are subvarieties whose only singularities come from the ambient  Proof. From the exponential short exact sequence  we have a long exact sequence in cohomology  H 1 (O ∗ X ) → H 2 ( X, Z ) → H 2 (O X ) ≃ H 0 , 2 ( X )  where the last isomorphisms is due to Steenbrink in [9]. Now, it is enough to prove the commutativity of the next diagram  where the last isomorphisms is due to Steenbrink in [9]. Now,  H 2 ( X, Z ) / / H 2 ( X, O X ) ≃ Dolbeault H 2 ( X, C ) deRham ≃ H 2 dR ( X, C ) / / H 0 , 2 ¯ ∂ ( X )  of the proof follows as the ( 1 , 1 ) -Lefschetz theorem in [6].  Remark 3.5 . For k = 1 and P d Σ as the projective space, we recover the classical ( 1 , 1 ) - Lefschetz theorem.  By the Hard Lefschetz Theorem for projective orbifolds (see [11] for details) we  By the Hard Lefschetz Theorem for projective orbifolds (see [11] for details) we get an isomorphism of cohomologies :  given by the Lefschetz morphism and since it is a morphism of Hodge structures, we have:  H 1 , 1 ( X, Q ) ≃ H dim X − 1 , dim X − 1 ( X, Q )  Corollary 3.6. If the dimension of X is 1 , 2 or 3 . The Hodge conjecture holds on X  Proof. If the dim C X = 1 the result is clear by the Hard Lefschetz theorem for projective orbifolds. The dimension 2 and 3 cases are covered by Theorem 3.5 and the Hard Lefschetz.  Cayley trick and Cayley proposition  The Cayley trick is a way to associate to a quasi-smooth intersection subvariety a quasi- smooth hypersurface. Let L 1 , . . . , L s be line bundles on P d Σ and let π ∶ P ( E ) → P d Σ be the projective space bundle associated to the vector bundle E = L 1 ⊕ ⋯ ⊕ L s . It is known that P ( E ) is a ( d + s − 1 ) -dimensional simplicial toric variety whose fan depends on the degrees of the line bundles and the fan Σ. Furthermore, if the Cox ring, without considering the grading, of P d Σ is C [ x 1 , . . . , x m ] then the Cox ring of P ( E ) is  Moreover for X a quasi-smooth intersection subvariety cut oﬀ by f 1 , . . . , f s with deg ( f i ) = [ L i ] we relate the hypersurface Y cut oﬀ by F = y 1 f 1 + ⋅ ⋅ ⋅ + y s f s which turns out to be quasi-smooth. For more details see Section 2 in [7].  We will denote P ( E ) as P d + s − 1 Σ ,X to keep track of its relation with X and P d Σ .  The following is a key remark.  Remark 4.1 . There is a morphism ι ∶ X → Y ⊂ P d + s − 1 Σ ,X . Moreover every point z ∶ = ( x, y ) ∈ Y with y ≠ 0 has a preimage. Hence for any subvariety W = V ( I W ) ⊂ X ⊂ P d Σ there exists W ′ ⊂ Y ⊂ P d + s − 1 Σ ,X such that π ( W ′ ) = W , i.e., W ′ = { z = ( x, y ) ∣ x ∈ W } .  For X ⊂ P d Σ a quasi-smooth intersection variety the morphism in cohomology induced by the inclusion i ∗ ∶ H d − s ( P d Σ , C ) → H d − s ( X, C ) is injective by Proposition 1.4 in [7].  Deﬁnition 4.2. The primitive cohomology of H d − s prim ( X ) is the quotient H d − s ( X, C )/ i ∗ ( H d − s ( P d Σ , C )) and H d − s prim ( X, Q ) with rational coeﬃcients.  H d − s ( P d Σ , C ) and H d − s ( X, C ) have pure Hodge structures, and the morphism i ∗ is com- patible with them, so that H d − s prim ( X ) gets a pure Hodge structure.  The next Proposition is the Cayley proposition.  Proposition 4.3. [Proposition 2.3 in [3] ] Let X = X 1 ∩⋅ ⋅ ⋅∩ X s be a quasi-smooth intersec- tion subvariety in P d Σ cut oﬀ by homogeneous polynomials f 1 . . . f s . Then for p ≠ d + s − 1 2 , d + s − 3 2  Remark 4.5 . The above isomorphisms are also true with rational coeﬃcients since H ● ( X, C ) = H ● ( X, Q ) ⊗ Q C . See the beginning of Section 7.1 in [10] for more details.  Theorem 5.1. Let Y = { F = y 1 f 1 + ⋯ + y k f k = 0 } ⊂ P 2 k + 1 Σ ,X be the quasi-smooth hypersurface associated to the quasi-smooth intersection surface X = X f 1 ∩ ⋅ ⋅ ⋅ ∩ X f k ⊂ P k + 2 Σ . Then on Y the Hodge conjecture holds.  the Hodge conjecture holds.  Proof. If H k,k prim ( X, Q ) = 0 we are done. So let us assume H k,k prim ( X, Q ) ≠ 0. By the Cayley proposition H k,k prim ( Y, Q ) ≃ H 1 , 1 prim ( X, Q ) and by the ( 1 , 1 ) -Lefschetz theorem for projective  toric orbifolds there is a non-zero algebraic basis λ C 1 , . . . , λ C n with rational coeﬃcients of H 1 , 1 prim ( X, Q ) , that is, there are n ∶ = h 1 , 1 prim ( X, Q ) algebraic curves C 1 , . . . , C n in X such that under the Poincar´e duality the class in homology [ C i ] goes to λ C i , [ C i ] ↦ λ C i . Recall that the Cox ring of P k + 2 is contained in the Cox ring of P 2 k + 1 Σ ,X without considering the grading. Considering the grading we have that if α ∈ Cl ( P k + 2 Σ ) then ( α, 0 ) ∈ Cl ( P 2 k + 1 Σ ,X ) . So the polynomials deﬁning C i ⊂ P k + 2 Σ can be interpreted in P 2 k + 1 X, Σ but with diﬀerent degree. Moreover, by Remark 4.1 each C i is contained in Y = { F = y 1 f 1 + ⋯ + y k f k = 0 } and  furthermore it has codimension k .  Claim: { C i } ni = 1 is a basis of prim ( ) . It is enough to prove that λ C i is diﬀerent from zero in H k,k prim ( Y, Q ) or equivalently that the cohomology classes { λ C i } ni = 1 do not come from the ambient space. By contradiction, let us assume that there exists a j and C ⊂ P 2 k + 1 Σ ,X such that λ C ∈ H k,k ( P 2 k + 1 Σ ,X , Q ) with i ∗ ( λ C ) = λ C j or in terms of homology there exists a ( k + 2 ) -dimensional algebraic subvariety V ⊂ P 2 k + 1 Σ ,X such that V ∩ Y = C j so they are equal as a homology class of P 2 k + 1 Σ ,X ,i.e., [ V ∩ Y ] = [ C j ] . It is easy to check that π ( V ) ∩ X = C j as a subvariety of P k + 2 Σ where π ∶ ( x, y ) ↦ x . Hence [ π ( V ) ∩ X ] = [ C j ] which is equivalent to say that λ C j comes from P k + 2 Σ which contradicts the choice of [ C j ] .  Remark 5.2 . Into the proof of the previous theorem, the key fact was that on X the Hodge conjecture holds and we translate it to Y by contradiction. So, using an analogous argument we have:  argument we have:  Proposition 5.3. Let Y = { F = y 1 f s +⋯+ y s f s = 0 } ⊂ P 2 k + 1 Σ ,X be the quasi-smooth hypersurface associated to a quasi-smooth intersection subvariety X = X f 1 ∩ ⋅ ⋅ ⋅ ∩ X f s ⊂ P d Σ such that d + s = 2 ( k + 1 ) . If the Hodge conjecture holds on X then it holds as well on Y .  Corollary 5.4. If the dimension of Y is 2 s − 1 , 2 s or 2 s + 1 then the Hodge conjecture holds on Y .  Proof. By Proposition 5.3 and Corollary 3.6.  [  ] Angella, D. Cohomologies of certain orbifolds. Journal of Geometry and Physics  (  ),  –  [  ] Batyrev, V. V., and Cox, D. A. On the Hodge structure of projective hypersur- faces in toric varieties. Duke Mathematical Journal  ,  (Aug  ). [  ] Bruzzo, U., and Montoya, W. On the Hodge conjecture for quasi-smooth in- tersections in toric varieties. S˜ao Paulo J. Math. Sci. Special Section: Geometry in Algebra and Algebra in Geometry (  ). [  ] Caramello Jr, F. C. Introduction to orbifolds. a  iv:  v  (  ). [  ] Cox, D., Little, J., and Schenck, H. Toric varieties, vol.  American Math- ematical Soc.,  [  ] Griffiths, P., and Harris, J. Principles of Algebraic Geometry. John Wiley & Sons, Ltd,  [  ] Mavlyutov, A. R. Cohomology of complete intersections in toric varieties. Pub- lished in Paciﬁc J. of Math.  No.  (  ),  –  [  ] Satake, I. On a Generalization of the Notion of Manifold. Proceedings of the National Academy of Sciences of the United States of America  ,  (  ),  –  [  ] Steenbrink, J. H. M. Intersection form for quasi-homogeneous singularities. Com- positio Mathematica  ,  (  ),  –  [  ] Voisin, C. Hodge Theory and Complex Algebraic Geometry I, vol.  of Cambridge Studies in Advanced Mathematics . Cambridge University Press,  [  ] Wang, Z. Z., and Zaffran, D. A remark on the Hard Lefschetz theorem for K¨ahler orbifolds. Proceedings of the American Mathematical Society  ,  (Aug  ).  [2] Batyrev, V. V., and Cox, D. A. On the Hodge structure of projective hypersur- faces in toric varieties. Duke Mathematical Journal 75, 2 (Aug 1994).  [  ] Bruzzo, U., and Montoya, W. On the Hodge conjecture for quasi-smooth in- tersections in toric varieties. S˜ao Paulo J. Math. Sci. Special Section: Geometry in Algebra and Algebra in Geometry (  ).  [3] Bruzzo, U., and Montoya, W. On the Hodge conjecture for quasi-smooth in- tersections in toric varieties. S˜ao Paulo J. Math. Sci. Special Section: Geometry in Algebra and Algebra in Geometry (2021).  A. R. Cohomology of complete intersections in toric varieties. Pub-', lookup_str='', metadata={'source': '/var/folders/ph/hhm7_zyx4l13k3v8z02dwp1w0000gn/T/tmpgq0ckaja/online_file.pdf'}, lookup_index=0)]

```

使用PDFMiner#

```code
from langchain.document_loaders import PDFMinerLoader

```

```code
loader = PDFMinerLoader("example_data/layout-parser-paper.pdf")

```

```code
data = loader.load()

```

使用PyPDFium2#

```code
from langchain.document_loaders import PyPDFium2Loader

```

```code
loader = PyPDFium2Loader("example_data/layout-parser-paper.pdf")

```

```code
data = loader.load()

```

使用PDFMiner生成HTML文本#
这对于将文本按语义分块成部分非常有帮助，因为输出的html内容可以通过
```code
BeautifulSoup
```
解析，以获取有关字体大小、页码、pdf页眉/页脚等更结构化和丰富的信息。

```code
from langchain.document_loaders import PDFMinerPDFasHTMLLoader

```

```code
loader = PDFMinerPDFasHTMLLoader("example_data/layout-parser-paper.pdf")

```

```code
data = loader.load()[0]   # entire pdf is loaded as a single Document

```

```code
from bs4 import BeautifulSoup
soup = BeautifulSoup(data.page_content,'html.parser')
content = soup.find_all('div')

```

```code
import re
cur_fs = None
cur_text = ''
snippets = []   # first collect all snippets that have the same font size
for c in content:
sp = c.find('span')
if not sp:
continue
st = sp.get('style')
if not st:
continue
fs = re.findall('font-size:(\d+)px',st)
if not fs:
continue
fs = int(fs[0])
if not cur_fs:
cur_fs = fs
if fs == cur_fs:
cur_text += c.text
else:
snippets.append((cur_text,cur_fs))
cur_fs = fs
cur_text = c.text
snippets.append((cur_text,cur_fs))
# Note: The above logic is very straightforward. One can also add more strategies such as removing duplicate snippets (as
# headers/footers in a PDF appear on multiple pages so if we find duplicatess safe to assume that it is redundant info)

```

```code
from langchain.docstore.document import Document
cur_idx = -1
semantic_snippets = []
# Assumption: headings have higher font size than their respective content
for s in snippets:
# if current snippet's font size > previous section's heading => it is a new heading
if not semantic_snippets or s[1] > semantic_snippets[cur_idx].metadata['heading_font']:
metadata={'heading':s[0], 'content_font': 0, 'heading_font': s[1]}
metadata.update(data.metadata)
semantic_snippets.append(Document(page_content='',metadata=metadata))
cur_idx += 1
continue

# if current snippet's font size <= previous section's content => content belongs to the same section (one can also create
# a tree like structure for sub sections if needed but that may require some more thinking and may be data specific)
if not semantic_snippets[cur_idx].metadata['content_font'] or s[1] <= semantic_snippets[cur_idx].metadata['content_font']:
semantic_snippets[cur_idx].page_content += s[0]
semantic_snippets[cur_idx].metadata['content_font'] = max(s[1], semantic_snippets[cur_idx].metadata['content_font'])
continue

# if current snippet's font size > previous section's content but less tha previous section's heading than also make a new
# section (e.g. title of a pdf will have the highest font size but we don't want it to subsume all sections)
metadata={'heading':s[0], 'content_font': 0, 'heading_font': s[1]}
metadata.update(data.metadata)
semantic_snippets.append(Document(page_content='',metadata=metadata))
cur_idx += 1

```

```code
semantic_snippets[4]

```

```code
Document(page_content='Recently, various DL models and datasets have been developed for layout analysis  tasks. The dhSegment [22] utilizes fully convolutional networks [20] for segmen-  tation tasks on historical documents. Object detection-based methods like Faster  R-CNN [28] and Mask R-CNN [12] are used for identifying document elements [38]  and detecting tables [30, 26]. Most recently, Graph Neural Networks [29] have also  been used in table detection [27]. However, these models are usually implemented  individually and there is no uniﬁed framework to load and use such models.  There has been a surge of interest in creating open-source tools for document  image processing: a search of document image analysis in Github leads to 5M  relevant code pieces 6; yet most of them rely on traditional rule-based methods  or provide limited functionalities. The closest prior research to our work is the  OCR-D project7, which also tries to build a complete toolkit for DIA. However,  similar to the platform developed by Neudecker et al. [21], it is designed for  analyzing historical documents, and provides no supports for recent DL models.  The DocumentLayoutAnalysis project8 focuses on processing born-digital PDF  documents via analyzing the stored PDF data. Repositories like DeepLayout9  and Detectron2-PubLayNet10 are individual deep learning models trained on  layout analysis datasets without support for the full DIA pipeline. The Document  Analysis and Exploitation (DAE) platform [15] and the DeepDIVA project [2]  aim to improve the reproducibility of DIA methods (or DL models), yet they  are not actively maintained. OCR engines like Tesseract [14], easyOCR11 and  paddleOCR12 usually do not come with comprehensive functionalities for other  DIA tasks like layout analysis.  Recent years have also seen numerous eﬀorts to create libraries for promoting  reproducibility and reusability in the ﬁeld of DL. Libraries like Dectectron2 [35],  6 The number shown is obtained by specifying the search type as ‘code’.  7 https://ocr-d.de/en/about  8 https://github.com/BobLd/DocumentLayoutAnalysis  9 https://github.com/leonlulu/DeepLayout  10 https://github.com/hpanwar08/detectron2  11 https://github.com/JaidedAI/EasyOCR  12 https://github.com/PaddlePaddle/PaddleOCR  4  Z. Shen et al.  Fig. 1: The overall architecture of LayoutParser. For an input document image,  the core LayoutParser library provides a set of oﬀ-the-shelf tools for layout  detection, OCR, visualization, and storage, backed by a carefully designed layout  data structure. LayoutParser also supports high level customization via eﬃcient  layout annotation and model training functions. These improve model accuracy  on the target samples. The community platform enables the easy sharing of DIA  models and whole digitization pipelines to promote reusability and reproducibility.  A collection of detailed documentation, tutorials and exemplar projects make  LayoutParser easy to learn and use.  AllenNLP [8] and transformers [34] have provided the community with complete  DL-based support for developing and deploying models for general computer  vision and natural language processing problems. LayoutParser, on the other  hand, specializes speciﬁcally in DIA tasks. LayoutParser is also equipped with a  community platform inspired by established model hubs such as Torch Hub [23]  and TensorFlow Hub [1]. It enables the sharing of pretrained models as well as  full document processing pipelines that are unique to DIA tasks.  There have been a variety of document data collections to facilitate the  development of DL models. Some examples include PRImA [3](magazine layouts),  PubLayNet [38](academic paper layouts), Table Bank [18](tables in academic  papers), Newspaper Navigator Dataset [16, 17](newspaper ﬁgure layouts) and  HJDataset [31](historical Japanese document layouts). A spectrum of models  trained on these datasets are currently available in the LayoutParser model zoo  to support diﬀerent use cases.  ', metadata={'heading': '2 Related Work  ', 'content_font': 9, 'heading_font': 11, 'source': 'example_data/layout-parser-paper.pdf'})

```

使用PyMuPDF#
这是PDF解析选项中最快的，包含有关PDF及其页面的详细元数据，以及每页返回一个文档。

```code
from langchain.document_loaders import PyMuPDFLoader

```

```code
loader = PyMuPDFLoader("example_data/layout-parser-paper.pdf")

```

```code
data = loader.load()

```

```code
data[0]

```

```code
Document(page_content='LayoutParser: A Uniﬁed Toolkit for Deep  Learning Based Document Image Analysis  Zejiang Shen1 (�), Ruochen Zhang2, Melissa Dell3, Benjamin Charles Germain  Lee4, Jacob Carlson3, and Weining Li5  1 Allen Institute for AI  shannons@allenai.org  2 Brown University  ruochen zhang@brown.edu  3 Harvard University  {melissadell,jacob carlson}@fas.harvard.edu  4 University of Washington  bcgl@cs.washington.edu  5 University of Waterloo  w422li@uwaterloo.ca  Abstract. Recent advances in document image analysis (DIA) have been  primarily driven by the application of neural networks. Ideally, research  outcomes could be easily deployed in production and extended for further  investigation. However, various factors like loosely organized codebases  and sophisticated model conﬁgurations complicate the easy reuse of im-  portant innovations by a wide audience. Though there have been on-going  eﬀorts to improve reusability and simplify deep learning (DL) model  development in disciplines like natural language processing and computer  vision, none of them are optimized for challenges in the domain of DIA.  This represents a major gap in the existing toolkit, as DIA is central to  academic research across a wide range of disciplines in the social sciences  and humanities. This paper introduces LayoutParser, an open-source  library for streamlining the usage of DL in DIA research and applica-  tions. The core LayoutParser library comes with a set of simple and  intuitive interfaces for applying and customizing DL models for layout de-  tection, character recognition, and many other document processing tasks.  To promote extensibility, LayoutParser also incorporates a community  platform for sharing both pre-trained models and full document digiti-  zation pipelines. We demonstrate that LayoutParser is helpful for both  lightweight and large-scale digitization pipelines in real-word use cases.  The library is publicly available at https://layout-parser.github.io.  Keywords: Document Image Analysis · Deep Learning · Layout Analysis  · Character Recognition · Open Source library · Toolkit.  1  Introduction  Deep Learning(DL)-based approaches are the state-of-the-art for a wide range of  document image analysis (DIA) tasks including document image classiﬁcation [11,  arXiv:2103.15348v2  [cs.CV]  21 Jun 2021  ', lookup_str='', metadata={'file_path': 'example_data/layout-parser-paper.pdf', 'page_number': 1, 'total_pages': 16, 'format': 'PDF 1.5', 'title': '', 'author': '', 'subject': '', 'keywords': '', 'creator': 'LaTeX with hyperref', 'producer': 'pdfTeX-1.40.21', 'creationDate': 'D:20210622012710Z', 'modDate': 'D:20210622012710Z', 'trapped': '', 'encryption': None}, lookup_index=0)

```

此外，在
```code
load
```
调用中，您可以将PyMuPDF文档 (opens in a new tab)中的任何选项作为关键字参数传递，并将其传递给
```code
get_text()
```
调用。
PyPDF目录#
从目录加载PDF文件

```code
from langchain.document_loaders import PyPDFDirectoryLoader

```

```code
loader = PyPDFDirectoryLoader("example_data")

```

```code
docs = loader.load()

```
ObsidianPowerpoint


PowerPoint – LangChain中文网

6大核心模块（Modules)索引（Indexes)文档加载器（Document Loaders）示例Powerpoint

PowerPoint
本文介绍如何将PowerPoint文档加载到我们可以在下游使用的文档格式中。
用法：

```code
from langchain.document_loaders import UnstructuredPowerPointLoader
loader = UnstructuredPowerPointLoader("example_data/fake-power-point.pptx")
data = loader.load()
```

返回结果是一个Document对象列表，其中包含每个幻灯片的内容，元数据和索引。
保留元素
在内部，Unstructured为不同的文本块创建不同的“元素”。默认情况下，我们将其组合在一起，但是您可以通过指定
```code
mode="elements"
```
来轻松保持该分离。PdfReadthedocs Documentation


ReadTheDocs文档 – LangChain中文网

6大核心模块（Modules)索引（Indexes)文档加载器（Document Loaders）示例Readthedocs Documentation

ReadTheDocs文档
本文介绍如何加载作为Read-The-Docs构建的一部分生成的html中的内容。
用法：

```code
from langchain.document_loaders import ReadTheDocsLoader
loader = ReadTheDocsLoader("rtdocs", features='html.parser')
docs = loader.load()
```

前提是html已经被抓取到文件夹中。这可以通过取消注释并运行以下命令来完成：

```code
#!wget -r -A -P rtdocs https://langchain.readthedocs.io/en/latest/
```

示例可以从这里 (opens in a new tab)查看。PowerpointReddit


Reddit – LangChain中文网

6大核心模块（Modules)索引（Indexes)文档加载器（Document Loaders）示例Reddit

Reddit

Reddit (reddit) 是一个美国社交新闻聚合、内容评级和讨论网站。

此加载程序使用 praw Python 包从 Subreddits 或 Reddit Application (opens in a new tab) 用户的帖子中获取文本。
创建一个 Reddit 应用程序并使用您的 Reddit API 凭据初始化加载程序。

```code
from langchain.document_loaders import RedditPostsLoader

```

```code
# !pip install praw

```

```code
# load using 'subreddit' mode
loader = RedditPostsLoader(
client_id="YOUR CLIENT ID",
client_secret="YOUR CLIENT SECRET",
user_agent="extractor by u/Master_Ocelot8179",
categories=['new', 'hot'],                              # List of categories to load posts from
mode = 'subreddit',
search_queries=['investing', 'wallstreetbets'],         # List of subreddits to load posts from
number_posts=20                                         # Default value is 10
)

# # or load using 'username' mode
# loader = RedditPostsLoader(
# client_id="YOUR CLIENT ID",
# client_secret="YOUR CLIENT SECRET",
# user_agent="extractor by u/Master_Ocelot8179",
# categories=['new', 'hot'],
# mode = 'username',
# search_queries=['ga3far', 'Master_Ocelot8179'], # List of usernames to load posts from
# number_posts=20
# )

# Note: Categories can be only of following value - "controversial" "hot" "new" "rising" "top"

```

```code
documents = loader.load()
documents[:5]

```

```code
[Document(page_content='Hello, I am not looking for investment advice. I will apply my own due diligence. However, I am interested if anyone knows as a UK resident how fees and exchange rate differences would impact performance?  I am planning to create a pie of index funds (perhaps UK, US, europe) or find a fund with a good track record of long term growth at low rates.   Does anyone have any ideas?', metadata={'post_subreddit': 'r/investing', 'post_category': 'new', 'post_title': 'Long term retirement funds fees/exchange rate query', 'post_score': 1, 'post_id': '130pa6m', 'post_url': 'https://www.reddit.com/r/investing/comments/130pa6m/long_term_retirement_funds_feesexchange_rate_query/', 'post_author': Redditor(name='Badmanshiz')}),
Document(page_content='I much prefer the Roth IRA and would rather rollover my 401k to that every year instead of keeping it in the limited 401k options. But if I rollover, will I be able to continue contributing to my 401k? Or will that close my account? I realize that there are tax implications of doing this but I still think it is the better option.', metadata={'post_subreddit': 'r/investing', 'post_category': 'new', 'post_title': 'Is it possible to rollover my 401k every year?', 'post_score': 3, 'post_id': '130ja0h', 'post_url': 'https://www.reddit.com/r/investing/comments/130ja0h/is_it_possible_to_rollover_my_401k_every_year/', 'post_author': Redditor(name='AnCap_Catholic')}),
Document(page_content='Have a general question?  Want to offer some commentary on markets?  Maybe you would just like to throw out a neat fact that doesn\'t warrant a self post?  Feel free to post here!   If your question is "I have $10,000, what do I do?" or other "advice for my personal situation" questions, you should include relevant information, such as the following:  * How old are you? What country do you live in?  \n* Are you employed/making income? How much?  \n* What are your objectives with this money? (Buy a house? Retirement savings?)  \n* What is your time horizon? Do you need this money next month? Next 20yrs?  \n* What is your risk tolerance? (Do you mind risking it at blackjack or do you need to know its 100% safe?)  \n* What are you current holdings? (Do you already have exposure to specific funds and sectors? Any other assets?)  \n* Any big debts (include interest rate) or expenses?  \n* And any other relevant financial information will be useful to give you a proper answer.    Please consider consulting our FAQ first - https://www.reddit.com/r/investing/wiki/faq\nAnd our [side bar](https://www.reddit.com/r/investing/about/sidebar) also has useful resources.    If you are new to investing - please refer to Wiki - [Getting Started](https://www.reddit.com/r/investing/wiki/index/gettingstarted/)  The reading list in the wiki has a list of books ranging from light reading to advanced topics depending on your knowledge level. Link here - [Reading List](https://www.reddit.com/r/investing/wiki/readinglist)  Check the resources in the sidebar.  Be aware that these answers are just opinions of Redditors and should be used as a starting point for your research. You should strongly consider seeing a registered investment adviser if you need professional support before making any financial decisions!', metadata={'post_subreddit': 'r/investing', 'post_category': 'new', 'post_title': 'Daily General Discussion and Advice Thread - April 27, 2023', 'post_score': 5, 'post_id': '130eszz', 'post_url': 'https://www.reddit.com/r/investing/comments/130eszz/daily_general_discussion_and_advice_thread_april/', 'post_author': Redditor(name='AutoModerator')}),
Document(page_content="Based on recent news about salt battery advancements and the overall issues of lithium, I was wondering what would be feasible ways to invest into non-lithium based battery technologies? CATL is of course a choice, but the selection of brokers I currently have in my disposal don't provide HK stocks at all.", metadata={'post_subreddit': 'r/investing', 'post_category': 'new', 'post_title': 'Investing in non-lithium battery technologies?', 'post_score': 2, 'post_id': '130d6qp', 'post_url': 'https://www.reddit.com/r/investing/comments/130d6qp/investing_in_nonlithium_battery_technologies/', 'post_author': Redditor(name='-manabreak')}),
Document(page_content='Hello everyone,  I would really like to invest in an ETF that follows spy or another big index, as I think this form of investment suits me best.   The problem is, that I live in Denmark where ETFs and funds are taxed annually on unrealised gains at quite a steep rate. This means that an ETF growing say 10% per year will only grow about 6%, which really ruins the long term effects of compounding interest.  However stocks are only taxed on realised gains which is why they look more interesting to hold long term.  I do not like the lack of diversification this brings, as I am looking to spend tonnes of time picking the right long term stocks.  It would be ideal to find a few stocks that over the long term somewhat follows the indexes. Does anyone have suggestions?  I have looked at Nasdaq Inc. which quite closely follows Nasdaq 100.   I really appreciate any help.', metadata={'post_subreddit': 'r/investing', 'post_category': 'new', 'post_title': 'Stocks that track an index', 'post_score': 7, 'post_id': '130auvj', 'post_url': 'https://www.reddit.com/r/investing/comments/130auvj/stocks_that_track_an_index/', 'post_author': Redditor(name='LeAlbertP')})]

```
Readthedocs DocumentationRoam


Roam – LangChain中文网

6大核心模块（Modules)索引（Indexes)文档加载器（Document Loaders）示例Roam

Roam
本文介绍如何从Roam数据库中加载文档。本文以此示例repo (opens in a new tab)为灵感。
使用方法：

从Roam Research中导出数据集。可以通过单击右上角的三个点，然后单击
```code
导出
```
进行操作。

在导出时，确保选择
```code
Markdown和CSV
```
格式选项。

这将在下载文件夹中生成一个
```code
.zip
```
文件。将
```code
.zip
```
文件移动到此存储库中。

运行以下命令解压缩zip文件（根据需要将
```code
Export...
```
替换为您自己的文件名)。

```code
unzip Roam-Export-1675782732639.zip -d Roam_DB
```

示例代码：

```code
from langchain.document_loaders import RoamLoader
loader = RoamLoader("Roam_DB")
docs = loader.load()
```

返回结果是一个Document对象列表，其中每个对象包含文档内容和元数据。RedditS3 Directory


s3 Directory – LangChain中文网

6大核心模块（Modules)索引（Indexes)文档加载器（Document Loaders）示例S3 Directory

s3 Directory
本文介绍如何从s3目录对象加载文档对象。
示例代码：

```code
from langchain.document_loaders import S3DirectoryLoader
```

需要安装
```code
boto3
```
包：

```code
!pip install boto3
```

初始化载入器：

```code
loader = S3DirectoryLoader("testing-hwc")
```

加载并返回文档对象列表：

```code
loader.load()
```

返回结果是一个Document对象列表，其中每个对象包含文档内容和元数据。
指定前缀：
您还可以指定前缀以更精细地控制要加载的文件。
示例代码：

```code
loader = S3DirectoryLoader("testing-hwc", prefix="fake")
loader.load()
```

返回结果是以前缀开头的Document对象列表，其中每个对象包含文档内容和元数据。RoamS3 File


s3 File – LangChain中文网

6大核心模块（Modules)索引（Indexes)文档加载器（Document Loaders）示例S3 File

s3 File
本文介绍如何从s3文件对象加载文档对象。
示例代码：

```code
from langchain.document_loaders import S3FileLoader
```

需要安装
```code
boto3
```
包：

```code
!pip install boto3
```

初始化载入器：

```code
loader = S3FileLoader("testing-hwc", "fake.docx")
```

加载并返回文档对象：

```code
loader.load()
```

返回结果是一个Document对象列表，其中每个对象包含文档内容和元数据。S3 DirectorySitemap


Sitemap Loader – LangChain中文网

6大核心模块（Modules)索引（Indexes)文档加载器（Document Loaders）示例Sitemap

Sitemap Loader
继承自WebBaseLoader，该loader将从给定的URL加载站点地图，并同时进行抓取和加载所有页面，将每个页面返回为文档。
使用WebBaseLoader并发地进行抓取。
同时进行请求有合理的限制，默认为每秒最多 2 次请求。
如果您不关心成为良好的用户，或者您控制您正在抓取的服务器而不关心负载，则可以更改“requests_per_second”参数以增加最大并发请求。请
注意，虽然这将加速抓取过程，但可能会导致服务器阻止您。请小心用！

```code
!pip install nest_asyncio

```

```code
Requirement already satisfied: nest_asyncio in /Users/tasp/Code/projects/langchain/.venv/lib/python3.10/site-packages (1.5.6)

[notice] A new release of pip available: 22.3.1 -> 23.0.1
[notice] To update, run: pip install --upgrade pip

```

```code
# fixes a bug with asyncio and jupyter
import nest_asyncio
nest_asyncio.apply()

```

```code
from langchain.document_loaders.sitemap import SitemapLoader

```

```code
sitemap_loader = SitemapLoader(web_path="https://langchain.readthedocs.io/sitemap.xml")

docs = sitemap_loader.load()

```

```code
docs[0]

```

```code
Document(page_content='\n\n\n\n\n\nWelcome to LangChain — 🦜🔗 LangChain 0.0.123\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nSkip to main content\n\n\n\n\n\n\n\n\n\n\nCtrl+K\n\n\n\n\n\n\n\n\n\n\n\n\n🦜🔗 LangChain 0.0.123\n\n\n\nGetting Started\n\nQuickstart Guide\n\nModules\n\nPrompt Templates\nGetting Started\nKey Concepts\nHow-To Guides\nCreate a custom prompt template\nCreate a custom example selector\nProvide few shot examples to a prompt\nPrompt Serialization\nExample Selectors\nOutput Parsers\n\n\nReference\nPromptTemplates\nExample Selector\n\n\n\n\nLLMs\nGetting Started\nKey Concepts\nHow-To Guides\nGeneric Functionality\nCustom LLM\nFake LLM\nLLM Caching\nLLM Serialization\nToken Usage Tracking\n\n\nIntegrations\nAI21\nAleph Alpha\nAnthropic\nAzure OpenAI LLM Example\nBanana\nCerebriumAI LLM Example\nCohere\nDeepInfra LLM Example\nForefrontAI LLM Example\nGooseAI LLM Example\nHugging Face Hub\nManifest\nModal\nOpenAI\nPetals LLM Example\nPromptLayer OpenAI\nSageMakerEndpoint\nSelf-Hosted Models via Runhouse\nStochasticAI\nWriter\n\n\nAsync API for LLM\nStreaming with LLMs\n\n\nReference\n\n\nDocument Loaders\nKey Concepts\nHow To Guides\nCoNLL-U\nAirbyte JSON\nAZLyrics\nBlackboard\nCollege Confidential\nCopy Paste\nCSV Loader\nDirectory Loader\nEmail\nEverNote\nFacebook Chat\nFigma\nGCS Directory\nGCS File Storage\nGitBook\nGoogle Drive\nGutenberg\nHacker News\nHTML\niFixit\nImages\nIMSDb\nMarkdown\nNotebook\nNotion\nObsidian\nPDF\nPowerPoint\nReadTheDocs Documentation\nRoam\ns3 Directory\ns3 File\nSubtitle Files\nTelegram\nUnstructured File Loader\nURL\nWeb Base\nWord Documents\nYouTube\n\n\n\n\nUtils\nKey Concepts\nGeneric Utilities\nBash\nBing Search\nGoogle Search\nGoogle Serper API\nIFTTT WebHooks\nPython REPL\nRequests\nSearxNG Search API\nSerpAPI\nWolfram Alpha\nZapier Natural Language Actions API\n\n\nReference\nPython REPL\nSerpAPI\nSearxNG Search\nDocstore\nText Splitter\nEmbeddings\nVectorStores\n\n\n\n\nIndexes\nGetting Started\nKey Concepts\nHow To Guides\nEmbeddings\nHypothetical Document Embeddings\nText Splitter\nVectorStores\nAtlasDB\nChroma\nDeep Lake\nElasticSearch\nFAISS\nMilvus\nOpenSearch\nPGVector\nPinecone\nQdrant\nRedis\nWeaviate\nChatGPT Plugin Retriever\nVectorStore Retriever\nAnalyze Document\nChat Index\nGraph QA\nQuestion Answering with Sources\nQuestion Answering\nSummarization\nRetrieval Question/Answering\nRetrieval Question Answering with Sources\nVector DB Text Generation\n\n\n\n\nChains\nGetting Started\nHow-To Guides\nGeneric Chains\nLoading from LangChainHub\nLLM Chain\nSequential Chains\nSerialization\nTransformation Chain\n\n\nUtility Chains\nAPI Chains\nSelf-Critique Chain with Constitutional AI\nBashChain\nLLMCheckerChain\nLLM Math\nLLMRequestsChain\nLLMSummarizationCheckerChain\nModeration\nPAL\nSQLite example\n\n\nAsync API for Chain\n\n\nKey Concepts\nReference\n\n\nAgents\nGetting Started\nKey Concepts\nHow-To Guides\nAgents and Vectorstores\nAsync API for Agent\nConversation Agent (for Chat Models)\nChatGPT Plugins\nCustom Agent\nDefining Custom Tools\nHuman as a tool\nIntermediate Steps\nLoading from LangChainHub\nMax Iterations\nMulti Input Tools\nSearch Tools\nSerialization\nAdding SharedMemory to an Agent and its Tools\nCSV Agent\nJSON Agent\nOpenAPI Agent\nPandas Dataframe Agent\nPython Agent\nSQL Database Agent\nVectorstore Agent\nMRKL\nMRKL Chat\nReAct\nSelf Ask With Search\n\n\nReference\n\n\nMemory\nGetting Started\nKey Concepts\nHow-To Guides\nConversationBufferMemory\nConversationBufferWindowMemory\nEntity Memory\nConversation Knowledge Graph Memory\nConversationSummaryMemory\nConversationSummaryBufferMemory\nConversationTokenBufferMemory\nAdding Memory To an LLMChain\nAdding Memory to a Multi-Input Chain\nAdding Memory to an Agent\nChatGPT Clone\nConversation Agent\nConversational Memory Customization\nCustom Memory\nMultiple Memory\n\n\n\n\nChat\nGetting Started\nKey Concepts\nHow-To Guides\nAgent\nChat Vector DB\nFew Shot Examples\nMemory\nPromptLayer ChatOpenAI\nStreaming\nRetrieval Question/Answering\nRetrieval Question Answering with Sources\n\n\n\n\n\nUse Cases\n\nAgents\nChatbots\nGenerate Examples\nData Augmented Generation\nQuestion Answering\nSummarization\nQuerying Tabular Data\nExtraction\nEvaluation\nAgent Benchmarking: Search + Calculator\nAgent VectorDB Question Answering Benchmarking\nBenchmarking Template\nData Augmented Question Answering\nUsing Hugging Face Datasets\nLLM Math\nQuestion Answering Benchmarking: Paul Graham Essay\nQuestion Answering Benchmarking: State of the Union Address\nQA Generation\nQuestion Answering\nSQL Question Answering Benchmarking: Chinook\n\n\nModel Comparison\n\nReference\n\nInstallation\nIntegrations\nAPI References\nPrompts\nPromptTemplates\nExample Selector\n\n\nUtilities\nPython REPL\nSerpAPI\nSearxNG Search\nDocstore\nText Splitter\nEmbeddings\nVectorStores\n\n\nChains\nAgents\n\n\n\nEcosystem\n\nLangChain Ecosystem\nAI21 Labs\nAtlasDB\nBanana\nCerebriumAI\nChroma\nCohere\nDeepInfra\nDeep Lake\nForefrontAI\nGoogle Search Wrapper\nGoogle Serper Wrapper\nGooseAI\nGraphsignal\nHazy Research\nHelicone\nHugging Face\nMilvus\nModal\nNLPCloud\nOpenAI\nOpenSearch\nPetals\nPGVector\nPinecone\nPromptLayer\nQdrant\nRunhouse\nSearxNG Search API\nSerpAPI\nStochasticAI\nUnstructured\nWeights & Biases\nWeaviate\nWolfram Alpha Wrapper\nWriter\n\n\n\nAdditional Resources\n\nLangChainHub\nGlossary\nLangChain Gallery\nDeployments\nTracing\nDiscord\nProduction Support\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n.rst\n\n\n\n\n\n\n\n.pdf\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nWelcome to LangChain\n\n\n\n\n Contents \n\n\n\nGetting Started\nModules\nUse Cases\nReference Docs\nLangChain Ecosystem\nAdditional Resources\n\n\n\n\n\n\n\n\nWelcome to LangChain#\nLarge language models (LLMs) are emerging as a transformative technology, enabling\ndevelopers to build applications that they previously could not.\nBut using these LLMs in isolation is often not enough to\ncreate a truly powerful app - the real power comes when you are able to\ncombine them with other sources of computation or knowledge.\nThis library is aimed at assisting in the development of those types of applications. Common examples of these types of applications include:\n❓ Question Answering over specific documents\n\nDocumentation\nEnd-to-end Example: Question Answering over Notion Database\n\n💬 Chatbots\n\nDocumentation\nEnd-to-end Example: Chat-LangChain\n\n🤖 Agents\n\nDocumentation\nEnd-to-end Example: GPT+WolframAlpha\n\n\nGetting Started#\nCheckout the below guide for a walkthrough of how to get started using LangChain to create an Language Model application.\n\nGetting Started Documentation\n\n\n\n\n\nModules#\nThere are several main modules that LangChain provides support for.\nFor each module we provide some examples to get started, how-to guides, reference docs, and conceptual guides.\nThese modules are, in increasing order of complexity:\n\nPrompts: This includes prompt management, prompt optimization, and prompt serialization.\nLLMs: This includes a generic interface for all LLMs, and common utilities for working with LLMs.\nDocument Loaders: This includes a standard interface for loading documents, as well as specific integrations to all types of text data sources.\nUtils: Language models are often more powerful when interacting with other sources of knowledge or computation. This can include Python REPLs, embeddings, search engines, and more. LangChain provides a large collection of common utils to use in your application.\nChains: Chains go beyond just a single LLM call, and are sequences of calls (whether to an LLM or a different utility). LangChain provides a standard interface for chains, lots of integrations with other tools, and end-to-end chains for common applications.\nIndexes: Language models are often more powerful when combined with your own text data - this module covers best practices for doing exactly that.\nAgents: Agents involve an LLM making decisions about which Actions to take, taking that Action, seeing an Observation, and repeating that until done. LangChain provides a standard interface for agents, a selection of agents to choose from, and examples of end to end agents.\nMemory: Memory is the concept of persisting state between calls of a chain/agent. LangChain provides a standard interface for memory, a collection of memory implementations, and examples of chains/agents that use memory.\nChat: Chat models are a variation on Language Models that expose a different API - rather than working with raw text, they work with messages. LangChain provides a standard interface for working with them and doing all the same things as above.\n\n\n\n\n\nUse Cases#\nThe above modules can be used in a variety of ways. LangChain also provides guidance and assistance in this. Below are some of the common use cases LangChain supports.\n\nAgents: Agents are systems that use a language model to interact with other tools. These can be used to do more grounded question/answering, interact with APIs, or even take actions.\nChatbots: Since language models are good at producing text, that makes them ideal for creating chatbots.\nData Augmented Generation: Data Augmented Generation involves specific types of chains that first interact with an external datasource to fetch data to use in the generation step. Examples of this include summarization of long pieces of text and question/answering over specific data sources.\nQuestion Answering: Answering questions over specific documents, only utilizing the information in those documents to construct an answer. A type of Data Augmented Generation.\nSummarization: Summarizing longer documents into shorter, more condensed chunks of information. A type of Data Augmented Generation.\nQuerying Tabular Data: If you want to understand how to use LLMs to query data that is stored in a tabular format (csvs, SQL, dataframes, etc) you should read this page.\nEvaluation: Generative models are notoriously hard to evaluate with traditional metrics. One new way of evaluating them is using language models themselves to do the evaluation. LangChain provides some prompts/chains for assisting in this.\nGenerate similar examples: Generating similar examples to a given input. This is a common use case for many applications, and LangChain provides some prompts/chains for assisting in this.\nCompare models: Experimenting with different prompts, models, and chains is a big part of developing the best possible application. The ModelLaboratory makes it easy to do so.\n\n\n\n\n\nReference Docs#\nAll of LangChain’s reference documentation, in one place. Full documentation on all methods, classes, installation methods, and integration setups for LangChain.\n\nReference Documentation\n\n\n\n\n\nLangChain Ecosystem#\nGuides for how other companies/products can be used with LangChain\n\nLangChain Ecosystem\n\n\n\n\n\nAdditional Resources#\nAdditional collection of resources we think may be useful as you develop your application!\n\nLangChainHub: The LangChainHub is a place to share and explore other prompts, chains, and agents.\nGlossary: A glossary of all related terms, papers, methods, etc. Whether implemented in LangChain or not!\nGallery: A collection of our favorite projects that use LangChain. Useful for finding inspiration or seeing how things were done in other applications.\nDeployments: A collection of instructions, code snippets, and template repositories for deploying LangChain apps.\nDiscord: Join us on our Discord to discuss all things LangChain!\nTracing: A guide on using tracing in LangChain to visualize the execution of chains and agents.\nProduction Support: As you move your LangChains into production, we’d love to offer more comprehensive support. Please fill out this form and we’ll set up a dedicated support Slack channel.\n\n\n\n\n\n\n\n\n\n\n\nnext\nQuickstart Guide\n\n\n\n\n\n\n\n\n\n Contents\n  \n\n\nGetting Started\nModules\nUse Cases\nReference Docs\nLangChain Ecosystem\nAdditional Resources\n\n\n\n\n\n\n\n\n\nBy Harrison Chase\n\n\n\n\n    \n      © Copyright 2023, Harrison Chase.\n      \n\n\n\n\n  Last updated on Mar 24, 2023.\n  \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n', lookup_str='', metadata={'source': 'https://python.langchain.com/en/stable/', 'loc': 'https://python.langchain.com/en/stable/', 'lastmod': '2023-03-24T19:30:54.647430+00:00', 'changefreq': 'weekly', 'priority': '1'}, lookup_index=0)

```

过滤站点地图 URL #
站点地图可以是包含数千个 URL 的大型文件。通常你不需要每一个。
您可以通过将字符串列表或正则表达式模式传递给
```code
url_filter
```
参数来过滤 URL。只有与其中一种模式匹配的 URL 才会被加载。

```code
loader = SitemapLoader(
"https://langchain.readthedocs.io/sitemap.xml",
filter_urls=["https://python.langchain.com/en/latest/"]
)
documents = loader.load()

```

```code
documents[0]

```

```code
Document(page_content='\n\n\n\n\n\nWelcome to LangChain — 🦜🔗 LangChain 0.0.123\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nSkip to main content\n\n\n\n\n\n\n\n\n\n\nCtrl+K\n\n\n\n\n\n\n\n\n\n\n\n\n🦜🔗 LangChain 0.0.123\n\n\n\nGetting Started\n\nQuickstart Guide\n\nModules\n\nModels\nLLMs\nGetting Started\nGeneric Functionality\nHow to use the async API for LLMs\nHow to write a custom LLM wrapper\nHow (and why) to use the fake LLM\nHow to cache LLM calls\nHow to serialize LLM classes\nHow to stream LLM responses\nHow to track token usage\n\n\nIntegrations\nAI21\nAleph Alpha\nAnthropic\nAzure OpenAI LLM Example\nBanana\nCerebriumAI LLM Example\nCohere\nDeepInfra LLM Example\nForefrontAI LLM Example\nGooseAI LLM Example\nHugging Face Hub\nManifest\nModal\nOpenAI\nPetals LLM Example\nPromptLayer OpenAI\nSageMakerEndpoint\nSelf-Hosted Models via Runhouse\nStochasticAI\nWriter\n\n\nReference\n\n\nChat Models\nGetting Started\nHow-To Guides\nHow to use few shot examples\nHow to stream responses\n\n\nIntegrations\nAzure\nOpenAI\nPromptLayer ChatOpenAI\n\n\n\n\nText Embedding Models\nAzureOpenAI\nCohere\nFake Embeddings\nHugging Face Hub\nInstructEmbeddings\nOpenAI\nSageMaker Endpoint Embeddings\nSelf Hosted Embeddings\nTensorflowHub\n\n\n\n\nPrompts\nPrompt Templates\nGetting Started\nHow-To Guides\nHow to create a custom prompt template\nHow to create a prompt template that uses few shot examples\nHow to work with partial Prompt Templates\nHow to serialize prompts\n\n\nReference\nPromptTemplates\nExample Selector\n\n\n\n\nChat Prompt Template\nExample Selectors\nHow to create a custom example selector\nLengthBased ExampleSelector\nMaximal Marginal Relevance ExampleSelector\nNGram Overlap ExampleSelector\nSimilarity ExampleSelector\n\n\nOutput Parsers\nOutput Parsers\nCommaSeparatedListOutputParser\nOutputFixingParser\nPydanticOutputParser\nRetryOutputParser\nStructured Output Parser\n\n\n\n\nIndexes\nGetting Started\nDocument Loaders\nCoNLL-U\nAirbyte JSON\nAZLyrics\nBlackboard\nCollege Confidential\nCopy Paste\nCSV Loader\nDirectory Loader\nEmail\nEverNote\nFacebook Chat\nFigma\nGCS Directory\nGCS File Storage\nGitBook\nGoogle Drive\nGutenberg\nHacker News\nHTML\niFixit\nImages\nIMSDb\nMarkdown\nNotebook\nNotion\nObsidian\nPDF\nPowerPoint\nReadTheDocs Documentation\nRoam\ns3 Directory\ns3 File\nSubtitle Files\nTelegram\nUnstructured File Loader\nURL\nWeb Base\nWord Documents\nYouTube\n\n\nText Splitters\nGetting Started\nCharacter Text Splitter\nHuggingFace Length Function\nLatex Text Splitter\nMarkdown Text Splitter\nNLTK Text Splitter\nPython Code Text Splitter\nRecursiveCharacterTextSplitter\nSpacy Text Splitter\ntiktoken (OpenAI) Length Function\nTiktokenText Splitter\n\n\nVectorstores\nGetting Started\nAtlasDB\nChroma\nDeep Lake\nElasticSearch\nFAISS\nMilvus\nOpenSearch\nPGVector\nPinecone\nQdrant\nRedis\nWeaviate\n\n\nRetrievers\nChatGPT Plugin Retriever\nVectorStore Retriever\n\n\n\n\nMemory\nGetting Started\nHow-To Guides\nConversationBufferMemory\nConversationBufferWindowMemory\nEntity Memory\nConversation Knowledge Graph Memory\nConversationSummaryMemory\nConversationSummaryBufferMemory\nConversationTokenBufferMemory\nHow to add Memory to an LLMChain\nHow to add memory to a Multi-Input Chain\nHow to add Memory to an Agent\nHow to customize conversational memory\nHow to create a custom Memory class\nHow to use multiple memroy classes in the same chain\n\n\n\n\nChains\nGetting Started\nHow-To Guides\nAsync API for Chain\nLoading from LangChainHub\nLLM Chain\nSequential Chains\nSerialization\nTransformation Chain\nAnalyze Document\nChat Index\nGraph QA\nHypothetical Document Embeddings\nQuestion Answering with Sources\nQuestion Answering\nSummarization\nRetrieval Question/Answering\nRetrieval Question Answering with Sources\nVector DB Text Generation\nAPI Chains\nSelf-Critique Chain with Constitutional AI\nBashChain\nLLMCheckerChain\nLLM Math\nLLMRequestsChain\nLLMSummarizationCheckerChain\nModeration\nPAL\nSQLite example\n\n\nReference\n\n\nAgents\nGetting Started\nTools\nGetting Started\nDefining Custom Tools\nMulti Input Tools\nBash\nBing Search\nChatGPT Plugins\nGoogle Search\nGoogle Serper API\nHuman as a tool\nIFTTT WebHooks\nPython REPL\nRequests\nSearch Tools\nSearxNG Search API\nSerpAPI\nWolfram Alpha\nZapier Natural Language Actions API\n\n\nAgents\nAgent Types\nCustom Agent\nConversation Agent (for Chat Models)\nConversation Agent\nMRKL\nMRKL Chat\nReAct\nSelf Ask With Search\n\n\nToolkits\nCSV Agent\nJSON Agent\nOpenAPI Agent\nPandas Dataframe Agent\nPython Agent\nSQL Database Agent\nVectorstore Agent\n\n\nAgent Executors\nHow to combine agents and vectorstores\nHow to use the async API for Agents\nHow to create ChatGPT Clone\nHow to access intermediate steps\nHow to cap the max number of iterations\nHow to add SharedMemory to an Agent and its Tools\n\n\n\n\n\nUse Cases\n\nPersonal Assistants\nQuestion Answering over Docs\nChatbots\nQuerying Tabular Data\nInteracting with APIs\nSummarization\nExtraction\nEvaluation\nAgent Benchmarking: Search + Calculator\nAgent VectorDB Question Answering Benchmarking\nBenchmarking Template\nData Augmented Question Answering\nUsing Hugging Face Datasets\nLLM Math\nQuestion Answering Benchmarking: Paul Graham Essay\nQuestion Answering Benchmarking: State of the Union Address\nQA Generation\nQuestion Answering\nSQL Question Answering Benchmarking: Chinook\n\n\n\nReference\n\nInstallation\nIntegrations\nAPI References\nPrompts\nPromptTemplates\nExample Selector\n\n\nUtilities\nPython REPL\nSerpAPI\nSearxNG Search\nDocstore\nText Splitter\nEmbeddings\nVectorStores\n\n\nChains\nAgents\n\n\n\nEcosystem\n\nLangChain Ecosystem\nAI21 Labs\nAtlasDB\nBanana\nCerebriumAI\nChroma\nCohere\nDeepInfra\nDeep Lake\nForefrontAI\nGoogle Search Wrapper\nGoogle Serper Wrapper\nGooseAI\nGraphsignal\nHazy Research\nHelicone\nHugging Face\nMilvus\nModal\nNLPCloud\nOpenAI\nOpenSearch\nPetals\nPGVector\nPinecone\nPromptLayer\nQdrant\nRunhouse\nSearxNG Search API\nSerpAPI\nStochasticAI\nUnstructured\nWeights & Biases\nWeaviate\nWolfram Alpha Wrapper\nWriter\n\n\n\nAdditional Resources\n\nLangChainHub\nGlossary\nLangChain Gallery\nDeployments\nTracing\nDiscord\nProduction Support\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n.rst\n\n\n\n\n\n\n\n.pdf\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nWelcome to LangChain\n\n\n\n\n Contents \n\n\n\nGetting Started\nModules\nUse Cases\nReference Docs\nLangChain Ecosystem\nAdditional Resources\n\n\n\n\n\n\n\n\nWelcome to LangChain#\nLangChain is a framework for developing applications powered by language models. We believe that the most powerful and differentiated applications will not only call out to a language model via an API, but will also:\n\nBe data-aware: connect a language model to other sources of data\nBe agentic: allow a language model to interact with its environment\n\nThe LangChain framework is designed with the above principles in mind.\nThis is the Python specific portion of the documentation. For a purely conceptual guide to LangChain, see here. For the JavaScript documentation, see here.\n\nGetting Started#\nCheckout the below guide for a walkthrough of how to get started using LangChain to create an Language Model application.\n\nGetting Started Documentation\n\n\n\n\n\nModules#\nThere are several main modules that LangChain provides support for.\nFor each module we provide some examples to get started, how-to guides, reference docs, and conceptual guides.\nThese modules are, in increasing order of complexity:\n\nModels: The various model types and model integrations LangChain supports.\nPrompts: This includes prompt management, prompt optimization, and prompt serialization.\nMemory: Memory is the concept of persisting state between calls of a chain/agent. LangChain provides a standard interface for memory, a collection of memory implementations, and examples of chains/agents that use memory.\nIndexes: Language models are often more powerful when combined with your own text data - this module covers best practices for doing exactly that.\nChains: Chains go beyond just a single LLM call, and are sequences of calls (whether to an LLM or a different utility). LangChain provides a standard interface for chains, lots of integrations with other tools, and end-to-end chains for common applications.\nAgents: Agents involve an LLM making decisions about which Actions to take, taking that Action, seeing an Observation, and repeating that until done. LangChain provides a standard interface for agents, a selection of agents to choose from, and examples of end to end agents.\n\n\n\n\n\nUse Cases#\nThe above modules can be used in a variety of ways. LangChain also provides guidance and assistance in this. Below are some of the common use cases LangChain supports.\n\nPersonal Assistants: The main LangChain use case. Personal assistants need to take actions, remember interactions, and have knowledge about your data.\nQuestion Answering: The second big LangChain use case. Answering questions over specific documents, only utilizing the information in those documents to construct an answer.\nChatbots: Since language models are good at producing text, that makes them ideal for creating chatbots.\nQuerying Tabular Data: If you want to understand how to use LLMs to query data that is stored in a tabular format (csvs, SQL, dataframes, etc) you should read this page.\nInteracting with APIs: Enabling LLMs to interact with APIs is extremely powerful in order to give them more up-to-date information and allow them to take actions.\nExtraction: Extract structured information from text.\nSummarization: Summarizing longer documents into shorter, more condensed chunks of information. A type of Data Augmented Generation.\nEvaluation: Generative models are notoriously hard to evaluate with traditional metrics. One new way of evaluating them is using language models themselves to do the evaluation. LangChain provides some prompts/chains for assisting in this.\n\n\n\n\n\nReference Docs#\nAll of LangChain’s reference documentation, in one place. Full documentation on all methods, classes, installation methods, and integration setups for LangChain.\n\nReference Documentation\n\n\n\n\n\nLangChain Ecosystem#\nGuides for how other companies/products can be used with LangChain\n\nLangChain Ecosystem\n\n\n\n\n\nAdditional Resources#\nAdditional collection of resources we think may be useful as you develop your application!\n\nLangChainHub: The LangChainHub is a place to share and explore other prompts, chains, and agents.\nGlossary: A glossary of all related terms, papers, methods, etc. Whether implemented in LangChain or not!\nGallery: A collection of our favorite projects that use LangChain. Useful for finding inspiration or seeing how things were done in other applications.\nDeployments: A collection of instructions, code snippets, and template repositories for deploying LangChain apps.\nTracing: A guide on using tracing in LangChain to visualize the execution of chains and agents.\nModel Laboratory: Experimenting with different prompts, models, and chains is a big part of developing the best possible application. The ModelLaboratory makes it easy to do so.\nDiscord: Join us on our Discord to discuss all things LangChain!\nProduction Support: As you move your LangChains into production, we’d love to offer more comprehensive support. Please fill out this form and we’ll set up a dedicated support Slack channel.\n\n\n\n\n\n\n\n\n\n\n\nnext\nQuickstart Guide\n\n\n\n\n\n\n\n\n\n Contents\n  \n\n\nGetting Started\nModules\nUse Cases\nReference Docs\nLangChain Ecosystem\nAdditional Resources\n\n\n\n\n\n\n\n\n\nBy Harrison Chase\n\n\n\n\n    \n      © Copyright 2023, Harrison Chase.\n      \n\n\n\n\n  Last updated on Mar 27, 2023.\n  \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n', lookup_str='', metadata={'source': 'https://python.langchain.com/en/latest/', 'loc': 'https://python.langchain.com/en/latest/', 'lastmod': '2023-03-27T22:50:49.790324+00:00', 'changefreq': 'daily', 'priority': '0.9'}, lookup_index=0)

```

本地文件Local Sitemap
站点地图加载器也可用于加载本地文件

```code
sitemap_loader = SitemapLoader(web_path="example_data/sitemap.xml", is_local=True)

docs = sitemap_loader.load()

```

```code
Fetching pages: 100%|####################################################################################################################################| 3/3 [00:00<00:00,  3.91it/s]

```
S3 FileSlack Directory


Slack (本地导出Zip文件) – LangChain中文网

6大核心模块（Modules)索引（Indexes)文档加载器（Document Loaders）示例Slack Directory

Slack (本地导出Zip文件)
本教程介绍了如何从Slack导出的Zip文件中加载文档。
为了获得这个Slack导出文件，请按照以下说明操作：
🧑 摄入自己的数据集的说明
导出您的Slack数据。您可以通过转到Workspace Management页面并单击导入/导出选项（
```code
{your_slack_domain}.slack.com/services/export
```
)来完成此操作。然后，选择正确的日期范围，然后单击“Start export”。当导出准备就绪时，Slack会向您发送电子邮件和DM。
下载将在您的下载文件夹中生成.zip文件（或者根据您的操作系统配置，可以在任何地方找到下载文件)。
复制.zip文件的路径，并将其分配为下面的LOCAL_ZIPFILE。

```code
from langchain.document_loaders import SlackDirectoryLoader
```

```code
# 可选择设置你的Slack URL，这将在文档中为您提供正确的URL。
SLACK_WORKSPACE_URL = "https://xxx.slack.com"
LOCAL_ZIPFILE = "" # 将本地路径粘贴到Slack zip文件中。

loader = SlackDirectoryLoader(LOCAL_ZIPFILE, SLACK_WORKSPACE_URL)
```

```code
docs = loader.load()
docs
```
SitemapSrt


字幕文件 – LangChain中文网

6大核心模块（Modules)索引（Indexes)文档加载器（Document Loaders）示例Srt

字幕文件
如何从字幕（.srt)文件中加载数据。

```code
from langchain.document_loaders import SRTLoader

```

```code
loader = SRTLoader("example_data/Star_Wars_The_Clone_Wars_S06E07_Crisis_at_the_Heart.srt")

```

```code
docs = loader.load()

```

```code
docs[0].page_content[:100]

```

```code
'<i>Corruption discovered\nat the core of the Banking Clan!</i> <i>Reunited, Rush Clovis\nand Senator A'

```
Slack DirectoryStripe


Stripe – LangChain中文网

6大核心模块（Modules)索引（Indexes)文档加载器（Document Loaders）示例Stripe

Stripe
本教程介绍了如何从Stripe REST API中加载数据到可以摄取到LangChain的格式，以及矢量化的示例用法。

```code
import os

from langchain.document_loaders import StripeLoader
from langchain.indexes import VectorstoreIndexCreator

```

Stripe API需要访问令牌，该令牌可以在Stripe控制面板中找到。
此文档加载器还需要一个
```code
resource
```
选项，该选项定义要加载的数据。
以下资源可用：

```code
balance_transations
```

Documentation (opens in a new tab)

```code
charges
```

Documentation (opens in a new tab)

```code
customers
```

Documentation (opens in a new tab)

```code
events
```

Documentation (opens in a new tab)

```code
refunds
```

Documentation (opens in a new tab)

```code
disputes
```

Documentation (opens in a new tab)

```code
stripe_loader = StripeLoader(os.environ["STRIPE_ACCESS_TOKEN"], "charges")

```

```code
# Create a vectorstore retriver from the loader
# see https://python.langchain.com/en/latest/modules/indexes/getting_started for more details

index = VectorstoreIndexCreator().from_loaders([stripe_loader])
stripe_doc_retriever = index.vectorstore.as_retriever()

```
SrtTelegram


Telegram – LangChain中文网

6大核心模块（Modules)索引（Indexes)文档加载器（Document Loaders）示例Telegram

Telegram

Telegram Messenger (opens in a new tab) 是一种全球可访问的免费增值、跨平台、加密、基于云的集中式即时消息服务。该应用程序还提供可选的端到端加密聊天和视频通话、VoIP、文件共享和其他一些功能。

This notebook covers how to load data from
```code
Telegram
```
into a format that can be ingested into LangChain.

```code
from langchain.document_loaders import TelegramChatLoader

```

```code
loader = TelegramChatLoader("example_data/telegram.json")

```

```code
loader.load()

```

```code
[Document(page_content="Henry on 2020-01-01T00:00:02: It's 2020...  Henry on 2020-01-01T00:00:04: Fireworks!  Grace ðŸ§¤ ðŸ\x8d’ on 2020-01-01T00:00:05: You're a minute late!  ", lookup_str='', metadata={'source': 'example_data/telegram.json'}, lookup_index=0)]

```
StripeTwitter


推特 Twitter – LangChain中文网

6大核心模块（Modules)索引（Indexes)文档加载器（Document Loaders）示例Twitter

推特 Twitter

推特 (opens in a new tab)是一个在线社交媒体和社交网络服务。

这个加载器使用
```code
Tweepy
```
Python包从一组
```code
Twitter
```
用户的推文中提取文本。
您必须使用您的
```code
Twitter API
```
令牌初始化加载器，并需要传入您想要提取的推特用户名。

```code
from langchain.document_loaders import TwitterTweetLoader

```

```code
#!pip install tweepy

```

```code
loader = TwitterTweetLoader.from_bearer_token(
oauth2_bearer_token="YOUR BEARER TOKEN",
twitter_users=['elonmusk'],
number_tweets=50,  # Default value is 100
)

# Or load from access token and consumer keys
# loader = TwitterTweetLoader.from_secrets(
# access_token='YOUR ACCESS TOKEN',
# access_token_secret='YOUR ACCESS TOKEN SECRET',
# consumer_key='YOUR CONSUMER KEY',
# consumer_secret='YOUR CONSUMER SECRET',
# twitter_users=['elonmusk'],
# number_tweets=50,
# )

```

```code
documents = loader.load()
documents[:5]

```

```code
[Document(page_content='@MrAndyNgo @REI One store after another shutting down', metadata={'created_at': 'Tue Apr 18 03:45:50 +0000 2023', 'user_info': {'id': 44196397, 'id_str': '44196397', 'name': 'Elon Musk', 'screen_name': 'elonmusk', 'location': 'A Shortfall of Gravitas', 'profile_location': None, 'description': 'nothing', 'url': None, 'entities': {'description': {'urls': []}}, 'protected': False, 'followers_count': 135528327, 'friends_count': 220, 'listed_count': 120478, 'created_at': 'Tue Jun 02 20:12:29 +0000 2009', 'favourites_count': 21285, 'utc_offset': None, 'time_zone': None, 'geo_enabled': False, 'verified': False, 'statuses_count': 24795, 'lang': None, 'status': {'created_at': 'Tue Apr 18 03:45:50 +0000 2023', 'id': 1648170947541704705, 'id_str': '1648170947541704705', 'text': '@MrAndyNgo @REI One store after another shutting down', 'truncated': False, 'entities': {'hashtags': [], 'symbols': [], 'user_mentions': [{'screen_name': 'MrAndyNgo', 'name': 'Andy Ngô 🏳\u200d🌈', 'id': 2835451658, 'id_str': '2835451658', 'indices': [0, 10]}, {'screen_name': 'REI', 'name': 'REI', 'id': 16583846, 'id_str': '16583846', 'indices': [11, 15]}], 'urls': []}, 'source': '<a href="http://twitter.com/download/iphone" rel="nofollow">Twitter for iPhone</a>', 'in_reply_to_status_id': 1648134341678051328, 'in_reply_to_status_id_str': '1648134341678051328', 'in_reply_to_user_id': 2835451658, 'in_reply_to_user_id_str': '2835451658', 'in_reply_to_screen_name': 'MrAndyNgo', 'geo': None, 'coordinates': None, 'place': None, 'contributors': None, 'is_quote_status': False, 'retweet_count': 118, 'favorite_count': 1286, 'favorited': False, 'retweeted': False, 'lang': 'en'}, 'contributors_enabled': False, 'is_translator': False, 'is_translation_enabled': False, 'profile_background_color': 'C0DEED', 'profile_background_image_url': 'http://abs.twimg.com/images/themes/theme1/bg.png', 'profile_background_image_url_https': 'https://abs.twimg.com/images/themes/theme1/bg.png', 'profile_background_tile': False, 'profile_image_url': 'http://pbs.twimg.com/profile_images/1590968738358079488/IY9Gx6Ok_normal.jpg', 'profile_image_url_https': 'https://pbs.twimg.com/profile_images/1590968738358079488/IY9Gx6Ok_normal.jpg', 'profile_banner_url': 'https://pbs.twimg.com/profile_banners/44196397/1576183471', 'profile_link_color': '0084B4', 'profile_sidebar_border_color': 'C0DEED', 'profile_sidebar_fill_color': 'DDEEF6', 'profile_text_color': '333333', 'profile_use_background_image': True, 'has_extended_profile': True, 'default_profile': False, 'default_profile_image': False, 'following': None, 'follow_request_sent': None, 'notifications': None, 'translator_type': 'none', 'withheld_in_countries': []}}),
Document(page_content='@KanekoaTheGreat @joshrogin @glennbeck Large ships are fundamentally vulnerable to ballistic (hypersonic) missiles', metadata={'created_at': 'Tue Apr 18 03:43:25 +0000 2023', 'user_info': {'id': 44196397, 'id_str': '44196397', 'name': 'Elon Musk', 'screen_name': 'elonmusk', 'location': 'A Shortfall of Gravitas', 'profile_location': None, 'description': 'nothing', 'url': None, 'entities': {'description': {'urls': []}}, 'protected': False, 'followers_count': 135528327, 'friends_count': 220, 'listed_count': 120478, 'created_at': 'Tue Jun 02 20:12:29 +0000 2009', 'favourites_count': 21285, 'utc_offset': None, 'time_zone': None, 'geo_enabled': False, 'verified': False, 'statuses_count': 24795, 'lang': None, 'status': {'created_at': 'Tue Apr 18 03:45:50 +0000 2023', 'id': 1648170947541704705, 'id_str': '1648170947541704705', 'text': '@MrAndyNgo @REI One store after another shutting down', 'truncated': False, 'entities': {'hashtags': [], 'symbols': [], 'user_mentions': [{'screen_name': 'MrAndyNgo', 'name': 'Andy Ngô 🏳\u200d🌈', 'id': 2835451658, 'id_str': '2835451658', 'indices': [0, 10]}, {'screen_name': 'REI', 'name': 'REI', 'id': 16583846, 'id_str': '16583846', 'indices': [11, 15]}], 'urls': []}, 'source': '<a href="http://twitter.com/download/iphone" rel="nofollow">Twitter for iPhone</a>', 'in_reply_to_status_id': 1648134341678051328, 'in_reply_to_status_id_str': '1648134341678051328', 'in_reply_to_user_id': 2835451658, 'in_reply_to_user_id_str': '2835451658', 'in_reply_to_screen_name': 'MrAndyNgo', 'geo': None, 'coordinates': None, 'place': None, 'contributors': None, 'is_quote_status': False, 'retweet_count': 118, 'favorite_count': 1286, 'favorited': False, 'retweeted': False, 'lang': 'en'}, 'contributors_enabled': False, 'is_translator': False, 'is_translation_enabled': False, 'profile_background_color': 'C0DEED', 'profile_background_image_url': 'http://abs.twimg.com/images/themes/theme1/bg.png', 'profile_background_image_url_https': 'https://abs.twimg.com/images/themes/theme1/bg.png', 'profile_background_tile': False, 'profile_image_url': 'http://pbs.twimg.com/profile_images/1590968738358079488/IY9Gx6Ok_normal.jpg', 'profile_image_url_https': 'https://pbs.twimg.com/profile_images/1590968738358079488/IY9Gx6Ok_normal.jpg', 'profile_banner_url': 'https://pbs.twimg.com/profile_banners/44196397/1576183471', 'profile_link_color': '0084B4', 'profile_sidebar_border_color': 'C0DEED', 'profile_sidebar_fill_color': 'DDEEF6', 'profile_text_color': '333333', 'profile_use_background_image': True, 'has_extended_profile': True, 'default_profile': False, 'default_profile_image': False, 'following': None, 'follow_request_sent': None, 'notifications': None, 'translator_type': 'none', 'withheld_in_countries': []}}),
Document(page_content='@KanekoaTheGreat The Golden Rule', metadata={'created_at': 'Tue Apr 18 03:37:17 +0000 2023', 'user_info': {'id': 44196397, 'id_str': '44196397', 'name': 'Elon Musk', 'screen_name': 'elonmusk', 'location': 'A Shortfall of Gravitas', 'profile_location': None, 'description': 'nothing', 'url': None, 'entities': {'description': {'urls': []}}, 'protected': False, 'followers_count': 135528327, 'friends_count': 220, 'listed_count': 120478, 'created_at': 'Tue Jun 02 20:12:29 +0000 2009', 'favourites_count': 21285, 'utc_offset': None, 'time_zone': None, 'geo_enabled': False, 'verified': False, 'statuses_count': 24795, 'lang': None, 'status': {'created_at': 'Tue Apr 18 03:45:50 +0000 2023', 'id': 1648170947541704705, 'id_str': '1648170947541704705', 'text': '@MrAndyNgo @REI One store after another shutting down', 'truncated': False, 'entities': {'hashtags': [], 'symbols': [], 'user_mentions': [{'screen_name': 'MrAndyNgo', 'name': 'Andy Ngô 🏳\u200d🌈', 'id': 2835451658, 'id_str': '2835451658', 'indices': [0, 10]}, {'screen_name': 'REI', 'name': 'REI', 'id': 16583846, 'id_str': '16583846', 'indices': [11, 15]}], 'urls': []}, 'source': '<a href="http://twitter.com/download/iphone" rel="nofollow">Twitter for iPhone</a>', 'in_reply_to_status_id': 1648134341678051328, 'in_reply_to_status_id_str': '1648134341678051328', 'in_reply_to_user_id': 2835451658, 'in_reply_to_user_id_str': '2835451658', 'in_reply_to_screen_name': 'MrAndyNgo', 'geo': None, 'coordinates': None, 'place': None, 'contributors': None, 'is_quote_status': False, 'retweet_count': 118, 'favorite_count': 1286, 'favorited': False, 'retweeted': False, 'lang': 'en'}, 'contributors_enabled': False, 'is_translator': False, 'is_translation_enabled': False, 'profile_background_color': 'C0DEED', 'profile_background_image_url': 'http://abs.twimg.com/images/themes/theme1/bg.png', 'profile_background_image_url_https': 'https://abs.twimg.com/images/themes/theme1/bg.png', 'profile_background_tile': False, 'profile_image_url': 'http://pbs.twimg.com/profile_images/1590968738358079488/IY9Gx6Ok_normal.jpg', 'profile_image_url_https': 'https://pbs.twimg.com/profile_images/1590968738358079488/IY9Gx6Ok_normal.jpg', 'profile_banner_url': 'https://pbs.twimg.com/profile_banners/44196397/1576183471', 'profile_link_color': '0084B4', 'profile_sidebar_border_color': 'C0DEED', 'profile_sidebar_fill_color': 'DDEEF6', 'profile_text_color': '333333', 'profile_use_background_image': True, 'has_extended_profile': True, 'default_profile': False, 'default_profile_image': False, 'following': None, 'follow_request_sent': None, 'notifications': None, 'translator_type': 'none', 'withheld_in_countries': []}}),
Document(page_content='@KanekoaTheGreat 🧐', metadata={'created_at': 'Tue Apr 18 03:35:48 +0000 2023', 'user_info': {'id': 44196397, 'id_str': '44196397', 'name': 'Elon Musk', 'screen_name': 'elonmusk', 'location': 'A Shortfall of Gravitas', 'profile_location': None, 'description': 'nothing', 'url': None, 'entities': {'description': {'urls': []}}, 'protected': False, 'followers_count': 135528327, 'friends_count': 220, 'listed_count': 120478, 'created_at': 'Tue Jun 02 20:12:29 +0000 2009', 'favourites_count': 21285, 'utc_offset': None, 'time_zone': None, 'geo_enabled': False, 'verified': False, 'statuses_count': 24795, 'lang': None, 'status': {'created_at': 'Tue Apr 18 03:45:50 +0000 2023', 'id': 1648170947541704705, 'id_str': '1648170947541704705', 'text': '@MrAndyNgo @REI One store after another shutting down', 'truncated': False, 'entities': {'hashtags': [], 'symbols': [], 'user_mentions': [{'screen_name': 'MrAndyNgo', 'name': 'Andy Ngô 🏳\u200d🌈', 'id': 2835451658, 'id_str': '2835451658', 'indices': [0, 10]}, {'screen_name': 'REI', 'name': 'REI', 'id': 16583846, 'id_str': '16583846', 'indices': [11, 15]}], 'urls': []}, 'source': '<a href="http://twitter.com/download/iphone" rel="nofollow">Twitter for iPhone</a>', 'in_reply_to_status_id': 1648134341678051328, 'in_reply_to_status_id_str': '1648134341678051328', 'in_reply_to_user_id': 2835451658, 'in_reply_to_user_id_str': '2835451658', 'in_reply_to_screen_name': 'MrAndyNgo', 'geo': None, 'coordinates': None, 'place': None, 'contributors': None, 'is_quote_status': False, 'retweet_count': 118, 'favorite_count': 1286, 'favorited': False, 'retweeted': False, 'lang': 'en'}, 'contributors_enabled': False, 'is_translator': False, 'is_translation_enabled': False, 'profile_background_color': 'C0DEED', 'profile_background_image_url': 'http://abs.twimg.com/images/themes/theme1/bg.png', 'profile_background_image_url_https': 'https://abs.twimg.com/images/themes/theme1/bg.png', 'profile_background_tile': False, 'profile_image_url': 'http://pbs.twimg.com/profile_images/1590968738358079488/IY9Gx6Ok_normal.jpg', 'profile_image_url_https': 'https://pbs.twimg.com/profile_images/1590968738358079488/IY9Gx6Ok_normal.jpg', 'profile_banner_url': 'https://pbs.twimg.com/profile_banners/44196397/1576183471', 'profile_link_color': '0084B4', 'profile_sidebar_border_color': 'C0DEED', 'profile_sidebar_fill_color': 'DDEEF6', 'profile_text_color': '333333', 'profile_use_background_image': True, 'has_extended_profile': True, 'default_profile': False, 'default_profile_image': False, 'following': None, 'follow_request_sent': None, 'notifications': None, 'translator_type': 'none', 'withheld_in_countries': []}}),
Document(page_content='@TRHLofficial What’s he talking about and why is it sponsored by Erik’s son?', metadata={'created_at': 'Tue Apr 18 03:32:17 +0000 2023', 'user_info': {'id': 44196397, 'id_str': '44196397', 'name': 'Elon Musk', 'screen_name': 'elonmusk', 'location': 'A Shortfall of Gravitas', 'profile_location': None, 'description': 'nothing', 'url': None, 'entities': {'description': {'urls': []}}, 'protected': False, 'followers_count': 135528327, 'friends_count': 220, 'listed_count': 120478, 'created_at': 'Tue Jun 02 20:12:29 +0000 2009', 'favourites_count': 21285, 'utc_offset': None, 'time_zone': None, 'geo_enabled': False, 'verified': False, 'statuses_count': 24795, 'lang': None, 'status': {'created_at': 'Tue Apr 18 03:45:50 +0000 2023', 'id': 1648170947541704705, 'id_str': '1648170947541704705', 'text': '@MrAndyNgo @REI One store after another shutting down', 'truncated': False, 'entities': {'hashtags': [], 'symbols': [], 'user_mentions': [{'screen_name': 'MrAndyNgo', 'name': 'Andy Ngô 🏳\u200d🌈', 'id': 2835451658, 'id_str': '2835451658', 'indices': [0, 10]}, {'screen_name': 'REI', 'name': 'REI', 'id': 16583846, 'id_str': '16583846', 'indices': [11, 15]}], 'urls': []}, 'source': '<a href="http://twitter.com/download/iphone" rel="nofollow">Twitter for iPhone</a>', 'in_reply_to_status_id': 1648134341678051328, 'in_reply_to_status_id_str': '1648134341678051328', 'in_reply_to_user_id': 2835451658, 'in_reply_to_user_id_str': '2835451658', 'in_reply_to_screen_name': 'MrAndyNgo', 'geo': None, 'coordinates': None, 'place': None, 'contributors': None, 'is_quote_status': False, 'retweet_count': 118, 'favorite_count': 1286, 'favorited': False, 'retweeted': False, 'lang': 'en'}, 'contributors_enabled': False, 'is_translator': False, 'is_translation_enabled': False, 'profile_background_color': 'C0DEED', 'profile_background_image_url': 'http://abs.twimg.com/images/themes/theme1/bg.png', 'profile_background_image_url_https': 'https://abs.twimg.com/images/themes/theme1/bg.png', 'profile_background_tile': False, 'profile_image_url': 'http://pbs.twimg.com/profile_images/1590968738358079488/IY9Gx6Ok_normal.jpg', 'profile_image_url_https': 'https://pbs.twimg.com/profile_images/1590968738358079488/IY9Gx6Ok_normal.jpg', 'profile_banner_url': 'https://pbs.twimg.com/profile_banners/44196397/1576183471', 'profile_link_color': '0084B4', 'profile_sidebar_border_color': 'C0DEED', 'profile_sidebar_fill_color': 'DDEEF6', 'profile_text_color': '333333', 'profile_use_background_image': True, 'has_extended_profile': True, 'default_profile': False, 'default_profile_image': False, 'following': None, 'follow_request_sent': None, 'notifications': None, 'translator_type': 'none', 'withheld_in_countries': []}})]

```
TelegramUnstructured File