hO d dl mZ d dlZd dlZd dlZd dlmZ d dlmZm Z m
Z
m Z m Z m
Z
mZmZ d dlZd dlmZ d dlmZmZ d dlmZ erd dlmZmZ d d lm Z G d
d e
Z G d d
Z G d d Z e G d de Z y) ) annotationsN)StringIO)
TYPE_CHECKINGAnyCallableLiteralOptional TypedDictUnioncast)beta)BaseDocumentTransformerDocument) RecursiveCharacterTextSplitter)IterableSequence) PageElementc : e Zd ZU dZded< ded< ded< ded< y) ElementTypezElement type as typed dict.strurlxpathcontentdict[str, str]metadataN)__name__
__module__ __qualname____doc____annotations__ [/var/www/html/eduruby.in/venv/lib/python3.12/site-packages/langchain_text_splitters/html.pyr r s % H J
L r" r c X e Zd ZdZ d d dZd
dZ d d dZd
dZddZy)HTMLHeaderTextSplittera~
Split HTML content into structured Documents based on specified headers.
Splits HTML content by detecting specified header tags (e.g., , ) and
creating hierarchical Document objects that reflect the semantic structure
of the original content. For each identified section, the splitter associates
the extracted text with metadata corresponding to the encountered headers.
If no specified headers are found, the entire content is returned as a single
Document. This allows for flexible handling of HTML input, ensuring that
information is organized according to its semantic headers.
The splitter provides the option to return each HTML element as a separate
Document or aggregate them into semantically meaningful chunks. It also
gracefully handles multiple levels of nested headers, creating a rich,
hierarchical representation of the content.
Example:
.. code-block:: python
from langchain_text_splitters.html_header_text_splitter import (
HTMLHeaderTextSplitter,
)
# Define headers for splitting on h1 and h2 tags.
headers_to_split_on = [("h1", "Main Topic"), ("h2", "Sub Topic")]
splitter = HTMLHeaderTextSplitter(
headers_to_split_on=headers_to_split_on,
return_each_element=False
)
html_content = """
Introduction
Welcome to the introduction section.
Background
Some background details here.
Conclusion
Final thoughts.
"""
documents = splitter.split_text(html_content)
# 'documents' now contains Document objects reflecting the hierarchy:
# - Document with metadata={"Main Topic": "Introduction"} and
# content="Introduction"
# - Document with metadata={"Main Topic": "Introduction"} and
# content="Welcome to the introduction section."
# - Document with metadata={"Main Topic": "Introduction",
# "Sub Topic": "Background"} and content="Background"
# - Document with metadata={"Main Topic": "Introduction",
# "Sub Topic": "Background"} and content="Some background details here."
# - Document with metadata={"Main Topic": "Conclusion"} and
# content="Conclusion"
# - Document with metadata={"Main Topic": "Conclusion"} and
# content="Final thoughts."
c t |d | _ t | j | _ | j D cg c] \ }}| c}}| _ || _ yc c}}w )a Initialize with headers to split on.
Args:
headers_to_split_on: A list of (header_tag,
header_name) pairs representing the headers that define splitting
boundaries. For example, [("h1", "Header 1"), ("h2", "Header 2")]
will split content by and tags, assigning their textual
content to the Document metadata.
return_each_element: If True, every HTML element encountered
(including headers, paragraphs, etc.) is returned as a separate
Document. If False, content under the same header hierarchy is
aggregated into fewer Documents.
c $ t | d dd S )Nr )int)xs r# z1HTMLHeaderTextSplitter.__init__..{ s s1Q48} r" )keyN)sortedheaders_to_split_ondictheader_mapping header_tagsreturn_each_element)selfr. r2 tag_s r# __init__z HTMLHeaderTextSplitter.__init__g sW & $* %<$
#4#;#; <.2.F.FGFCCG#6 Hs Ac 6 | j t | S )a] Split the given text into a list of Document objects.
Args:
text: The HTML text to split.
Returns:
A list of split Document objects. Each Document contains
`page_content` holding the extracted text and `metadata` that maps
the header hierarchy to their corresponding titles.
split_text_from_filer r3 texts r#
split_textz!HTMLHeaderTextSplitter.split_text s (($ 88r" c
( |j d| t j |fd|j d| i|j D ci c] \ }}|dk7 s || c}}}|j | j |j S c c}}w )a= Fetch text content from a URL and split it into documents.
Args:
url: The URL to fetch content from.
timeout: Timeout for the request. Defaults to 10.
**kwargs: Additional keyword arguments for the request.
Returns:
A list of split Document objects. Each Document contains
`page_content` holding the extracted text and `metadata` that maps
the header hierarchy to their corresponding titles.
Raises:
requests.RequestException: If the HTTP request fails.
timeout)
setdefaultrequestsgetitemsraise_for_statusr< r; )r3 r r> kwargskvresponses r# split_text_from_urlz*HTMLHeaderTextSplitter.split_text_from_url s $ )W-<<
JJy'2
!' A1!y.q!tA
!!#x}}-- Bs
BBc t |t r't |d 5 }|j }ddd n|j }t | j S # 1 sw Y #xY w)a Split HTML content from a file into a list of Document objects.
Args:
file: A file path or a file-like object containing HTML content.
Returns:
A list of split Document objects. Each Document contains
`page_content` holding the extracted text and `metadata` that maps
the header hierarchy to their corresponding titles.
zutf-8)encodingN)
isinstancer openreadlist_generate_documents)r3 filef html_contents r# r9 z+HTMLHeaderTextSplitter.split_text_from_file sb dC dW-
( vvx
(
( 99;LD,,\:;;
(
(s A""A+c # K ddl m} ||d }|j r |j n|}i g d fd }|g}|r|j } t | j }
ddlm |j fdt |
D t | d d } | sa| j d
d D cg c] } t | j }
} d
j d |
D }|st! t | j" }| | j$ v r| j& s
| }|r| t) | dd } j- D cg c] \ }\ }}}||k\ s| }}}}}|D ] } |= | j. | }|||f |< j- D ci c] \ }}||d
}}}t1 || n j- D cg c] \ }\ }}}||k s| }}}}|D ] } |= | j& r6 j- D ci c] \ }}||d
} }}t1 || n j3 | |r| j& s | }|r| yyy# t $ r}d}t | |d}~ww xY wc c} w # t* $ r d}Y dw xY wc c}}}}w c c}}w c c}}}w c c}}w w)a Private method that performs a DFS traversal over the DOM and yields.
Document objects on-the-fly. This approach maintains the same splitting
logic (headers vs. non-headers, chunking, etc.) while walking the DOM
explicitly in code.
Args:
html_content: The raw HTML content.
Yields:
Document objects as they are created.
r
BeautifulSoupzEUnable to import BeautifulSoup. Please install via `pip install bs4`.N html.parserc sydj d D } j | j syj D ci c] \ }}||d
}}}t | | S c c}}w )z6Finalize the accumulated chunk into a single Document.Nz
c 3 B K | ] }|j s| y wNstrip).0lines r# zUHTMLHeaderTextSplitter._generate_documents..finalize_chunk.. s $Tdtzz|T$Ts r page_contentr )joinclearr[ rB r )
final_textrE rF
final_metaactive_headers
current_chunks r# finalize_chunkzBHTMLHeaderTextSplitter._generate_documents..finalize_chunk ss $Tm$TTJ ! ##%.<.B.B.DEda!QqT'EJEjI I Fs A1)Tagc 3 < K | ] }t | s| y wrY )rK )r\ childrh s r# r^ z=HTMLHeaderTextSplitter._generate_documents.. s E39Os nameTFstring recursive c 3 & K | ] }|s| y wrY r! )r\ elems r# r^ z=HTMLHeaderTextSplitter._generate_documents.. s H$4 Hs r( i' r_ )returnzOptional[Document])bs4rU ImportErrorbodypoprN children bs4.elementrh extendreversedgetattrfind_allr r[ ra lenparentsr1 r2 r)
ValueErrorrB r0 r append) r3 rR rU emsgsoupru rg stacknoderw r4 rj
text_elements node_text dom_depthdoclevelrE r5 lvldheaders_to_remover, header_namerF header_metaheaders_out_of_scopemetarh re rf s @@@r# rO z*HTMLHeaderTextSplitter._generate_documents s * ) \=9 IItyy4
;=#%
J 99;DDMM*H ' LL #+H#5
$-C "]]$%] H E
" M H- HHI D ./I d&&&//(*C ! ! ABLE -;,@,@,B% %(![aacUlA%! % - ,C&s+, # 1 1# 6 /8%.K{+ 4B3G3G3I J41aq!A$w J J I LL +9*>*>*@( (&!YaAIPQMA($ ( 0 ,C&s+, ++0>0D0D0FG1AqtGGDG" DII "((3C H '' "C (E *W
c" ) *V * " ! E!% K( Hs KI7 B'K4 JA"K7J KJ-
,J-
09K)J59'K J;2J;6-K#K3&K K7 J JJK J*&K)J** KN)F)r. list[tuple[str, str]]r2 boolrr Noner; r rr list[Document])
)r r r> r) rD r rr r rP r rr r )rR r rr r ) r r r r r6 r< rH r9 rO r! r" r# r% r% ' sh =D %*7 27 "7
74 9 (*..!$.58. .6<$tr" r% c b e Zd ZdZ d
dZd dZd dZ d
ddZddZddZ dd Z
y)HTMLSectionSplitterz\Splitting HTML files based on specified tag and font sizes.
Requires lxml package.
c t | | _ t j t j
dz j
| _ || _ y)ay Create a new HTMLSectionSplitter.
Args:
headers_to_split_on: list of tuples of headers we want to track mapped to
(arbitrary) keys for metadata. Allowed header values: h1, h2, h3, h4,
h5, h6 e.g. [("h1", "Header 1"), ("h2", "Header 2"].
**kwargs (Any): Additional optional arguments for customizations.
z xsl/converting_to_header.xsltN) r/ r. pathlibPath__file__parentabsolute xslt_pathrD )r3 r. rD s r# r6 z HTMLSectionSplitter.__init__8 sA $((;#< LL " ) ),K K
(*
r" c g g }}|D ]8 }|j |j |j |j : | j || }t di | j
}|j
| S )zSplit documents.) metadatasr! )r r` r create_documentsr rD split_documents)r3 documentstextsr r results
text_splitters r# r z#HTMLSectionSplitter.split_documentsL sw ry +C LL)) * S\\ * + '''C6EE
,,W55r" c 6 | j t | S )zKSplit HTML text string.
Args:
text: HTML text
r8 r: s r# r< z HTMLSectionSplitter.split_textX s (($ 88r" Nc |xs i gt | z }g }t | D ] \ }}| j | D ] }t j || }|j
D ]' } |j
| dk( s|d |j
| < ) i ||j
}t
|j | }
|j |
|S )z&Create documents from a list of texts.#TITLE#Titler_ ) r} enumerater< copydeepcopyr r r` r ) r3 r r
_metadatasr ir; chunkr r, new_docs r# r z$HTMLSectionSplitter.create_documents` s 32$U"3
' *GAt .
* ==A 7 >> @C ~~c*i7.6w.?s+@ :h9%..9"0B0BXV )
* * r" c ddl m} ||d }t | j j }g }|j
dg| }t | D ] \ }} t d| }
|dk( rd} d } g }
n(|
j j } |
j } g }
|
j D ]B }|d
z t | k r
|||d
z k( r n$t |t s2|
j! | D d j# |
j }|d k7 s|j! | || d
|S # t $ r}d}t | |d}~ww xY w)a Split an HTML document into sections based on specified header tags.
This method uses BeautifulSoup to parse the HTML content and divides it into
sections based on headers defined in `headers_to_split_on`. Each section
contains the header text, content under the header, and the tag name.
Args:
html_doc (str): The HTML document to be split into sections.
Returns:
List[Dict[str, Optional[str]]]: A list of dictionaries representing
sections.
Each dictionary contains:
* 'header': The header text or a default title for the first section.
* 'content': The content under the header.
* 'tag_name': The name of the header tag (e.g., "h1", "h2").
r rT zzUnable to import BeautifulSoup/PageElement, please install with `pip install bs4`.NrV ru r r h1r( ro )headerr tag_name)rs rU rt rN r. keysr| r r r; r[ rk
next_elementsr} rK r r ra )r3 html_docrU r r r headerssectionsr r header_elementcurrent_headercurrent_header_tagsection_contentelementr s r# split_html_by_headersz)HTMLSectionSplitter.split_html_by_headersr se & * ) X}5t//446702-- 2' 23"7+ IAv !- 8NAv!*%)"-/!/!4!4!:!:!<%3%8%8""$)77
4q53w<'Gwq1u~,E gs+#**73
4
hh/557G"}"0#*$6% 4 M *C c" ) *s D* * E3EEc ddl m} |j d }|j ddd }|j
j }|j t | | }|j | j | } |j | | }
|
| } t | S # t $ r}d}t | |d}~ww xY w)
a Convert specific HTML tags to headers using an XSLT transformation.
This method uses an XSLT file to transform the HTML content, converting
certain tags into headers for easier parsing. If no XSLT path is provided,
the HTML content is returned unchanged.
Args:
html_content (str): The HTML content to be transformed.
Returns:
str: The transformed HTML content as a string.
r )etreez>Unable to import lxml, please install with `pip install lxml`.NT)
no_networkF)resolve_entitiesr load_dtd)access_control) lxmlr rt
HTMLParser XMLParserXSLTAccessControlDENY_ALLparser r XSLTr ) r3 rR r r r html_parser xslt_parseractree xslt_tree transformresults r# convert_possible_tags_to_headerz3HTMLSectionSplitter.convert_possible_tags_to_header s * " &&$&7 oo "te &
$
$
-
-{{8L1;?KK < JJyJ< 4 6{' *RC c" ) *s B! ! B=*B88B=c |j }| j | }| j | }|D cg c]: }t t d|d | j
t
|d |d i < c}S c c}w )zSplit HTML content from a file into a list of Document objects.
Args:
file: A file path or a file-like object containing HTML content.
Returns:
A list of split Document objects.
r r r r )r )getvaluer r r r r. r )r3 rP file_contentr sections r# r9 z(HTMLSectionSplitter.split_text_from_file s }} ;;LI --l; $
UGI ./,,S1D-EF I
s ?A9)r. r rD r rr r )r zIterable[Document]rr r r rY )r list[str]r z Optional[list[dict[Any, Any]]]rr r )r r rr z list[dict[str, Optional[str]]])rR r rr r r ) r r r r r6 r r< r r r r9 r! r" r# r r 2 sg
2
(
69 MQ +I $;z"H
r" r c e Zd ZdZdddddddddddddddddd dd Zdd
Z dd Zdd Zdd
ZddZ ddZ
ddZ ddZ ddZ
d dZy) HTMLSemanticPreservingSplittera Split HTML content preserving semantic structure.
Splits HTML content by headers into generalized chunks, preserving semantic
structure. If chunks exceed the maximum chunk size, it uses
RecursiveCharacterTextSplitter for further splitting.
The splitter preserves full HTML elements (e.g., , ) and converts
links to Markdown-like links. It can also preserve images, videos, and audio
elements by converting them into Markdown format. Note that some chunks may
exceed the maximum size to maintain semantic integrity.
.. versionadded: 0.3.5
Example:
.. code-block:: python
from langchain_text_splitters.html import HTMLSemanticPreservingSplitter
def custom_iframe_extractor(iframe_tag):
```
Custom handler function to extract the 'src' attribute from an