yasbd-hooks: Reusable, opt-in callbacks for YASBD sentence boundary detection
If you deal with scientific text, you've probably run into a classic parsing headache: general English rule sets fail on edge cases like distinguishing chemical symbols (Br., Ca.) from abbreviations, or handling flattened OCR superscript citations (word.12,13).
Core YASBD stays generalist to avoid false positives and pipeline bloat, so Bruno spun up a companion package: yasbd-hooks. It handles these messy edge cases via clean, opt-in callbacks.
What it handles right now, at least partially:
- chemical_notation(): Protects element symbols and analytical terms from false sentence splits.
- academic_citations(): Keeps complex journal abbreviation structures intact.
- clause_boundaries(): Handles clause-level segmentation on specific punctuation.
- ocr_artifacts(): Recovers missing space boundaries and flattened superscript citation blocks.
- compose_hooks(): Chains multiple callbacks together cleanly without messing up character offsets.
We're looking for contributors to test more corpora, refine rules, and build out new domain hooks. Pull requests, test cases, and issues are very welcome.
Repo: brunnojob/yasbd-hooks
1 point · 1 comment
I've been working on a multilingual sentence boundary detection library in Python to replace PySBD
I've been developing YASBD, a Python library for sentence boundary detection.
The goal was to build something lightweight, fast, and accurate while supporting a wider range of languages.
It's currently at 39 supported languages, and I've spent quite some time testing and refining its behavior across different kinds of text.
I wrote about the development journey and some of the testing that got it here in a blog (link in the comment).
1 point · 1 comment
yasbd-hooks: Reusable, opt-in callbacks for YASBD sentence boundary detection
If you deal with scientific text, you've probably run into a classic parsing headache: general English rule sets fail on edge cases like distinguishing chemical symbols (Br., Ca.) from abbreviations, or handling flattened OCR superscript citations (word.12,13).
Core YASBD stays generalist to avoid false positives and pipeline bloat, so Bruno spun up a companion package: yasbd-hooks. It handles these messy edge cases via clean, opt-in callbacks.
What it handles right now, at least partially:
- chemical_notation(): Protects element symbols and analytical terms from false sentence splits.
- academic_citations(): Keeps complex journal abbreviation structures intact.
- clause_boundaries(): Handles clause-level segmentation on specific punctuation.
- ocr_artifacts(): Recovers missing space boundaries and flattened superscript citation blocks.
- compose_hooks(): Chains multiple callbacks together cleanly without messing up character offsets.
We're looking for contributors to test more corpora, refine rules, and build out new domain hooks. Pull requests, test cases, and issues are very welcome.
Repo: brunnojob/yasbd-hooks
1 point
Mentions
yasbd-hooks: Reusable, opt-in callbacks for YASBD sentence boundary detection If you deal with scientific text, you've probably run into a classic parsing headache: general English rule sets fail on edge cases like distinguishing chemical symbols (Br., Ca.) from abbreviations, or handling flattened OCR superscript citations (word.12,13). Core YASBD stays generalist to avoid false positives and pipeline bloat, so Bruno spun up a companion package: yasbd-hooks. It handles these messy edge cases via clean, opt-in callbacks. What it handles right now, at least partially: - chemical_notation(): Protects element symbols and analytical terms from false sentence splits. - academic_citations(): Keeps complex journal abbreviation structures intact. - clause_boundaries(): Handles clause-level segmentation on specific punctuation. - ocr_artifacts(): Recovers missing space boundaries and flattened superscript citation blocks. - compose_hooks(): Chains multiple callbacks together cleanly without messing up character offsets. We're looking for contributors to test more corpora, refine rules, and build out new domain hooks. Pull requests, test cases, and issues are very welcome. Repo: brunnojob/yasbd-hooks
I've been working on a multilingual sentence boundary detection library in Python to replace PySBD I've been developing YASBD, a Python library for sentence boundary detection. The goal was to build something lightweight, fast, and accurate while supporting a wider range of languages. It's currently at 39 supported languages, and I've spent quite some time testing and refining its behavior across different kinds of text. I wrote about the development journey and some of the testing that got it here in a blog (link in the comment).
yasbd-hooks: Reusable, opt-in callbacks for YASBD sentence boundary detection If you deal with scientific text, you've probably run into a classic parsing headache: general English rule sets fail on edge cases like distinguishing chemical symbols (Br., Ca.) from abbreviations, or handling flattened OCR superscript citations (word.12,13). Core YASBD stays generalist to avoid false positives and pipeline bloat, so Bruno spun up a companion package: yasbd-hooks. It handles these messy edge cases via clean, opt-in callbacks. What it handles right now, at least partially: - chemical_notation(): Protects element symbols and analytical terms from false sentence splits. - academic_citations(): Keeps complex journal abbreviation structures intact. - clause_boundaries(): Handles clause-level segmentation on specific punctuation. - ocr_artifacts(): Recovers missing space boundaries and flattened superscript citation blocks. - compose_hooks(): Chains multiple callbacks together cleanly without messing up character offsets. We're looking for contributors to test more corpora, refine rules, and build out new domain hooks. Pull requests, test cases, and issues are very welcome. Repo: brunnojob/yasbd-hooks