108 lines
3.9 KiB
Markdown
108 lines
3.9 KiB
Markdown
# markdown-tokens
|
|
|
|
Tokenize Markdown into a line-based stream of spans. Markers stay in the text. Each span knows where it starts and ends, and whether it opens or closes a formatting construct.
|
|
|
|
```js
|
|
import { tokenize } from 'markdown-tokens';
|
|
|
|
tokenize('Hello **world**');
|
|
// [
|
|
// [
|
|
// { startIndex: 0, endIndex: 6, content: 'Hello ' },
|
|
// { startIndex: 6, endIndex: 8, content: '**', formattingStart: 'bold' },
|
|
// { startIndex: 8, endIndex: 13, content: 'world' },
|
|
// { startIndex: 13, endIndex: 15, content: '**', formattingEnd: 'bold' }
|
|
// ]
|
|
// ]
|
|
```
|
|
|
|
## Install
|
|
|
|
```bash
|
|
bun add markdown-tokens
|
|
# or
|
|
npm install markdown-tokens
|
|
```
|
|
|
|
Requires an ESM runtime. The package ships as `"type": "module"`.
|
|
|
|
## API
|
|
|
|
### `tokenize(markdown: string)`
|
|
|
|
Returns `Token[][]`: one array per input line (including blank lines as `[]`).
|
|
|
|
#### Token
|
|
|
|
| Field | Type | Required | Notes |
|
|
| --- | --- | --- | --- |
|
|
| `startIndex` | `number` | yes | Inclusive, relative to the line |
|
|
| `endIndex` | `number` | yes | Exclusive, relative to the line |
|
|
| `content` | `string` | yes | Exact substring, markers included |
|
|
| `formattingStart` | `string` | no | Opens a construct |
|
|
| `formattingEnd` | `string` | no | Closes a construct |
|
|
| `formattingMetadata` | `string` | no | Extra tag on a span (language, heading level) |
|
|
|
|
A token never carries both `formattingStart` and `formattingEnd`.
|
|
|
|
#### Formatting types
|
|
|
|
| Value | Used for |
|
|
| --- | --- |
|
|
| `bold` | `**…**` / `__…__` |
|
|
| `italic` | `*…*` / `_…_` |
|
|
| `strikethrough` | `~~…~~` |
|
|
| `code` | `` `…` `` |
|
|
| `hyperlinkText` | `[text]` |
|
|
| `hyperlinkAddress` | `(url)` after a link |
|
|
| `imageText` | `![alt]` |
|
|
| `imageAddress` | `(src)` after an image |
|
|
| `listItem` | `- ` prefix |
|
|
| `orderedListItem` | `1. ` style prefix |
|
|
| `codeblock` | ` ``` ` open / close |
|
|
| `heading` | `#`…`######` prefix |
|
|
| `quote` | `>` prefix, or a continued quote line |
|
|
|
|
#### Metadata values
|
|
|
|
| Value | Meaning |
|
|
| --- | --- |
|
|
| `codeblockLanguage` | Text after the opening fence (`js`, `ts`, …) |
|
|
| `h1` … `h6` | Heading level on the `#` token |
|
|
|
|
Nested markers are emitted as separate open/close tokens, so `**_bold italic_**` produces four formatting tokens around the inner text.
|
|
|
|
Block constructs worth knowing about:
|
|
|
|
- List and heading prefixes only emit `formattingStart` (no matching end token).
|
|
- A quote that continues on the next line without `>` still gets `formattingStart: 'quote'` on that line.
|
|
- Inside a fenced code block, body lines are plain content tokens until the closing fence.
|
|
|
|
## Development
|
|
|
|
```bash
|
|
bun install
|
|
bun test
|
|
```
|
|
|
|
Behavior is defined by `mod.test.js`. That suite walks a full Markdown sample and checks both the Joi schema and exact token output.
|
|
|
|
## Implementations
|
|
|
|
Several model runs produced independent `tokenize` implementations against the same tests. Each lives on its own branch, named `{version}_{model}-{tooling}`:
|
|
|
|
| Branch | Model / tooling |
|
|
| --- | --- |
|
|
| [`0.1.0_claude4-kiro`](https://git.kaki87.net/KaKi87/markdown-tokens/src/branch/0.1.0_claude4-kiro) | Claude 4 via Kiro |
|
|
| [`0.1.0_claude4-kiro_2`](https://git.kaki87.net/KaKi87/markdown-tokens/src/branch/0.1.0_claude4-kiro_2) | Claude 4 via Kiro (second run) |
|
|
| [`0.1.0_claude4-jb+ghcp`](https://git.kaki87.net/KaKi87/markdown-tokens/src/branch/0.1.0_claude4-jb%2Bghcp) | Claude 4 via JetBrains + GitHub Copilot |
|
|
| [`0.1.0_gpt5-jb+ghcp`](https://git.kaki87.net/KaKi87/markdown-tokens/src/branch/0.1.0_gpt5-jb%2Bghcp) | GPT-5 via JetBrains + GitHub Copilot |
|
|
| [`0.1.1_claude4.5-kiro`](https://git.kaki87.net/KaKi87/markdown-tokens/src/branch/0.1.1_claude4.5-kiro) | Claude 4.5 via Kiro |
|
|
| [`0.1.3_gemini3flash-antigravity`](https://git.kaki87.net/KaKi87/markdown-tokens/src/branch/0.1.3_gemini3flash-antigravity) | Gemini 3 Flash via Antigravity |
|
|
|
|
[`master`](https://git.kaki87.net/KaKi87/markdown-tokens/src/branch/master) keeps the stub and the test contract. Check out a branch above to use a completed implementation.
|
|
|
|
## License
|
|
|
|
MIT
|
|
|