markdown-tokens/README.md

markdown-tokens

Tokenize Markdown into a line-based stream of spans. Markers stay in the text. Each span knows where it starts and ends, and whether it opens or closes a formatting construct.

import { tokenize } from 'markdown-tokens';

tokenize('Hello **world**');
// [
//   [
//     { startIndex: 0, endIndex: 6, content: 'Hello ' },
//     { startIndex: 6, endIndex: 8, content: '**', formattingStart: 'bold' },
//     { startIndex: 8, endIndex: 13, content: 'world' },
//     { startIndex: 13, endIndex: 15, content: '**', formattingEnd: 'bold' }
//   ]
// ]

Install

bun add markdown-tokens
# or
npm install markdown-tokens

Requires an ESM runtime. The package ships as "type": "module".

API

tokenize(markdown: string)

Returns Token[][]: one array per input line (including blank lines as []).

Token

Field	Type	Required	Notes
startIndex	number	yes	Inclusive, relative to the line
endIndex	number	yes	Exclusive, relative to the line
content	string	yes	Exact substring, markers included
formattingStart	string	no	Opens a construct
formattingEnd	string	no	Closes a construct
formattingMetadata	string	no	Extra tag on a span (language, heading level)

A token never carries both formattingStart and formattingEnd.

Formatting types

Value	Used for
bold	**…** / __…__
italic	*…* / _…_
strikethrough	~~…~~
code	`…`
hyperlinkText	[text]
hyperlinkAddress	(url) after a link
imageText	![alt]
imageAddress	(src) after an image
listItem	- prefix
orderedListItem	1. style prefix
codeblock	``` open / close
heading	#…###### prefix
quote	> prefix, or a continued quote line

Metadata values

Value	Meaning
codeblockLanguage	Text after the opening fence (js, ts, …)
h1 … h6	Heading level on the # token

Nested markers are emitted as separate open/close tokens, so **_bold italic_** produces four formatting tokens around the inner text.

Block constructs worth knowing about:

• List and heading prefixes only emit formattingStart (no matching end token).
• A quote that continues on the next line without > still gets formattingStart: 'quote' on that line.
• Inside a fenced code block, body lines are plain content tokens until the closing fence.

Development

bun install
bun test

Behavior is defined by mod.test.js. That suite walks a full Markdown sample and checks both the Joi schema and exact token output.

Implementations

Several model runs produced independent tokenize implementations against the same tests. Each lives on its own branch, named {version}_{model}-{tooling}:

Branch	Model / tooling
0.1.0_claude4-kiro	Claude 4 via Kiro
0.1.0_claude4-kiro_2	Claude 4 via Kiro (second run)
0.1.0_claude4-jb+ghcp	Claude 4 via JetBrains + GitHub Copilot
0.1.0_gpt5-jb+ghcp	GPT-5 via JetBrains + GitHub Copilot
0.1.1_claude4.5-kiro	Claude 4.5 via Kiro
0.1.3_gemini3flash-antigravity	Gemini 3 Flash via Antigravity

master keeps the stub and the test contract. Check out a branch above to use a completed implementation.

License

MIT