The problem was larger than the project
Most Flamehaven Lab Notes begin after a project has already gone through some abuse.
We build something, test it, find the part that does not behave as cleanly as the architecture suggested, and keep working until there is something worth documenting.
Sometimes the result is a functioning mechanism. Sometimes the useful result is a failed assumption, a test contract that turned out to be weaker than the implementation it was supposed to verify, or a boundary that had to move somewhere less convenient.
One recent project spent days stalled before implementation because the test contract itself was not yet trustworthy. That kind of detour is common in our work: the next engineering problem often turns into a research problem before it becomes code.
A state-machine problem can lead into stopping theory and state semantics. An on-device constraint can push us into inference research. A BioAI project may depend on a qualification buried in a methods section, while another implementation turns on a mathematical expression in an appendix.
Flamehaven has three people, and at the moment much of the internal research, architecture, implementation and technical review is being carried by two of us.
There is no sensible way for two people to hold that research surface in their heads. We stopped expecting ourselves to.
There is another project behind several of our projects
When the next engineering decision exceeds what we can justify locally, we often leave the project for a while and go back to the research.
That may mean an arXiv paper, technical documentation, a repository, an engineering article, a benchmark report, or occasionally a conference talk or transcript. We bring the relevant material into a separate ingestion layer, turn it into working context, and then return to the code.
engineering problem
->
knowledge boundary
->
external research
->
research-ingestion layer
->
AI-usable working knowledge
->
reasoning / design revision
->
code / test / experiment
->
next boundary
(back to external research)
We have been doing this long enough that the ingestion layer became a project of its own.
The technology at its centre began in a disappointingly old-fashioned form.
It was a prompt.
There is no service here, and no framework.
The Flamehaven Knowledge Extractor is a single natural-language specification, currently v6.8.2, that a capable long-context model executes against a supplied source. It has a version, declared budgets and a precedence order, and it produces an artifact another session can retrieve from.
💾 Full specification: Flamehaven Knowledge Extractor v6.8.2
Open the link and the first impression is not flattering.
It reads as a wall of terse directives, RFC keywords and YAML fragments, which is roughly what a SKILL.md looks like if you have not met one before.
What makes it different
That is what it is.
It is not documentation for another system. It is the system the model executes. States, budgets, precedence and failure conditions live inside the specification itself.
Most extraction prompts, including several of ours, are written so that the output looks complete. Ask for structured claims and you will get structured claims, whether or not the source supported them.
This specification is built the other way around. Its precedence order puts compression below evidence, so when protected material will not fit the budget the artifact reports budget_status: overflow_required instead of trimming.
Unresolved relations are allowed to stay UNKNOWN. A check that never ran is recorded as not_run, which is a different value from a check that ran and found nothing.
Claims the extractor derived are separated from claims the source made, so a later reader can delete the first without losing the second. And the run is not permitted to describe its own self-checking as verification, because the invariant block says L1 != INDEPENDENT_VERIFICATION.
🔥 Most of the design effort went into making the artifact report **where the run failed, why it failed, and what remained unresolved**. That is an unusual thing to optimise a prompt for, and it is why the file is 3,300 lines instead of thirty.
How it works
Using it takes three inputs and one invocation. You paste the specification, the document you want extracted - a paper, a technical article, a talk transcript - and a profile naming what kind of document it is:
MASTER SPEC
+ COMPLETE SOURCE
+ SOURCE PROFILE
->
KNOWLEDGE ARTIFACT
The source profile is one of research-paper, technical-document or youtube-transcript, describing document morphology, with an optional domain overlay (currently only bio) describing semantic contract. Those two axes stay orthogonal; the specification refuses combined profile names. execution_model is single-pass, and every phase runs inside that one invocation.
What comes back is not prose. It is a set of claims, each labelled as source-attributable or extractor-derived, each carrying a locator into the original document, with explicit states for the things the run could not resolve.
It opens with about eight lines of doctrine that do more work than any other section:
ABSTAIN>FABRICATE
SOURCE_FORM>MODEL_REPAIR
STRUCTURE_FIRST>LINEAR_READ
PRESERVE_FIRST>RECONCILE_LATER
SEMANTICS>SERIALIZATION
NOT_RUN!=RUN_NONE
CHECK!=VERIFICATION
SOURCE!=IR
None of that is subtle. It is written that way because ambiguity in a normative clause becomes ambiguity in every execution that follows.
IR is the specification's term for execution state - coverage maps, candidate groups, internal scan results - which the pipeline needs and which is not knowledge about the source. Uppercase names like NOT_RUN and RUN_NONE are doctrine-level distinctions; the lowercase not_run and not_evaluated appearing later are the field values an artifact actually carries.
The rest of this article is what those 3,300 lines are for: which rules are in there, why each one was added, what broke when we tested the output against the paper it came from, and which of those rules we now think should not be in a prompt at all.
Why it stayed a prompt, and not an agent
We started closer to one, and it is the reason this specification exists.
An early workflow had Gemini fetch and read an arXiv paper through a browser. The output looked like a competent analysis. It was not, because later sections and appendix material had never entered the effective context, and nothing in the answer indicated a gap.
When a system acquires its own sources, the evidence that actually entered the execution stops being something you controlled or can inspect afterwards. Access failure becomes invisible precisely when it matters.
That is the property we needed to hold fixed. The primary source is supplied, not fetched, and two adjacent lines near the top of the specification settle the rest of it:
Model memory MAY assist parsing/query construction.
Model memory MUST_NOT be evidence.
Retrieval still exists, and it is bounded rather than autonomous. The specification permits at most one retrieval phase, with a declared budget:
retrieval_budget:
queries: 3
supplemental_sources: 5
Anything retrieved enters as SUPPLEMENTAL, which may clarify or challenge the primary source and MUST_NOT overwrite PRIMARY. An autonomous agent would decide its own next action; here the action budget is declared before the run and the retrieved material is subordinate by contract.
Orchestration would not have answered the question we actually had. Agents still need instructions, selected context, tool contracts, schemas and action boundaries. Adding execution surfaces does not remove the problem of deciding what information reaches the model and what evidence permits the system to act again. It puts another layer on top of that problem.
Why not code either
Capacity was the mundane reason.
Two people carrying looping systems, on-device inference, mathematical mechanisms, BioAI and verification cannot also maintain an ingestion product, so a specification we can edit in an afternoon and rerun the same evening survives our actual week better than another service would.
Portability mattered too. We wanted the same object to run outside our own infrastructure, against somebody else's source, with whatever capable model they already pay for.
The more interesting reason appeared later.
Natural language turned out to be a cheap place to find out whether a proposed rule was even enforceable by the model before committing any software to it. Writing a constraint costs an afternoon, and watching the same rule survive repeated executions tells you something about whether it belongs in the prompt at all.
What the prompt actually has to do
Asking a frontier model to summarise a paper is not interesting. They do it well.
The engineering problem is compression under a constraint that summaries do not respect: how do you hand a paper to a model months later without handing it the whole paper, and without losing the parts that turn out to matter?
A 30-page paper can fit inside a modern context window. Carrying the entire paper into every later engineering session, alongside code, prior context and two or three other sources, is a different question. Something has to be dropped. The entire design problem is deciding what, and being able to tell afterwards what was dropped and why.
The second half is retrieval shape. Chunking a summary can produce chunks that retrieve cleanly and answer badly, because the retrievable unit and the meaningful unit stop being the same thing as soon as a claim depends on a condition stated three sections away.
The specification runs as a single pass with an explicit phase order:
A ACQUIRE
B MAP
C EXTRACT_EXTERNAL
D FORM_GROUPS
E CONTRACT_SCAN
F RETRIEVE
G SYNTHESIZE
H INLINE_CHECK
I COMPRESS
J SERIALIZE
Compression is phase I. It runs after extraction, after relation forming, after the contract scan and after the inline check, so that by the time anything gets dropped the system already knows which material is load-bearing.
The precedence chain fixes compression near the bottom on purpose:
INVARIANT > SOURCE > PROVENANCE > CONTRACT
> COVERAGE > RETRIEVAL > COMPRESSION > STYLE
Extraction priority is tiered per source profile. For research papers:
- Tier A - formal problem, architecture, interfaces and state, algorithm and control, material equations, thresholds and constants, experimental boundaries, controls, ablations, failure modes, limitations, implementation appendix.
- Tier B - results, metrics, baselines, datasets and splits, samples and seeds, models, generalization, assumptions.
- Tier C - motivation, related work, framing, generic conclusions.
Tier C is usually cheap to recover later. The material in Tier A often is not.
The budget is declared and soft:
artifact_budget_tokens: 6000
budget_mode: soft
chunk_budget_tokens: 400
Drop order runs Tier-C first, then redundant narrative, then low-value Tier-B enumeration, then contract findings that resolved as compatible and are unused downstream. Against that sits a never-drop list: material equation literals, hard conflicts, unresolved tensions, probable source or table defects, material qualifications, author limitations, and any required UNKNOWN.
When protected material alone exceeds the target, the artifact reports budget_status: overflow_required rather than trimming into the protected set. The budget loses.
The 400-token chunk figure is the retrieval half, and it only works alongside a rule about what a claim is allowed to be:
- One claim carries one independently retrievable proposition.
- Equations, thresholds, algorithm steps, table values, experimental conditions, conflicting forms and identifier bindings all get split rather than bundled.
- A claim never combines a source statement with an interpretation and a recommendation.
That last constraint is what makes retrieval behave. A chunk containing all three is not independently auditable as a single proposition, because a retriever pulling it cannot tell the reader which part came from the paper. Split them and a downstream validator can reject the interpretation while the source statement survives.
Locators carry the rest, in priority order: timestamp, exact heading, equation, table, appendix, paragraph. A compressed claim that cannot be reopened at its source location is not compressed, it is discarded with extra steps.
What the output looks like
A conventional analysis of the transcriptomics benchmark we use most often would tell you that it compares transcriptomics foundation models against simpler approaches like PCA and scVI, and finds the simpler methods surprisingly strong for perturbation analysis. 1 That is the useful headline, and it is the kind of result that changes which baseline you implement first.
The extractor keeps material at another level:
C024 [EXTERNAL]
location: Section 3.4
k = floor(sqrt(n))
n = number of samples in the reference set
C054 [EXTERNAL]
location: Appendix C.4
k = 251
C097 [DERIVED]
from: C024 + C054
The source contains two unreconciled
neighbor-count specifications.
C024 and C054 stay attributable to the paper. C097 is ours - a relation the extractor produced from two source observations, which the authors never wrote and never claimed.
A later validator might well decide C097 is overstated or misbound or explained somewhere we did not look, and when that happens the relation has to be revisable without touching the two observations underneath it, because otherwise a bad derivation quietly takes the source record down with it.
This is also what makes the artifact reusable across sessions rather than merely storable. A later session can retrieve C024 and C054 without inheriting C097 as though the paper had stated the relation, and it can retrieve C097 while still being able to see that the relation was assembled rather than reported.
A source-bound claim also has to stay reopenable at its locator. The ALCE work separates fluency, correctness and citation quality when evaluating generated text with citations 5, and while our locator contract is narrower - it only asks whether the specific location attached to a bound claim actually supports that claim - the underlying separation between text that reads as supported and text that is supported is the same one.
The remaining states exist because absent information is very easy to turn into false completion:
UNKNOWN - the available evidence did not justify resolving a relation.not_run - a defined check never executed. Materially different from a check that ran and returned nothing, and conflating the two is exactly how a downstream system ends up assuming a clean bill of health that nobody ever issued.- Validation state - records what kind of review happened, rather than letting every second look collapse into a generic
verified.
All of this structure has a cost we cannot argue away.
The format-restriction study found performance degradation under structured-output constraints, with stricter formats generally producing larger reductions, including tested conditions where parsing failure itself was close to zero 6. That is an inconvenient result for a system whose entire output is heavily structured. We have no basis for assuming our system is exempt from that cost.
Schema compliance tells you nothing about whether the evidence inside the schema is any good. We keep structure only where it leaves something operational downstream. Where none of that matters, a conventional summary is probably both cheaper and better, and we use one.
Then we started checking whether any of it was faithful
For a long time the evidence that this worked was that we kept using the artifacts. That is weaker than it sounds, because an artifact only has to be useful to survive; it does not have to be accurate.
So we picked one source and checked the artifact back against the canonical paper.
The paper was Benchmarking Transcriptomics Foundation Models for Perturbation Analysis: one PCA still rules them all 1, a 2024 benchmark comparing pretrained transcriptomics foundation models with classical approaches such as PCA and scVI across biologically motivated perturbation-analysis tasks. Its headline result is relatively easy to summarize: the simpler baselines remain surprisingly strong.
That headline was not why we chose it.
It was a useful stress test because it combines:
- mathematical notation where a missing symbol can change implementation meaning
- dense result tables spanning multiple models, processing variants and metrics
- domain-specific qualifications that are easy to flatten during summarization
- cross-section dependencies where related details appear in different sections or appendices
Some of the useful relations are therefore not written together anywhere; they have to be preserved separately and connected later.
Table 7 gives a sense of the density.
The L1000 results alone place multiple model families, four processing variants and several evaluation metrics into one table. Elsewhere, equations and experimental parameters have to be connected back to those results. A conventional summary can get the main conclusion right while discarding exactly the material an engineering session may need later.
We chose the paper not because we expected it to contain errors, but because it was a demanding input for the kind of reusable context the Knowledge Extractor is meant to produce.
We recorded the check against the source as:
Primary source:
arXiv:2410.13956v2
Representation checked:
canonical arXiv HTML
Draft verification date:
21 Aug 2026
Status:
source-visible discrepancies;
not author-confirmed errata
Author notification:
pending before publication
Our rule for these is deliberately narrow: preserve the observable discrepancy, keep it apart from any explanation, and never let a plausible cause get promoted into a source claim.
- Two metric columns, one repeated sequence. In Table 7, the four UCE processing variants report iLISI values of
26.303, 26.744, 26.820, and 25.916. Table 8 reproduces the same sequence, with the same uncertainties, under a different metric: Top-5 kNN accuracy 1. The repetition is source-visible; its cause is not established. What matters for the extractor is that the correspondence survives compression long enough to be inspected.
- A p-value direction worth checking. The Perturbation Consistency section asks whether observed similarity is significantly higher than chance, while the printed expression counts null similarities less than or equal to the observed value and associates significance with
p < 0.05 1. We preserve both the prose and the expression rather than infer what the authors intended.
- Two neighbour-count specifications. Section 3.4 defines
k = floor(sqrt(n)), while Appendix C.4 separately reports a fixed 251 neighbours for kNN evaluation 1. The canonical paper does not visibly reconcile the two.
Then, while checking that last case, we broke it ourselves.
The failure that was not a reasoning failure
We reached a confident and completely wrong conclusion about k = floor(sqrt(n)). What mattered was not simply that we were wrong, but where the error had entered the pipeline.
One text representation we were using had lost the radical upstream and presented something equivalent to k = floor(n). The model read that representation correctly, reasoned from it consistently, and produced an internally coherent result.
Rereading could not fix it. The missing symbol was no longer present in the input.
Returning to the canonical rendering revealed a second instance of the same problem: the transformed representation had also dropped the square root from M * sqrt(n_b * g) in a Structural Integrity expression 1. The surrounding equations contain another source-visible movement of a 1/n_b factor, which we preserve for inspection without trying to resolve the derivation here.
The important finding for the Knowledge Extractor was architectural. We had been treating failures like this as cases where "the model missed something" and revising the prompt accordingly. But this error occurred before reasoning began.
A better prompt cannot recover information that never reaches the model.
That changed where we looked for failures. Source access, representation, extraction and relation formation could no longer be treated as one undifferentiated prompting problem.
Notation fidelity eventually becomes implementation fidelity. A missing radical, denominator or symbol binding stops being typography the moment somebody turns the preserved mechanism into code.
Where "the model missed something" stopped being useful
We did not start with a taxonomy.
For a long time, very different failures all got described as "the model missed something." That sent us back to prompt wording even when the actual problem was incomplete source access, a damaged representation, lost material during extraction, or a relation that was never formed.
That label eventually became too coarse to be useful.
SOURCE
->
ACCESS
->
REPRESENTATION
->
EXTRACTION
->
RELATION
->
ARTIFACT
- Access failure - relevant source material never becomes available to the execution, as in the browser workflow above. This is distinct from context utilisation: material may be present yet still used unreliably, as prior work on positional effects shows 2.
- Representation failure - available material changes before reasoning begins. The lost radical is our clearest instance; OCR-driven RAG research shows a broader version of the same problem, where semantic and formatting noise propagate downstream 3.
- Extraction failure - the representation is intact, but material needed later does not survive into the artifact: qualifications disappear, equations are dropped, or scope broadens. Scientific summarisation work has documented this kind of overgeneralisation even under explicit accuracy instructions 4.
- Relation failure - the evidence survives individually, but the system never connects the pieces.
sqrt(n) and 251 can both be present while the artifact still fails to test whether they describe the same experimental parameter.
The distinction matters because each failure points somewhere different.
An access failure is not fixed by asking the model to read more carefully. A representation failure is not fixed by better reasoning over already damaged input. An extraction failure belongs in retention and compression logic. A relation failure requires the system to compare evidence that is locally correct but globally disconnected.
This was one of the points where the Knowledge Extractor stopped behaving like a long instruction prompt and started behaving like a pipeline with diagnosable stages.
Post-artifact controls sit on a separate path:
ARTIFACT
->
VALIDATION
->
REPAIR
->
SERIALIZATION
->
DOWNSTREAM USE
Keeping those paths separate lets us ask two different questions: where knowledge was lost, and what authority the resulting artifact is allowed to carry.
Why the prompt acquired controls, and where they still break
Nothing in the first extractor prompt resembled a governance architecture. Most of the controls in v6.8.2 arrived after a specific failure, usually one that could not be explained by prompt wording alone.
Some of those controls are now explicit. Others are still incomplete. The useful distinction is not whether a safeguard exists in principle, but whether the system can identify it, reproduce the execution around it, and assign the result the right level of authority.
1. Rule identity
v6.8.2 has numbered sections, named states and a final invariant block, but its normative clauses still do not carry stable rule identifiers.
There is nothing like an R-EQ-003 that a regression fixture can bind to directly.
That becomes a practical problem once the specification starts changing. A rule can move, split, or be rewritten without changing its intent, and a later failure then becomes harder to attribute to one stable normative object. We can diff two versions of the document. We cannot yet say that fixture X remained bound to rule Y across three revisions.
We found this gap while checking this article against v6.8.2. That is also why we are describing it rather than quietly adding identifiers first.
The missing addresses are not an implemented feature of the extractor. They are a tooling gap around the specification, and one of the clearer places where prose alone is no longer enough.
2. Runtime provenance
Rule identity answers which constraint we are talking about. It does not tell us exactly what execution produced an artifact.
That requires runtime provenance.
A specification version by itself is not enough to reconstruct a run. Model snapshot, reasoning-effort setting and other execution parameters also matter, and some of our older comparative runs did not record all of them. Those runs remain useful as observations, but they cannot be reproduced exactly.
v6.8.2 exposes another unfinished part of that provenance directly:
master_spec_hash: "UNKNOWN"
The value is accurate. It also means the current version label identifies the specification more strongly than it identifies the exact bytes executed in a particular run.
This is where DSPy and PDL became useful reference points for us.
- DSPy models LM pipelines through declarative modules optimized against metrics 7.
- PDL treats prompt construction as a declarative, data-oriented programming problem 8.
The Knowledge Extractor is neither system. What mattered was the architectural implication: once a prompt has versions, budgets, execution parameters and measurable failure conditions, the prompt text alone no longer describes the execution.
3. Validation authority
Even a perfectly identified rule and a reproducible execution do not answer the next question: what is that execution allowed to claim?
v6.8.2 separates five checking layers:
L0 deterministic structural/exact checks
L1 same-pass self-check
L2 independent same-model replay
L3 different-model/runtime semantic audit
L4 executable/domain validator
These layers do not authorize the same state.
v6.8.2 executes only L1 semantic checking unless an external validator actually runs, and L1 MUST_NOT claim independent verification is a rule rather than a caveat. The final invariant block states the same boundary as:
L1 != INDEPENDENT_VERIFICATION
There is empirical reason to keep that distinction.
- Work on intrinsic self-correction found that models attempting to correct reasoning without external feedback often failed to improve and sometimes degraded 9.
- A separate line of work separated mistake finding from mistake correction and found that supplying ground-truth error-location information improved correction across the five reasoning tasks studied 10.
Those results describe different correction conditions. Another pass over the same evidence is not equivalent to receiving new evidence.
In practice, a repair attempt becomes materially different only when the next execution receives something the previous one did not have: a different source representation, an independent signal, a localized error, or an executable counterexample.
The lost-radical case made this concrete. Rereading the damaged representation could not recover a symbol that had disappeared before inference began.
The same authority discipline is why the specification keeps NOT_RUN, RUN_NONE and UNKNOWN distinct. Absence, execution-with-no-finding, and unresolved evidence are not interchangeable states.
What the three controls add up to
The three control questions are therefore different:
Rule identity
Which rule is this?
Runtime provenance
Which execution produced this artifact?
Validation authority
What may that execution legitimately claim?
A failure in any one of them leaves a different kind of ambiguity. Without rule identity, the constraint itself is difficult to track across revisions. Without runtime provenance, the execution cannot be reconstructed precisely. Without validation authority, a result can be given more evidential weight than the check that produced it can support.
Together, they describe something broader than prompt quality: whether a model-mediated result can be identified, reconstructed and assigned only the authority its evidence supports.
Where prompting stops
The Knowledge Extractor no longer assumes that every control belongs inside the prompt.
v6.8.2 explicitly distinguishes deterministic structural and exact checks at L0, semantic checking at L1, replay and cross-runtime audit at L2-L3, and executable or domain validation at L4.
The division is functional. A prompt is useful where the system has to interpret meaning, preserve qualifications, form relations or expose uncertainty. Ordinary software is better suited to checks whose result should not vary across another probabilistic pass.
Some of those boundaries are straightforward:
- budget arithmetic can be computed
- schema conformance can be checked
- locators can be resolved against a fixed source
- claim IDs and references can be traversed as a graph
- deterministic invariants can fail closed
The prompt carries the semantic contract; deterministic components enforce the parts of that contract that can be made exact.
This is also how we now separate four kinds of engineering around the system:
Prompt engineering
defines model behaviour
Context engineering
determines available information
Agent engineering
determines available actions
Verification engineering
determines what result may authorize action
The Knowledge Extractor began mostly in the first category. Source acquisition and representation pushed it into context engineering. L0 and the validation hierarchy introduced verification engineering.
Prompt engineering remains useful where behaviour is semantic. Verification engineering begins where a result can be checked or enforced exactly.
So why not just say "analyze this paper"?
For ordinary reading we still do. If somebody wants to understand a paper quickly and can return to the source when a detail matters, a conventional analysis is usually the better tool.
The difference begins when the answer has to survive the reading session.
A simple analysis is solving a reading problem. The Knowledge Extractor is solving a reuse problem.
The extra structure only earns its cost if it remains useful downstream:
- a source claim can be reopened at its locator
- a derived relation can be revised without rewriting the observations beneath it
- an unresolved state can survive retrieval without being silently completed
- an equation can return to a later coding session as an equation, not as somebody's memory of what the paper probably said
That is the practical distinction.
The Knowledge Extractor is meant to preserve the parts of a source that still matter after the reading session ends, when the work has moved on to design, implementation, testing or verification.
If any part of this work helps your own research or code development, even a little, then sharing it was worth it.
💾 Full specification: Flamehaven Knowledge Extractor v6.8.2
References
1 Ihab Bendidi, Shawn Whitfield, Kian Kenyon-Dean, Hanene Ben Yedder, Yassir El Mesbahi, Emmanuel Noutahi, and Alisandra K. Denton. 2024. Benchmarking Transcriptomics Foundation Models for Perturbation Analysis: one PCA still rules them all. arXiv:2410.13956v2. Revised 4 November 2024.
2 Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the Middle: How Language Models Use Long Contexts. *Transactions of the Association for Computational Linguistics*, 12:157-173. DOI: 10.1162/tacl_a_00638. arXiv:2307.03172.
3 Junyuan Zhang, Qintong Zhang, Bin Wang, Linke Ouyang, Zichen Wen, Ying Li, Ka-Ho Chow, Conghui He, and Wentao Zhang. 2025. OCR Hinders RAG: Evaluating the Cascading Impact of OCR on Retrieval-Augmented Generation. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV 2025), pp. 17443-17453. arXiv:2412.02592.
4 Uwe Peters and Benjamin Chin-Yee. 2025. Generalization bias in large language model summarization of scientific research. Royal Society Open Science, 12(4):241776. DOI: 10.1098/rsos.241776. arXiv:2504.00025.
5 Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. 2023. Enabling Large Language Models to Generate Text with Citations. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 6465-6488. DOI: 10.18653/v1/2023.emnlp-main.398. arXiv:2305.14627.
6 Zhi Rui Tam, Cheng-Kuang Wu, Yi-Lin Tsai, Chieh-Yen Lin, Hung-yi Lee, and Yun-Nung Chen. 2024. Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, pp. 1218-1236. DOI: 10.18653/v1/2024.emnlp-industry.91. arXiv:2408.02442.
7 Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts. 2024. DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines. arXiv:2310.03714. Presented at ICLR 2024, whose venue record uses the title variant DSPy: Compiling Declarative Language Model Calls into State-of-the-Art Pipelines.
8 Mandana Vaziri, Louis Mandel, Claudio Spiess, and Martin Hirzel. 2024. PDL: A Declarative Prompt Programming Language. arXiv:2410.19135.
9 Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. 2024. Large Language Models Cannot Self-Correct Reasoning Yet. *The Twelfth International Conference on Learning Representations (ICLR 2024)*. arXiv:2310.01798.
10 Gladys Tyen, Hassan Mansoor, Victor Cărbune, Peter Chen, and Tony Mak. 2024. LLMs cannot find reasoning errors, but can correct them given the error location. Findings of the Association for Computational Linguistics: ACL 2024, pp. 13894-13908. DOI: 10.18653/v1/2024.findings-acl.826. arXiv:2311.08516.
Flamehaven Lab Notes documents the research, implementations, failed assumptions, audits and engineering decisions that emerge while we build.