Laboratory development programme · Stage 1
What can AI take on in your work?
Independent applied AI research laboratory
AI can write a convincing summary while missing a required condition or hiding a contradiction. Before deployment, find out where that result is acceptable and where an error creates risk. What can AI take on, and what must a person check?
The first stage of the laboratory development programme is complete: a study using synthetic tasks in a local environment.
Where the laboratory could help
№1
Approved yesterday. Changed today.
An engineer approves a solution. Later, one parameter in a source document changes.
What we test →№2
The cheapest proposal was incomplete.
One supplier includes installation and testing. Another quotes equipment only. A third substitutes alternatives for part of the specification.
What we test →Research across the use of AI
Meaning, stability, efficiency, controllability and human responsibility. Below is the completed Stage 1; newer document studies still require independent human review. Agent controllability remains a developing direction.
LydStone Lab websiteThe tasks selected for Stage 1 — and why
Stage 1 of the programme covered seven task groups found in work with requests, documents and operational signals.
These tasks can hide an error behind a plausible answer, while correctness can be checked against predefined facts, rules and constraints.
- W1
Request clarification and dialogue
If a key condition is missing, AI should ask for it rather than silently invent it.
A missing-information clarification scenario was studied.
- W3
Extracting data from documents
The system must find the right fact and distinguish missing information from an invitation to invent it.
The selected corpus was too easy to distinguish configurations.
- W4
Reconciling conflicting sources
Two convincing sources can disagree. AI must notice the conflict and avoid presenting a guess as a supported conclusion.
Recognition of irreducible ambiguity was studied.
- W5
Inference through a chain of rules
Several correct steps do not guarantee a correct conclusion: it must follow from the given facts and rules.
Rule chains with increasing difficulty were studied.
- W6
Cause diagnosis
A plausible explanation of a fault does not establish its cause or justify the next action.
Measurements were collected, but the difficulty ladder did not reliably distinguish configurations.
- W7
Monitoring and signal triage
Transient noise can be mistaken for a persistent problem and direct attention to the wrong place.
Signal discrimination under changing transient noise was studied.
- W9
Writing under constraints
Fluent text can omit a required condition or include something explicitly forbidden.
Compliance with simultaneous required and forbidden conditions was studied.
One synthetic scenario with difficulty levels was tested in each group. Conclusions apply to those scenarios and configurations, not to every task in the group. W3 and W6 do not serve as routing evidence.
Tool use (W2) and retrieval with knowledge-base answers (W8) were separated and deferred: each needs its own evaluation tools and methodology.
The LydStone Lab development programme
The programme aims to make AI selection for a specific process a testable engineering decision, connecting task requirements, evidence of fitness and resource needs.
Stage 1 established the foundation: the CogniScope measurement system, a set of task groups, reproducible experiments and explicit evidence boundaries. The methodology underwent cross-model review by several models from different vendors and was revised in response to the comments; the reservations are preserved. No independent external expert review has been carried out; it remains a step in the programme.
Further development concerns customer-specific examples, broader scenarios and testing under different conditions. Hardware transfer, load and operational reliability each require their own validation; Stage 1 does not establish them.
CogniScope — the programme’s technical core
CogniScope is the AI qualification research system used to build the LydStone Lab evidence base. It connects scenarios, model configurations, reproducible runs, response evaluation, resource measurements and reports traceable to experimental records.
In Stage 1 of the programme, the system was used for local synthetic testing. Further work develops its capabilities; readiness must be established by separate results.
The unit of qualification is configuration × workload × execution profile. Quality is considered alongside latency, VRAM and output tokens.
In the tested scenarios, the same configuration has different sufficiency boundaries. For example, 27B Q4 without reasoning reaches L0 for irreducible ambiguity, L1 for rule-chain inference and L2 for constrained text.
L0–L3 describe difficulty within each scenario. L3+ means the entire tested range passed and no boundary was found within it. It makes no claim about levels outside that range.
Sufficiency depends on the task. L3+ means no boundary was found within the tested range.
Original Stage 1 figure in Russian. The English explanation preserves its results and limitations.
Open full-size figureThree findings from Stage 1 of the programme
W4
97.4% of cases passed. Why did the configuration fail?
For irreducible ambiguity, W4, 27B Q4 without reasoning passed L0: 30 of 30 cases, with no fatal errors. Median latency was 884 ms and output length was 51 tokens.
At L1, 38 of 39 cases passed. However, one error violated the predefined zero-fatal-error allowance. Verdict: FAIL (RED).
The case pass rate does not replace acceptance criteria. A good average cannot compensate for an error the process cannot tolerate.
With stronger reasoning, the same 27B Q4 passed the tested range through L3: 15 of 15 cases at L3, median latency of 12,719 ms and 732 output tokens.
27B Q4 without reasoning: one fatal error at L1 causes a fail under zero allowance.
Original Stage 1 figure in Russian. The English explanation preserves its results and limitations.
Open full-size figureW9
The next capability level has a resource cost
For text with required and forbidden conditions, W9, 27B Q4 without reasoning passed L2: 14/14 cases, 1,482 ms, 58 output tokens and 20,809 MB of VRAM.
27B Q4 with stronger reasoning passed L3: 16/16 cases, 15,540 ms, 839 output tokens and 20,807 MB of VRAM.
Moving between these confirmed levels is associated with approximately 10.5× the latency and 14.5× the output tokens. VRAM remained around 20.8 GB.
Difficulty and reasoning mode both change. This is the cost of moving between confirmed levels, not the isolated effect of reasoning.
Confirmed L2 versus L3. Difficulty and reasoning mode both change; the effect of reasoning is not isolated.
Original Stage 1 figure in Russian. The English explanation preserves its results and limitations.
Open full-size figureW5
An evidence-based refusal to recommend is a result
For rule-chain inference, W5, no tested configuration achieved unconditional PASS (GREEN) at L3.
Two 27B configurations with stronger reasoning were CONDITIONAL (AMBER): Q4 at 21/22 and Q6 at 21/23. FAIL (RED): 9B Q4 with reasoning at 3/4; 27B Q4 without reasoning at 3/6; 9B Q4 without reasoning at 2/6.
This outcome stays in the report. The next step may be a different decomposition, refined criteria or a new test. Insufficient evidence does not justify declaring a winner.
No tested configuration qualifies unconditionally at L3. Conditional and negative outcomes are retained.
Original Stage 1 figure in Russian. The English explanation preserves its results and limitations.
Open full-size figureWhat the evidence establishes and where it stops
Stage 1 of the laboratory development programme demonstrated the method’s basic ability to measure configuration sufficiency for a specific AI task in a synthetic local environment.
How the methodology was reviewed. The method description, the acceptance criteria, and the boundary-search procedure were reviewed by several large language models from different vendors. The models worked from a shared brief, independently of one another and without access to each other's responses. The comments were addressed, part of the methodology was revised, and the review was repeated. Comments and reservations are preserved alongside the results.
What this is not. This is cross-model machine review, not an independent audit. The reviewer carries no professional responsibility for its conclusions, and models can share blind spots — including with the models the laboratory itself studies. We state this directly because scenarios W1–W9 exist for exactly that reason: a plausible model answer is regularly wrong. The laboratory cannot apply a weaker criterion to its own methodology than to its subject of study.
What can be verified independently of the reviewer. The protocol, the experiment conditions, the saved results — including negative ones — and the pinned evidence base cogniscope-stage1b-final 0dac1339. These materials can be re-checked without relying on anyone's conclusions.
The methodology is a subject of research: the methodology is versioned, each conclusion is tied to the method version that produced it, and negative and superseded results are preserved. This study uses the laboratory’s own criteria. Cross-model review does not replace independent external expertise. Applying a conclusion to an operating workflow requires real examples, agreed criteria and human review.
This page provides the method description, individual task results and original Stage 1 figures. Reproducing an experiment also requires the original dataset, configuration and execution procedure; viewing this page does not accomplish that by itself.
The study is limited to one host, Stage 1 scenarios and pilot statistics. The boundary-search procedure requires further validation. Comparing different model generations and reasoning modes does not isolate the effect of model size.
Stage 1 does not establish production safety, load or concurrent-request behaviour, cloud-versus-local economics, hybrid routing or a full review of a customer architecture.
The insufficiently discriminating W3 and W6 results are retained and do not serve as routing evidence. W2 and W8 are deferred.
Frozen evidence from Stage 1 of the programme: cogniscope-stage1b-final0dac1339fbaa2cb51741c102bdfa0ac148936d9a
Proposed service
Test your task on examples
Start with one task: what a colleague receives, what output they need and which errors must not slip through.
We start with the decision that depends on the test. We then record the hypothesis, review how work is divided between AI, deterministic code and people, and agree on configurations, examples and acceptance criteria.
What you provide
- The current process and desired outcome
- Agreed anonymised or synthetic examples
- Data and operating constraints
What you receive
- A qualification report and a map of critical nodes and criteria
- A matrix of tested configurations and confirmed or rejected assumptions
- A resource profile, applicability boundaries and the next validation step
Scope, examples, criteria, schedule and price are agreed after the first conversation and an assessment of input readiness. No fixed duration is promised beforehand.
Architecture ownership and the final decision remain with the client. The sprint is not a safety certification or a substitute for load testing. Cloud-versus-local economics and hybrid routing require separate validation. Testing on real data inside the client perimeter and subsequent operational stages are scoped separately.
Describe the task, the decision you need to make and the error you cannot tolerate. That is enough for an initial message.
Discuss a task to testDmitry Vaneev · LydStone Lab