elad-granot (AI researcher at Trullion, an AI finance platform for accountants and auditors) explains when to train small language models instead of calling a big LLM. Use case: auditors receive one PDF bundling invoices, bank transfers, contracts, and inventory lists. It must be segmented into sub-documents and each one classified. Forcing a generative LLM to do segmentation (chunk IDs, “from ID x to y”) is awkward; a fine-tuned segmentation model is natural. Results versus Gemini 2.5 Flash: ~230× faster (before batching) and F1 rising from ~80% to over 90%, because the loss targets the task. Training data was synthetic but realistic: real public filings and invoices stitched together, giving labels for free. Costs: datasets, tuning, training instabilities, serving (GPU servers, batching), and cost that only wins above some volume. Distillation from big models helps, but check the model license. Bigger picture: LLM orchestrators with SLM specialists, privacy via on-device models, and tiny recursive models beating big ones on narrow benchmarks.
Show: langtalks
Concepts: small-language-models · model-selection · decision-classifiers · llm-pipelines