Why I built the lab
I wanted to understand the infrastructure behind an LLM by running it myself and connecting it to software I use. My background is in production reliability. A local lab gives me a place to learn the inference workload directly: how a runtime is configured, what the hardware can accommodate, and what changes when another application depends on the model.
What I actually run
I run Qwen-family models on an NVIDIA RTX 4060 and a 30B-class model on Apple Silicon. My work uses llama.cpp, Open WebUI, and local API-compatible endpoints. It includes runtime configuration and integration with coding agents and automation workflows.
My experiments also include Jan for local model use and management, OpenCode and other coding agents, and MCP-compatible integrations. I am exploring how model selection, runtime settings, and tool configuration fit together. These integrations are part of the experimentation, rather than a finished automation platform.
The lab is an evolving collection of local setups. I use it to work through runtime configuration and integration one step at a time.
From a model to a usable system
There are different questions at each boundary. The hardware has to accommodate the chosen configuration. The runtime has to load and serve the model. A client needs an endpoint it can call and a response it can use. Open WebUI provides an interaction layer; API access lets me work with the model from other tools.
Keeping those boundaries separate helps me reason about integration. If an application cannot use a local model, I need to distinguish a client configuration problem from a runtime or resource problem before changing the model itself. That is the connection to my existing reliability work: follow the request through the system instead of treating every symptom as the same failure.
The diagram above is a conceptual map of these layers. Its evaluation stage describes the direction of the lab, not a completed benchmark harness.
Choices and trade-offs
I am exploring quantization, model/runtime configuration, and memory and compute constraints. Working across Apple Silicon and NVIDIA hardware makes the configuration important: the model name alone is not enough to describe the experiment.
The useful record for a comparison would include the exact model artifact, quantization, runtime version, hardware, context settings, input, and generation limits. Without that record, I cannot separate a configuration change from a meaningful difference in behavior.
My focus is on making an integration understandable before trying to optimize it. A model being available through an API is useful progress, but it leaves questions about request limits, errors, cancellation, and concurrent use to investigate.
Learning the workload
Alongside runtime integration, I’m developing my understanding of Python, NumPy, PyTorch, and model inference fundamentals. I want to connect what I observe at the endpoint to the computation and resource choices underneath it. This learning runs alongside my systems work.
Model benchmarking and comparison are part of my experimentation. I want to understand which differences come from the model and which come from its configuration.
Next steps
Next, I want to make the serving and reliability experiments more controlled: investigate vLLM, first-token and token-delivery latency, queueing, resource saturation, and failure/recovery behavior.
I plan to record the configuration and workload for each run so I can repeat a comparison and understand what changed.
What I am learning
The lab connects model execution to the same concerns I encounter in other systems: configuration, dependencies, resource limits, and failure behavior. I am building the workload-specific knowledge through local experimentation while bringing an experienced operator’s questions to it.