For much of the generative-AI boom, running a large language model locally had a distinctly experimental feel. It was something developers did on a powerful gaming PC, a Mac with enough unified memory or a workstation assembled specifically to see how much intelligence could be squeezed onto hardware sitting under a desk. Cloud APIs, meanwhile, were where the serious models lived. If a company wanted state-of-the-art AI, the assumption was that its data would travel to somebody else’s infrastructure.
That division is beginning to look outdated. Local and self-hosted models are not replacing cloud AI, and the largest frontier systems still require infrastructure far beyond what most organizations would operate themselves. But a growing class of smaller open models is becoming capable enough that enterprises can make a genuine architectural choice: send every task to a remote model provider, or run at least part of the AI stack on infrastructure they control.
IBM’s new Granite 4.2 family is a useful signal of how far that transition has progressed. Released this week in 3B, 8B and 30B parameter variants, the open models are designed around reasoning, tool use, coding and agentic workflows rather than merely lightweight text generation. IBM explicitly supports deployment across cloud, on-premises and edge environments, while the models are already available through local-model tooling including Ollama. The significance is less that IBM has released another LLM than that the feature set once associated primarily with large hosted models is moving into models organizations can download and operate themselves.
Local models are growing out of the hobbyist phase
The first generation of local LLM experimentation came with obvious compromises. Smaller models were useful for autocomplete, simple summarization and casual chat, but complex reasoning or reliable coding often sent users back to a frontier cloud API. Running larger models required substantial memory, inference could be slow and configuring the software stack was its own project.
That experience has changed quickly. Tools such as Ollama and LM Studio have reduced much of the deployment friction, while quantization and more efficient inference engines have made surprisingly capable models practical on commodity workstations. At the same time, model developers have invested heavily in improving performance at smaller parameter counts. The question is increasingly not whether a local model can produce coherent output, but whether it is good enough for a specific production workload.
Granite 4.2 illustrates that shift. IBM’s 3B model is explicitly positioned for resource-constrained and edge deployments, while the 8B model targets general enterprise applications and the 30B variant is intended for more demanding reasoning and coding. All three support a 128,000-token context window, and IBM has added native reasoning modes and tool calling. The 8B and 30B models also receive specialized reinforcement learning for agentic tasks such as software engineering, terminal work and search-driven workflows.
These are not features aimed primarily at someone running a chatbot for fun on a laptop. They are capabilities designed for software systems that need to retrieve information, choose tools, execute steps and participate in business workflows.
The enterprise argument is control
The strongest case for local AI has never been that every self-hosted model is smarter than the best cloud model. It is that intelligence is only one variable in a production system. Enterprises also care about where data goes, how predictable costs are, what happens during a network outage, which versions can be deployed, how systems are audited and whether a critical workflow depends on a vendor changing an API or pricing model.
A cloud API is extraordinarily convenient because much of the infrastructure problem disappears. The provider handles accelerators, scaling, model upgrades and availability. For many workloads, that trade remains compelling. But the convenience comes with architectural dependencies. Requests leave the organization’s environment, inference costs scale with usage, latency depends partly on network conditions and the organization operates within the capabilities and policies exposed by the provider.
Self-hosting reverses that trade. The organization assumes responsibility for hardware, inference software, updates, monitoring and capacity planning, but gains much more control over the complete execution environment. Sensitive prompts and retrieved documents can remain inside company infrastructure. A specific model version can be pinned. Access controls can be integrated with existing systems. Inference capacity can be purchased as infrastructure rather than consumed indefinitely as metered tokens.
That matters particularly for workloads that are repetitive, high-volume and sensitive. An internal document classifier does not necessarily need the most powerful reasoning model on the market. Neither does every retrieval task, code-analysis job, customer-service routing decision or structured-data extraction pipeline. If an 8B or 30B model performs the required task reliably, the value of sending every request to a frontier model becomes less obvious.
Agents make local deployment more interesting
The agentic capabilities in Granite 4.2 are especially important because AI agents change the privacy and infrastructure equation. A chatbot might receive a document and return an answer. An agent may interact continuously with source-code repositories, internal databases, terminals, ticketing systems and business applications. The amount of proprietary context exposed to the model can be dramatically larger.
Running the model inside the same controlled environment as those tools can simplify the trust boundary. It does not automatically make an AI agent safe; a local agent can still make bad decisions, misuse permissions or damage systems. But organizations gain the option to keep inference and data movement within infrastructure they govern rather than making a remote model endpoint part of every agentic transaction.
This also changes latency. Agent workflows frequently involve many model calls rather than one. A system might reason about a task, call a tool, inspect the result, revise its plan and call another tool several times before finishing. Network latency that is trivial in a single chat response can become more noticeable when multiplied across a long chain of operations. Local inference can reduce that round-trip cost when the model and the tools live close to one another.
IBM’s positioning reflects this emerging architecture. Granite 4.2 is designed to reason about which tools to use and in what sequence, and its models can be deployed across cloud, on-premises and edge environments. The 3B, 8B and 30B sizes also allow companies to match model capacity to different jobs instead of treating one giant model as the universal endpoint.
The economics change when AI becomes infrastructure
Cloud inference is easy to understand when AI usage is occasional. Paying per token avoids buying expensive hardware that might sit idle. But as AI moves from a feature used by employees a few times per day to infrastructure embedded continuously inside software, the calculation changes.
An agent monitoring code, processing documents or executing internal workflows may generate enormous volumes of inference. At sufficient utilization, owning or reserving compute can become attractive because the marginal cost of another local request behaves differently from another metered API call. The exact break-even point varies dramatically with model size, hardware, electricity, utilization, staffing and cloud pricing, so claims that local inference is universally cheaper should be treated skeptically.
Predictability, however, has value of its own. A company can provision a known amount of GPU capacity for a known workload and establish a relatively understandable operating envelope. That is different from launching an AI feature whose successful adoption directly increases an external inference bill. For finance and infrastructure teams, predictable capacity can be nearly as important as the lowest theoretical cost per token.
Local models can also support routing strategies rather than an all-or-nothing migration. A smaller private model can handle routine work while difficult requests escalate to a more powerful cloud model. Sensitive data can remain local while generic reasoning is sent elsewhere. High-volume background tasks can run on owned infrastructure, with frontier APIs reserved for cases where their additional capability justifies the expense.
Open weights change the relationship with the model
Granite 4.2 is released under the Apache 2.0 license, allowing organizations to download, customize and deploy the models commercially. That matters because self-hosting is not only about where inference runs. It changes the relationship between the organization and the model itself.
With a hosted proprietary system, customization typically happens through prompts, retrieval, fine-tuning options exposed by the vendor and whatever controls the API provides. An open model can be integrated more deeply. Companies can fine-tune it for domain-specific behavior, optimize inference for particular hardware, inspect the deployment stack and decide when or whether to adopt a newer version.
That flexibility is particularly attractive in regulated environments or long-lived enterprise systems, where uncontrolled model changes can create validation problems. Predictability can be a feature. The newest model is not automatically the best model if every upgrade requires retesting a critical workflow.
IBM has leaned into this enterprise framing with Granite more broadly, emphasizing transparency, commercial licensing and deployment flexibility. Granite 4.2 extends the proposition into reasoning and agents, suggesting that open enterprise models no longer need to remain confined to simple back-office NLP tasks.
Local does not mean laptop
One reason the local-AI discussion can become confused is that “local” describes several very different deployment models. It can mean a developer running a 3B model on a notebook computer through Ollama. It can also mean an enterprise serving a 30B model from a rack of GPUs inside its own data center, or a managed private environment in which the organization controls the infrastructure and data boundary.
For enterprises, the latter scenarios are likely to matter more. The goal is not necessarily to make every employee’s laptop an AI server. It is to move inference closer to the organization’s data and control plane. On-premises clusters, private clouds, edge servers and individual workstations exist along the same continuum: AI does not have to live exclusively behind somebody else’s public API.
This is also why tools that began with developers matter strategically. Ollama’s simple local-model workflow makes experimentation cheap. A team can download a model, test it against real tasks and discover where it is sufficient before committing to a larger deployment architecture. IBM lists Ollama among the supported ways to access Granite 4.2, alongside platforms ranging from Hugging Face and GitHub to hosted inference providers. The same model family can therefore move between a laptop experiment and enterprise infrastructure without forcing the application concept to be rebuilt around a completely different model.
The cloud is not disappearing
None of this means local models are about to make frontier cloud AI obsolete. The most capable hosted systems continue to offer reasoning, multimodality and generalization that smaller models may not match. Cloud providers also deliver operational simplicity and enormous elastic capacity that many organizations have no desire to reproduce internally.
The more plausible future is heterogeneous. Enterprises will use multiple models at multiple levels of capability, running in multiple places. Some workloads will demand the best frontier model available. Others will prioritize privacy, latency or cost. Some agents will operate locally and escalate difficult reasoning to cloud systems. Edge models will handle tasks where connectivity is unavailable or undesirable. The architecture will look less like choosing one AI provider and more like designing a compute stack.
That is the larger significance of releases such as Granite 4.2. A 3B or 8B model with reasoning and tool use does not need to beat every frontier benchmark to matter. It only needs to cross the threshold where an organization can trust it with a useful class of work. Once enough smaller models cross that threshold, the assumption that serious AI must always be rented remotely begins to weaken.
Local AI started as an enthusiast story because enthusiasts were willing to tolerate the rough edges. Now the tooling is easier, the models are stronger and the enterprise incentives are becoming clearer. Privacy, predictable deployment, customization and control are not niche preferences. They are standard infrastructure concerns.
The next phase of AI may therefore be less centralized than the first one suggested. Cloud models will remain essential, but they will increasingly share the stack with models running inside companies, on private servers and at the edge. The interesting question is no longer whether local LLMs can run. It is how much of the work organizations once assumed had to go to the cloud can quietly move back home.