Open-Weight or Hosted Models: How to Choose

Every AI project reaches a fork. Call a model through a provider’s API, or download a model and run it yourself. The first is fast and simple. The second offers control, and brings costs and responsibilities of its own.

The terms in this debate are used loosely, which causes confusion, especially around the word “open”. This guide explains the options, the trade-offs and a practical way to decide, for a small team building its first feature and an enterprise setting platform policy.

The terms

• Hosted or proprietary models: the provider keeps the model weights private and offers access through an interface, billed by usage or subscription. You send data to the provider.
• Open-weight models: the trained parameters are published for download, under a licence that sets terms of use. You can run them on your own servers, in your cloud account or through a host. Training code and training data may not be published.
• Open-source AI: the Open Source Initiative’s Open Source AI Definition requires four freedoms to use, study, modify and share the system, and access to the preferred form for modification: enough information about training data, the complete code to train and run it and the parameters. Many open-weight models do not meet that definition, since they omit data information or code, or restrict uses in their licences.

So “open” can mean open weights with a custom licence, or fully open source in the stricter sense. Read the licence for each model.

What changes when you choose open weights

Control and privacy

You decide where the model runs and what data it sees. Data need not leave your environment, which helps with sensitive information, regulation and customer demands. You also take on responsibility for securing the system. Our guide to data residency and sovereignty explains related requirements.

Customisation

You can fine-tune open-weight models on your data, modify them and optimise them for your hardware, subject to licence terms. Our article on RAG, fine-tuning or prompting explains customisation options. Hosted providers may also offer fine-tuning, within their rules.

Cost structure

Hosted models charge per use. Open-weight models charge you for compute, which is mostly fixed capacity: GPUs or other accelerators, memory, storage, networking and the people who run them. At low or irregular volume, hosted is usually cheaper. At high, steady volume, self-hosting can cost less per request if you keep hardware busy. Include engineering time, monitoring, upgrades and idle capacity. Our guides to measuring AI cost and GPU infrastructure for AI help build the model.

Quality and capability

Leading hosted models often lead on the hardest tasks, such as complex reasoning and broad knowledge, though open-weight models are competitive on many tasks, especially after tuning for a narrow domain. Quality differs by task, language and context length. Test before you assume.

Reliability and operations

Hosted services provide scaling, uptime commitments and updates. Self-hosting means you manage capacity, latency, failover, upgrades and security patches. Models change, so you must plan updates and regression testing.

Speed to start

Hosted models take minutes to start. Self-hosting takes longer to set up, though managed services now host open-weight models for you, which gives a middle route: open weights, with someone else operating the infrastructure.

Vendor dependence

Hosted providers can change prices, models, terms and availability, and may retire versions. Open-weight models let you keep running a version you have validated, though you must maintain it.

Safety and compliance

Hosted providers apply safety systems and content filters, and publish policies and compliance documents. With open weights you decide on safeguards, and you own the responsibility to implement them. Our guides to prompt injection and AI agent governance describe controls you will need either way.

A decision framework

Score each factor for your use case.

1. Data sensitivity and rules: can data be sent to a provider under acceptable terms? If not, lean towards self-hosting or a private deployment.
2. Quality requirement: does the task need the strongest available model? Test candidates on your evaluation set. See our guide to evaluating LLM output.
3. Volume and pattern: steady high volume favours self-hosting. Low or spiky volume favours hosted.
4. Latency and location: do you need low latency near users or devices, or offline operation? Smaller open-weight models can run at the edge.
5. Customisation needs: deep tuning or architectural changes favour open weights.
6. Team skills: do you have people to run and secure model infrastructure?
7. Time to market: hosted wins when speed matters.
8. Licence terms: check commercial use, user limits, restrictions on uses and obligations on outputs.
9. Risk and continuity: consider supplier stability, model deprecation and the cost of switching.
10. Sovereignty and policy: national, sector or customer requirements for domestic or controlled infrastructure.

Practical patterns

• Hosted first, then reassess: start with a hosted model to prove value quickly, then move high-volume or sensitive workloads to open-weight models when justified.
• Hybrid routing: send simple tasks to a small, cheap model and hard ones to a stronger model, whether hosted or self-hosted.
• Private deployment of hosted models: some providers offer dedicated instances or deployment in your cloud account with stronger isolation, a middle path.
• Open weights via managed hosting: use a third party that runs open-weight models and sells access, to gain flexibility without running GPUs.
• Edge and on-device: small open-weight models for offline or privacy-sensitive features.
• Abstraction layer: build against a common interface so you can swap models. Keep prompts, evaluation sets and guardrails portable.

What to test

• Quality on your tasks, with your data, languages and formats.
• Latency and throughput under load. See our guide to performance testing.
• Cost per task at expected volumes.
• Behaviour on safety and policy tests, including injection attempts.
• Robustness to long inputs and edge cases.
• Operational factors: deployment effort, monitoring and upgrade path.

Repeat the tests as models change, since new versions appear often.

Governance

• Maintain an inventory of models in use, with versions, licences and owners.
• Define approval criteria for models and providers.
• Review licences and data processing terms with legal advice.
• Monitor for model updates, vulnerabilities and deprecations.
• Document intended use and limits.
• Apply security controls for model files, such as scanning downloaded artefacts and verifying sources, since malicious or tampered models are a risk. Our guide to software supply chain security applies here too.

Common mistakes

• Treating “open” as meaning free of restrictions.
• Choosing by benchmark headlines without testing on your tasks.
• Underestimating the cost and effort of self-hosting.
• Sending sensitive data to a provider without checking terms.
• Locking prompts and logic to one provider’s quirks.
• Ignoring licence terms.
• Forgetting regression tests when models update.

A starting plan

1. Define the use case, data rules and quality bar.
2. Build a small evaluation set.
3. Test one or two hosted models and one or two open-weight models.
4. Estimate total cost at expected volume.
5. Choose, design for portability and monitor.
6. Review every six months.

How SoftSapien supports model decisions

SoftSapien evaluates hosted and open-weight options on your own tasks, builds the integration and infrastructure and operates it with security and cost control. See our AI development and cloud and datacenter solutions services.

To compare options for your use case, visit the contact page and chat with our team, send a message or schedule a free consultation call.