You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+52-44Lines changed: 52 additions & 44 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -15,7 +15,7 @@ A terminal chat client for running **local LLMs on consumer hardware**. Chat wit
15
15
16
16
Perfect for developers who want a **self-hosted ChatGPT alternative** running on their gaming PC or homelab server. Also useful for local AI coding assistants, agentic workflows, and agent development.
17
17
18
-
> Named after ZORAC, the intelligent Ganymean computer from James P. Hogan's *The Gentle Giants of Ganymede*.
18
+
> Named after ZORAC, the intelligent Ganymean computer from James P. Hogan's _The Gentle Giants of Ganymede_.
19
19
20
20
## Install
21
21
@@ -68,7 +68,7 @@ That's it. Your conversation is automatically saved and restored between session
68
68
69
69
### Rich Terminal UI with Live Streaming
70
70
71
-
*Interactive chat with real-time streaming responses, markdown rendering, and performance metrics*
71
+
_Interactive chat with real-time streaming responses, markdown rendering, and performance metrics_
@@ -180,7 +180,7 @@ Source users can also use a `.env` file in the project root. See [Configuration
180
180
181
181
Zorac tracks tokens to stay within your model's context window:
182
182
183
-
```
183
+
```text
184
184
You: /tokens
185
185
📊 Token usage:
186
186
Current: ~3421 tokens
@@ -204,14 +204,14 @@ When the conversation exceeds `MAX_INPUT_TOKENS`, Zorac automatically summarizes
204
204
205
205
Runs on **consumer gaming GPUs**:
206
206
207
-
| GPU | VRAM | Model Size | Performance |
208
-
|-----|------|------------|-------------|
209
-
| RTX 4090 | 24GB | Up to 24B (AWQ) | 60-65 tok/s |
207
+
| GPU | VRAM | Model Size| Performance |
208
+
|-----------|----|---------------|----------- |
209
+
| RTX 4090 | 24GB | Up to 24B (AWQ) | 60-65 tok/s |
210
210
| RTX 3090 Ti | 24GB | Up to 24B (AWQ) | 55-60 tok/s |
211
-
| RTX 3090 | 24GB | Up to 24B (AWQ) | 55-60 tok/s |
212
-
| RTX 4080 | 16GB | Up to 14B (AWQ) | 45-50 tok/s |
213
-
| RTX 4070 Ti | 12GB | Up to 7B (AWQ) | 40-45 tok/s |
214
-
| RTX 3080 | 10GB | Up to 7B (AWQ) | 35-40 tok/s |
211
+
| RTX 3090 | 24GB | Up to 24B (AWQ) | 55-60 tok/s |
212
+
| RTX 4080 | 16GB | Up to 14B (AWQ) | 45-50 tok/s |
213
+
| RTX 4070 Ti | 12GB | Up to 7B (AWQ) | 40-45 tok/s |
214
+
| RTX 3080 | 10GB | Up to 7B (AWQ)| 35-40 tok/s |
215
215
216
216
See [Server Setup](https://zorac.lowgravitylab.com/SERVER_SETUP/) for optimization details.
217
217
@@ -240,48 +240,56 @@ You can use any vLLM-compatible model (Llama, Qwen, Phi, DeepSeek, etc.) by chan
240
240
<summary><b>Can I run this without a GPU?</b></summary>
241
241
242
242
No, this requires an NVIDIA GPU with at least 10GB VRAM. CPU-only inference is too slow for interactive chat.
243
+
243
244
</details>
244
245
245
246
<details>
246
247
<summary><b>How does this compare to Ollama?</b></summary>
247
248
248
249
Zorac uses vLLM for faster inference (60+ tok/s vs Ollama's 20-30 tok/s on the same hardware) and supports advanced features like tool calling for agentic workflows. Ollama is easier to set up but slower.
250
+
249
251
</details>
250
252
251
253
<details>
252
254
<summary><b>Do I need to be online?</b></summary>
253
255
254
256
Only for the initial model download (~14GB for Mistral-24B-AWQ). After that, everything runs completely offline.
257
+
255
258
</details>
256
259
257
260
<details>
258
261
<summary><b>Is this legal? Can I use this commercially?</b></summary>
259
262
260
263
Yes. Mistral-Small is Apache 2.0 licensed (free commercial use). vLLM is also Apache 2.0.
264
+
261
265
</details>
262
266
263
267
<details>
264
268
<summary><b>What about AMD GPUs or Mac M-series?</b></summary>
265
269
266
270
This is specifically for NVIDIA GPUs using CUDA. For AMD, you'd need ROCm support (experimental). For Mac M-series, check out MLX or llama.cpp instead.
271
+
267
272
</details>
268
273
269
274
<details>
270
275
<summary><b>How much does it cost to run?</b></summary>
271
276
272
277
Electricity for an RTX 4090 at ~300W is roughly $0.05-0.10 per hour. Far cheaper than API costs for heavy usage.
278
+
273
279
</details>
274
280
275
281
<details>
276
282
<summary><b>How do I copy text from the chat?</b></summary>
277
283
278
284
Zorac uses mouse reporting for scrolling, which can interfere with native text selection in some terminals. In **iTerm2**, hold **Option (⌥)** while clicking and dragging to select text, then copy with **Cmd+C** as usual. Most terminals support a similar modifier key — check your terminal's documentation for its equivalent.
285
+
279
286
</details>
280
287
281
288
<details>
282
289
<summary><b>What other models can I run?</b></summary>
283
290
284
291
Any model with vLLM support: Llama, Qwen, Phi, DeepSeek, etc. Just change the `VLLM_MODEL` setting. See [vLLM supported models](https://docs.vllm.ai/en/stable/models/supported_models/).
0 commit comments