| llama_model_loader: loaded meta data with 24 key-value pairs and 195 tensors from /home/karanpc/Desktop/Ollama Copy/ultra-fast-llm-pro/model/model.gguf (version GGUF V3 (latest)) | |
| llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. | |
| llama_model_loader: - kv 0: general.architecture str = phi3 | |
| llama_model_loader: - kv 1: general.name str = Phi3 | |
| llama_model_loader: - kv 2: phi3.context_length u32 = 4096 | |
| llama_model_loader: - kv 3: phi3.embedding_length u32 = 3072 | |
| llama_model_loader: - kv 4: phi3.feed_forward_length u32 = 8192 | |
| llama_model_loader: - kv 5: phi3.block_count u32 = 32 | |
| llama_model_loader: - kv 6: phi3.attention.head_count u32 = 32 | |
| llama_model_loader: - kv 7: phi3.attention.head_count_kv u32 = 32 | |
| llama_model_loader: - kv 8: phi3.attention.layer_norm_rms_epsilon f32 = 0.000010 | |
| llama_model_loader: - kv 9: phi3.rope.dimension_count u32 = 96 | |
| llama_model_loader: - kv 10: general.file_type u32 = 15 | |
| llama_model_loader: - kv 11: tokenizer.ggml.model str = llama | |
| llama_model_loader: - kv 12: tokenizer.ggml.pre str = default | |
| llama_model_loader: - kv 13: tokenizer.ggml.tokens arr[str,32064] = ["<unk>", "<s>", "</s>", "<0x00>", "<... | |
| llama_model_loader: - kv 14: tokenizer.ggml.scores arr[f32,32064] = [0.000000, 0.000000, 0.000000, 0.0000... | |
| llama_model_loader: - kv 15: tokenizer.ggml.token_type arr[i32,32064] = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ... | |
| llama_model_loader: - kv 16: tokenizer.ggml.bos_token_id u32 = 1 | |
| llama_model_loader: - kv 17: tokenizer.ggml.eos_token_id u32 = 32000 | |
| llama_model_loader: - kv 18: tokenizer.ggml.unknown_token_id u32 = 0 | |
| llama_model_loader: - kv 19: tokenizer.ggml.padding_token_id u32 = 32000 | |
| llama_model_loader: - kv 20: tokenizer.ggml.add_bos_token bool = true | |
| llama_model_loader: - kv 21: tokenizer.ggml.add_eos_token bool = false | |
| llama_model_loader: - kv 22: tokenizer.chat_template str = {{ bos_token }}{% for message in mess... | |
| llama_model_loader: - kv 23: general.quantization_version u32 = 2 | |
| llama_model_loader: - type f32: 65 tensors | |
| llama_model_loader: - type q4_K: 81 tensors | |
| llama_model_loader: - type q5_K: 32 tensors | |
| llama_model_loader: - type q6_K: 17 tensors | |
| print_info: file format = GGUF V3 (latest) | |
| print_info: file type = Q4_K - Medium | |
| print_info: file size = 2.23 GiB (5.01 BPW) | |
| init_tokenizer: initializing tokenizer for type 1 | |
| load: control-looking token: 32007 '<|end|>' was not control-type; this is probably a bug in the model. its type will be overridden | |
| load: control-looking token: 32000 '<|endoftext|>' was not control-type; this is probably a bug in the model. its type will be overridden | |
| load: control token: 2 '</s>' is not marked as EOG | |
| load: control token: 1 '<s>' is not marked as EOG | |
| load: printing all EOG tokens: | |
| load: - 32000 ('<|endoftext|>') | |
| load: - 32007 ('<|end|>') | |
| load: special tokens cache size = 67 | |
| load: token to piece cache size = 0.1690 MB | |
| print_info: arch = phi3 | |
| print_info: vocab_only = 0 | |
| print_info: n_ctx_train = 4096 | |
| print_info: n_embd = 3072 | |
| print_info: n_layer = 32 | |
| print_info: n_head = 32 | |
| print_info: n_head_kv = 32 | |
| print_info: n_rot = 96 | |
| print_info: n_swa = 0 | |
| print_info: is_swa_any = 0 | |
| print_info: n_embd_head_k = 96 | |
| print_info: n_embd_head_v = 96 | |
| print_info: n_gqa = 1 | |
| print_info: n_embd_k_gqa = 3072 | |
| print_info: n_embd_v_gqa = 3072 | |
| print_info: f_norm_eps = 0.0e+00 | |
| print_info: f_norm_rms_eps = 1.0e-05 | |
| print_info: f_clamp_kqv = 0.0e+00 | |
| print_info: f_max_alibi_bias = 0.0e+00 | |
| print_info: f_logit_scale = 0.0e+00 | |
| print_info: f_attn_scale = 0.0e+00 | |
| print_info: n_ff = 8192 | |
| print_info: n_expert = 0 | |
| print_info: n_expert_used = 0 | |
| print_info: causal attn = 1 | |
| print_info: pooling type = 0 | |
| print_info: rope type = 2 | |
| print_info: rope scaling = linear | |
| print_info: freq_base_train = 10000.0 | |
| print_info: freq_scale_train = 1 | |
| print_info: n_ctx_orig_yarn = 4096 | |
| print_info: rope_finetuned = unknown | |
| print_info: model type = 3B | |
| print_info: model params = 3.82 B | |
| print_info: general.name = Phi3 | |
| print_info: vocab type = SPM | |
| print_info: n_vocab = 32064 | |
| print_info: n_merges = 0 | |
| print_info: BOS token = 1 '<s>' | |
| print_info: EOS token = 32000 '<|endoftext|>' | |
| print_info: EOT token = 32007 '<|end|>' | |
| print_info: UNK token = 0 '<unk>' | |
| print_info: PAD token = 32000 '<|endoftext|>' | |
| print_info: LF token = 13 '<0x0A>' | |
| print_info: EOG token = 32000 '<|endoftext|>' | |
| print_info: EOG token = 32007 '<|end|>' | |
| print_info: max token length = 48 | |
| load_tensors: loading model tensors, this can take a while... (mmap = true) | |
| load_tensors: layer 0 assigned to device CPU, is_swa = 0 | |
| load_tensors: layer 1 assigned to device CPU, is_swa = 0 | |
| load_tensors: layer 2 assigned to device CPU, is_swa = 0 | |
| load_tensors: layer 3 assigned to device CPU, is_swa = 0 | |
| load_tensors: layer 4 assigned to device CPU, is_swa = 0 | |
| load_tensors: layer 5 assigned to device CPU, is_swa = 0 | |
| load_tensors: layer 6 assigned to device CPU, is_swa = 0 | |
| load_tensors: layer 7 assigned to device CPU, is_swa = 0 | |
| load_tensors: layer 8 assigned to device CPU, is_swa = 0 | |
| load_tensors: layer 9 assigned to device CPU, is_swa = 0 | |
| load_tensors: layer 10 assigned to device CPU, is_swa = 0 | |
| load_tensors: layer 11 assigned to device CPU, is_swa = 0 | |
| load_tensors: layer 12 assigned to device CPU, is_swa = 0 | |
| load_tensors: layer 13 assigned to device CPU, is_swa = 0 | |
| load_tensors: layer 14 assigned to device CPU, is_swa = 0 | |
| load_tensors: layer 15 assigned to device CPU, is_swa = 0 | |
| load_tensors: layer 16 assigned to device CPU, is_swa = 0 | |
| load_tensors: layer 17 assigned to device CPU, is_swa = 0 | |
| load_tensors: layer 18 assigned to device CPU, is_swa = 0 | |
| load_tensors: layer 19 assigned to device CPU, is_swa = 0 | |
| load_tensors: layer 20 assigned to device CPU, is_swa = 0 | |
| load_tensors: layer 21 assigned to device CPU, is_swa = 0 | |
| load_tensors: layer 22 assigned to device CPU, is_swa = 0 | |
| load_tensors: layer 23 assigned to device CPU, is_swa = 0 | |
| load_tensors: layer 24 assigned to device CPU, is_swa = 0 | |
| load_tensors: layer 25 assigned to device CPU, is_swa = 0 | |
| load_tensors: layer 26 assigned to device CPU, is_swa = 0 | |
| load_tensors: layer 27 assigned to device CPU, is_swa = 0 | |
| load_tensors: layer 28 assigned to device CPU, is_swa = 0 | |
| load_tensors: layer 29 assigned to device CPU, is_swa = 0 | |
| load_tensors: layer 30 assigned to device CPU, is_swa = 0 | |
| load_tensors: layer 31 assigned to device CPU, is_swa = 0 | |
| load_tensors: layer 32 assigned to device CPU, is_swa = 0 | |
| load_tensors: tensor 'token_embd.weight' (q4_K) (and 114 others) cannot be used with preferred buffer type CPU_REPACK, using CPU instead | |
| load_tensors: CPU_REPACK model buffer size = 1242.00 MiB | |
| load_tensors: CPU_Mapped model buffer size = 2281.66 MiB | |
| repack: repack tensor blk.0.attn_output.weight with q4_K_8x8 | |
| repack: repack tensor blk.0.ffn_up.weight with q4_K_8x8 | |
| .repack: repack tensor blk.1.attn_output.weight with q4_K_8x8 | |
| repack: repack tensor blk.1.ffn_up.weight with q4_K_8x8 | |
| .repack: repack tensor blk.2.attn_output.weight with q4_K_8x8 | |
| .repack: repack tensor blk.2.ffn_up.weight with q4_K_8x8 | |
| .repack: repack tensor blk.3.attn_output.weight with q4_K_8x8 | |
| repack: repack tensor blk.3.ffn_down.weight with q4_K_8x8 | |
| .repack: repack tensor blk.3.ffn_up.weight with q4_K_8x8 | |
| .repack: repack tensor blk.4.attn_output.weight with q4_K_8x8 | |
| repack: repack tensor blk.4.ffn_up.weight with q4_K_8x8 | |
| .repack: repack tensor blk.5.attn_output.weight with q4_K_8x8 | |
| repack: repack tensor blk.5.ffn_down.weight with q4_K_8x8 | |
| .repack: repack tensor blk.5.ffn_up.weight with q4_K_8x8 | |
| .repack: repack tensor blk.6.attn_output.weight with q4_K_8x8 | |
| repack: repack tensor blk.6.ffn_down.weight with q4_K_8x8 | |
| .repack: repack tensor blk.6.ffn_up.weight with q4_K_8x8 | |
| .repack: repack tensor blk.7.attn_output.weight with q4_K_8x8 | |
| repack: repack tensor blk.7.ffn_up.weight with q4_K_8x8 | |
| .repack: repack tensor blk.8.attn_output.weight with q4_K_8x8 | |
| repack: repack tensor blk.8.ffn_down.weight with q4_K_8x8 | |
| repack: repack tensor blk.8.ffn_up.weight with q4_K_8x8 | |
| .repack: repack tensor blk.9.attn_output.weight with q4_K_8x8 | |
| repack: repack tensor blk.9.ffn_down.weight with q4_K_8x8 | |
| repack: repack tensor blk.9.ffn_up.weight with q4_K_8x8 | |
| .repack: repack tensor blk.10.attn_output.weight with q4_K_8x8 | |
| repack: repack tensor blk.10.ffn_up.weight with q4_K_8x8 | |
| .repack: repack tensor blk.11.attn_output.weight with q4_K_8x8 | |
| repack: repack tensor blk.11.ffn_up.weight with q4_K_8x8 | |
| .repack: repack tensor blk.12.attn_output.weight with q4_K_8x8 | |
| .repack: repack tensor blk.12.ffn_down.weight with q4_K_8x8 | |
| repack: repack tensor blk.12.ffn_up.weight with q4_K_8x8 | |
| .repack: repack tensor blk.13.attn_output.weight with q4_K_8x8 | |
| .repack: repack tensor blk.13.ffn_down.weight with q4_K_8x8 | |
| repack: repack tensor blk.13.ffn_up.weight with q4_K_8x8 | |
| .repack: repack tensor blk.14.attn_output.weight with q4_K_8x8 | |
| .repack: repack tensor blk.14.ffn_up.weight with q4_K_8x8 | |
| .repack: repack tensor blk.15.attn_output.weight with q4_K_8x8 | |
| repack: repack tensor blk.15.ffn_down.weight with q4_K_8x8 | |
| .repack: repack tensor blk.15.ffn_up.weight with q4_K_8x8 | |
| .repack: repack tensor blk.16.attn_output.weight with q4_K_8x8 | |
| repack: repack tensor blk.16.ffn_down.weight with q4_K_8x8 | |
| .repack: repack tensor blk.16.ffn_up.weight with q4_K_8x8 | |
| .repack: repack tensor blk.17.attn_output.weight with q4_K_8x8 | |
| repack: repack tensor blk.17.ffn_up.weight with q4_K_8x8 | |
| .repack: repack tensor blk.18.attn_output.weight with q4_K_8x8 | |
| repack: repack tensor blk.18.ffn_down.weight with q4_K_8x8 | |
| .repack: repack tensor blk.18.ffn_up.weight with q4_K_8x8 | |
| .repack: repack tensor blk.19.attn_output.weight with q4_K_8x8 | |
| repack: repack tensor blk.19.ffn_down.weight with q4_K_8x8 | |
| .repack: repack tensor blk.19.ffn_up.weight with q4_K_8x8 | |
| .repack: repack tensor blk.20.attn_output.weight with q4_K_8x8 | |
| repack: repack tensor blk.20.ffn_down.weight with q4_K_8x8 | |
| .repack: repack tensor blk.20.ffn_up.weight with q4_K_8x8 | |
| .repack: repack tensor blk.21.attn_output.weight with q4_K_8x8 | |
| repack: repack tensor blk.21.ffn_up.weight with q4_K_8x8 | |
| .repack: repack tensor blk.22.attn_output.weight with q4_K_8x8 | |
| repack: repack tensor blk.22.ffn_down.weight with q4_K_8x8 | |
| repack: repack tensor blk.22.ffn_up.weight with q4_K_8x8 | |
| .repack: repack tensor blk.23.attn_output.weight with q4_K_8x8 | |
| repack: repack tensor blk.23.ffn_down.weight with q4_K_8x8 | |
| repack: repack tensor blk.23.ffn_up.weight with q4_K_8x8 | |
| .repack: repack tensor blk.24.attn_output.weight with q4_K_8x8 | |
| repack: repack tensor blk.24.ffn_up.weight with q4_K_8x8 | |
| .repack: repack tensor blk.25.attn_output.weight with q4_K_8x8 | |
| repack: repack tensor blk.25.ffn_down.weight with q4_K_8x8 | |
| .repack: repack tensor blk.25.ffn_up.weight with q4_K_8x8 | |
| .repack: repack tensor blk.26.attn_output.weight with q4_K_8x8 | |
| repack: repack tensor blk.26.ffn_down.weight with q4_K_8x8 | |
| .repack: repack tensor blk.26.ffn_up.weight with q4_K_8x8 | |
| .repack: repack tensor blk.27.attn_output.weight with q4_K_8x8 | |
| repack: repack tensor blk.27.ffn_up.weight with q4_K_8x8 | |
| .repack: repack tensor blk.28.attn_output.weight with q4_K_8x8 | |
| .repack: repack tensor blk.28.ffn_up.weight with q4_K_8x8 | |
| .repack: repack tensor blk.29.attn_output.weight with q4_K_8x8 | |
| repack: repack tensor blk.29.ffn_up.weight with q4_K_8x8 | |
| .repack: repack tensor blk.30.attn_output.weight with q4_K_8x8 | |
| repack: repack tensor blk.30.ffn_up.weight with q4_K_8x8 | |
| .repack: repack tensor blk.31.attn_output.weight with q4_K_8x8 | |
| repack: repack tensor blk.31.ffn_up.weight with q4_K_8x8 | |
| ........................................... | |
| llama_context: constructing llama_context | |
| llama_context: n_seq_max = 16 | |
| llama_context: n_ctx = 4096 | |
| llama_context: n_ctx_per_seq = 256 | |
| llama_context: n_batch = 512 | |
| llama_context: n_ubatch = 512 | |
| llama_context: causal_attn = 1 | |
| llama_context: flash_attn = 0 | |
| llama_context: kv_unified = false | |
| llama_context: freq_base = 10000.0 | |
| llama_context: freq_scale = 1 | |
| llama_context: n_ctx_per_seq (256) < n_ctx_train (4096) -- the full capacity of the model will not be utilized | |
| set_abort_callback: call | |
| llama_context: CPU output buffer size = 1.96 MiB | |
| create_memory: n_ctx = 4096 (padded) | |
| llama_kv_cache_unified: layer 0: dev = CPU | |
| llama_kv_cache_unified: layer 1: dev = CPU | |
| llama_kv_cache_unified: layer 2: dev = CPU | |
| llama_kv_cache_unified: layer 3: dev = CPU | |
| llama_kv_cache_unified: layer 4: dev = CPU | |
| llama_kv_cache_unified: layer 5: dev = CPU | |
| llama_kv_cache_unified: layer 6: dev = CPU | |
| llama_kv_cache_unified: layer 7: dev = CPU | |
| llama_kv_cache_unified: layer 8: dev = CPU | |
| llama_kv_cache_unified: layer 9: dev = CPU | |
| llama_kv_cache_unified: layer 10: dev = CPU | |
| llama_kv_cache_unified: layer 11: dev = CPU | |
| llama_kv_cache_unified: layer 12: dev = CPU | |
| llama_kv_cache_unified: layer 13: dev = CPU | |
| llama_kv_cache_unified: layer 14: dev = CPU | |
| llama_kv_cache_unified: layer 15: dev = CPU | |
| llama_kv_cache_unified: layer 16: dev = CPU | |
| llama_kv_cache_unified: layer 17: dev = CPU | |
| llama_kv_cache_unified: layer 18: dev = CPU | |
| llama_kv_cache_unified: layer 19: dev = CPU | |
| llama_kv_cache_unified: layer 20: dev = CPU | |
| llama_kv_cache_unified: layer 21: dev = CPU | |
| llama_kv_cache_unified: layer 22: dev = CPU | |
| llama_kv_cache_unified: layer 23: dev = CPU | |
| llama_kv_cache_unified: layer 24: dev = CPU | |
| llama_kv_cache_unified: layer 25: dev = CPU | |
| llama_kv_cache_unified: layer 26: dev = CPU | |
| llama_kv_cache_unified: layer 27: dev = CPU | |
| llama_kv_cache_unified: layer 28: dev = CPU | |
| llama_kv_cache_unified: layer 29: dev = CPU | |
| llama_kv_cache_unified: layer 30: dev = CPU | |
| llama_kv_cache_unified: layer 31: dev = CPU | |
| llama_kv_cache_unified: CPU KV buffer size = 1536.00 MiB | |
| llama_kv_cache_unified: size = 1536.00 MiB ( 256 cells, 32 layers, 16/16 seqs), K (f16): 768.00 MiB, V (f16): 768.00 MiB | |
| llama_context: enumerating backends | |
| llama_context: backend_ptrs.size() = 1 | |
| llama_context: max_nodes = 1560 | |
| llama_context: worst-case: n_tokens = 512, n_seqs = 16, n_outputs = 0 | |
| graph_reserve: reserving a graph for ubatch with n_tokens = 512, n_seqs = 16, n_outputs = 512 | |
| graph_reserve: reserving a graph for ubatch with n_tokens = 16, n_seqs = 16, n_outputs = 16 | |
| graph_reserve: reserving a graph for ubatch with n_tokens = 512, n_seqs = 16, n_outputs = 512 | |
| llama_context: CPU compute buffer size = 73.01 MiB | |
| llama_context: graph nodes = 1126 | |
| llama_context: graph splits = 1 | |
| INFO: Started server process [12669] | |
| INFO: Waiting for application startup. | |
| INFO: Application startup complete. | |
| INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit) | |
| Batch loop started... | |
| INFO: 127.0.0.1:59484 - "POST /chat?prompt=Count%20to%2010 HTTP/1.1" 200 OK | |