AI & ML interests

Local LLMs

Recent Activity

Shrijanagain 
posted an update 3 days ago
view post
Post
68
Building a frontier model doesn't strictly require massive superclusters it demands continuous engineering, smart strategy, strong data pipelines, and sheer determination.

We just crossed a massive milestone in our continuous pre-training phase: 1 Billion Tokens Processed!

Key Progress Updates --

Pre-Training Status -- Uninterrupted continuous run

Target Steps -- 211,033
Current Steps -- 31,000

Tokens Processed: 1,015,808,000+ (1B+ tokens passed through the network)
Architecture: 0.7B parameter footprint, 32 local experts (2 active per token), 262k vocabulary

All checkpoints, logs, and datasets are tracked live in the
Collection - https://huggingface.co/collections/Shrijanagain/how-to-build-a-frontier-model-from-zero
Shrijanagain 
posted an update 6 days ago
view post
Post
122
Building a frontier model doesn't strictly require massive superclusters—it demands continuous engineering, smart strategy, strong data pipelines, and sheer determination.

For the past 17 days, my 0.7B Mixture-of-Experts (MoE) model has been in a continuous pre-training phase, and today we hit a major milestone.

Key Progress Updates --

Pre-Training Duration: 17 days of uninterrupted run
Target Steps: 211,033
Current Steps: 25,030 (25,000+ steps completed)

Tokens Processed: 750M+ tokens passed through the network

Architecture: 0.7B parameter footprint, 32 local experts (2 active per token), 262k vocabulary

All checkpoints, logs, and datasets are tracked live in the Collection -

https://hf.co/collections/Shrijanagain/how-to-build-a-frontier-model-from-zero

25k steps down. Scaling continuous pre-training depth next. Let me know your thoughts in the comments!