Tinyllama from scratch

Experimental minimal tinyllama training and finetuning on Tesla Nvidia P100

Full dataset

Download TinyStories_all_data.tar.gz from: https://huggingface.co/datasets/roneneldan/TinyStories

Stories models

https://huggingface.co/karpathy/tinyllamas/

Download 15M model and put it in ./res/asset/models/karpathy/tinyllama/

Training from scratch

start with preparing dataset:

(use full dataset for better results)

./src-finetuning/prepare_instruct_dataset.py

Train from scratch using the generated data:

./src-finetuning/instruct_training_from_scratch.py

Instruct lora finetune using 15M tinyllama model

./src-finetuning/instruct_lora_finetune.py

Inference

Example use of inferencing script:

python src-finetuning/generate.py --model_path='./build/models/lora_story_teller_110M.pt' --prompt='Write a story. In the story, try to use the verb "climb", the noun "ring" and the adjective "messy". Possible story:' --temperature=0.1 --top_k=10

Acknowledgements

Based on:

• https://github.com/cindysridykhan/instruct_storyteller_tinyllama2/
• https://github.com/ruirui-zhang/instruct_storyteller_tinyllama2
• https://github.com/karpathy/nanoGPT
• https://github.com/karpathy/llama2.c

License

Apache-2.0