Bridging the gap between Arts and Science
Independent research I conducted during my time at Columbia.
In collaboration with Jacklyn Tsai and Rick Xu. We developed a constrained sampling strategy, proved to be helpful in eliciting hallucinations in white-box LLMs. We used it to improve LLAMA2-7B score on StrategyQA and CommonsenseQA by 3% without any further training.
I break down how I built, pretrained and fine-tuned DNA foundation models using PyTorch Lightning with data parallelism, learning rate scheduling, gradient accumulation, gradient checkpointing and others. Dive in to see how State Space Models can yields an efficient architecture concurrencing classical transformer-based models.
RAG-Maestro is an up-to-date LLM assistant designed to provide clear and concise explanations of scientific concepts and cite relevant papers. It is powered by a custom paper bot scraper and a RAG-Pipeline leveraging gpt-3.5 and text-embedding-ada-002 from OpenAI
My Python implementation of classic sampling strategies discussed in The Curious Case of Neural Text Degeneration.
Nucleus Sampling and Top-K Sampling are the main contributions.
This project is an attempt to build a language model generating on "old-fashioned" English by fine-tuning state-of-the-art models on ancient texts. Leveraging LoRA (Low Rank Adaptation), I built a Shakespearian-Falcon-7B capable of completing your prompts in poems and old-fashioned English. Further advancements for this projects include turning it into a conversational agent.