Langchain LLM Update

Saturday, April 20, 2024

As many of you know, I have been building a personal AI assistant using Large Language Model transformers. The goal is to have it access documents and internet feeds to take care of different tasks.

This week I have been experimenting with Meta’s newest iteration of their open source transformer, Llama3. I downloaded their 8B model and started some experiments. Initial results show their changes to the EOT (end of token) or stop token, which results in the model continuously answering itself. While this leads to some ammusing replies, it’s a little less than ideal. I need to go back to their document page and update the ‘assistant’ end of token standard (commonly <|eot|>). The Quant GGUF file for Llama (from Quant) works properly, but I’m not as happy with the decoded results that it seems to be giving. I’ll have to continue exploring solutions in Langchain.

I also have been comparing both Mixtral (8x7b and 8x22b) and Qwen (32b) and both have been giving amazing and snappy results on my Macbook M3 Max with 36 gigs of RAM. My results are consistent with other benchmarks and prototyping results that others are getting.

I’ll end by saying, it’s an exciting time to be experimenting with LLMs. There are so many open source models that are hitting very high marks, and fine tuning and building and chaining RAGs onto the base models is becoming significantly more powerful.
#ll #qwen #llama3 #langchain #mixtral #ai




Ciao! I'm Scott Sullivan, an software developer and machine learning nerd. I divide my time between the tranquil countryside of Lancaster, Pennsylvania, and northern Italy, visiting family, close to Cinque Terre and La Spezia. Professionally, I'm using my Master's in Data Analytics and my Bachelor's degree in Computer Science, to create compelling software products that user AI, run lighting, robots, and automation effects for a large Christian theatrical productions to spread the message of Christ's salvation.