ML 122: Comparing LLMs on Colab (20 pts extra)

What You Need

A Web browser

Purpose

To use and compare various Large Language Models, running them in a Google Colab under your own control, the way many companies will want to.

Opening the Github

In a browser, go to

A colab gradio web UI for running Large Language Models

Scroll down to the "Colab" section, as shown below.

Stanford's Alpaca

On the "gpt4-x-alpaca-13b-native-4bit-128g" line, right-click the "Open in Colab" button, outlined in red in the image above, and open it in a new tab.

A Google Colab page opens, with a block of code. Run the code.

A pop-up warns that the colab was not authored by Google. Click "Run anyway".

After about five minutes, you'll see a public URL for your GPT server ending in "gradio.live", outlined in red in the image below.

Opening the Web Interface

Click the public URL. The Web interface opens, as shown below.

Paste in this query, and then press the Enter key, or click the Generate button.

Why is the sky blue?

The answer is correct, as shown below.

Letter Search

Clear the history, by scrolling to the bottom of the page and clicking the "Clear history" button, and clicking Confirm.

Now enter this query:

Does the letter "e" appear in the word "ketchup"?

The answer is not accurate, as shown below.

Clear the history, and enter this query:

Consider the word "ketchup". List all the letters in that word.

Clear the history, and enter this query:

Consider the word "ketchup". List all the letters in that word. Also list the number of times each letter appears. Now answer the question below. Q: Does the letter "e" appear in the word "ketchup"? A:

Flag ML 122.1: Alpaca's Answer (5 pts)

Clear the history, and enter this query:

Q: How many times does each letter appear in the word "snowball"? A: S,1; N;1, O;1, W;1, B;1, A;1, L;2 Q: How many times does each letter appear in the word "flounder"? A: F,1; L;1, O;1, U;1, N;1, D;1, E;1 R;1 Q: How many times does each letter appear in the word "water"? A: W,1; A;1, T;1, E;1, R;1 Q: How many times does each letter appear in the word "ketchup"? A:

The flag is covered by a green rectangle in the image below.

Clearing your Colab

Close the Web interface for the LLM.

In Google Colab, from the menu bar, click Runtime, "Disconnect and delete runtime".

Click Yes.

Close the Colab tab.

Meta's LLaMA

On this page:

A colab gradio web UI for running Large Language Models

Launch the stable-vicuna-13B-GPTQ-4bit-128g model.

Run the queries above. Compare the answers to the answers from the other model.

Flag ML 122.2: LLaMA's Answer (5 pts)
Clear the history, and enter this query:
What are the most common letters in the English alphabet?
The flag is covered by a green rectangle in the image below.

Open Assistant

Clear your Colab and launch the oasst-llama13b-GPTQ-4bit-128g model.

Run the queries above. Compare the answers to the answers from the other model.

Flag ML 122.3: Open Assistant's Answer (5 pts)
Clear the history, and enter this query:
What is the next number in this sequence: 1, 3, 5, 7, 9? Explain how you determined your answer.
The flag is covered by a green rectangle in the image below. The answer comes out different each time you regenerate it, but the flag should remain the same.

WizardLM

Clear your Colab and launch the wizard-lm-uncensored-13b-GPTQ-4bit-128g model.

Run the queries above. Compare the answers to the answers from the other model.

Flag ML 122.4: WizardLM's Answer (5 pts)
Clear the history, and enter this query:
Q: How many times does each letter appear in the word "snowball"? A: S,1; N;1, O;1, W;1, B;1, A;1, L;2 Q: How many times does each letter appear in the word "flounder"? A: F,1; L;1, O;1, U;1, N;1, D;1, E;1 R;1 Q: How many times does each letter appear in the word "water"? A: W,1; A;1, T;1, E;1, R;1 Q: How many times does each letter appear in the word "ketchup"? A:
The flag is covered by a green rectangle in the image below. The answer comes out different each time you regenerate it, but the flag should remain the same.

Posted 5-19-23