Starting up this blog again (MATS)

I’ll be attending MATS, studying mechanistic interpretability with Adrià Garriga-Alonso as my mentor.

I’ve been spending the last few months skilling up on this, and the program starts on Monday. I think doing a weekly/biweekly blog post would be useful for documenting what I learn.

Broadly, it’s been interesting learning about mech interp. It’s a field where you rarely have concrete proofs, instead you just find lines of evidence for/against a claim and try to build arguments based on those. Those arguments may be later overturned, that’s okay. This means that your “results” are less what theories you have, and more about what evidence you found.

The main areas I’m interested in (subject to change) are:

(I’ll talk about and explain these more in later posts if I end up working on them)

- Mechanistic Interpretability of the steering vectors learned by Contrastive Activation Adding

- Implementing Activation Adding in some of the popular chatbot uis (for example, emotion control) or making an instruction following vector+Hermes (to get more people interested in improving activation adding)

- Making automated circuit discovery do more of typical analysis steps/making the output more nice (Would be cool to have ACDC output a Jupyter notebook full of all the evidence it found based on the data you gave it)

- Automatic discovery of multi-word tokens entities based on seeing when first few layers shove meaning into the last token (and when ablated away it changes meaning on later token)

- Each head in each layer “enriches” tokens with additional meaning. Look at the statistics of this over a large dataset. Given any token, visualize its enrichments: What do the top PCA components unembed into? What are the statistics of unembedding each enrichment (scaled by their magnitude)

(LLMs have garbage enrichment that they’ll do, like confidently adding a sport to someone that doesn’t do sports. This isn’t an issue for LLMs because they won’t look up those wrong enrichments since their lookups are ANDs over name and sport. But this will mean I get many irrelevant enrichments. Can I filter based on lookups?)

I’ll also probably make a few blog posts just documenting some of the cool stuff I’ve learned in my readings.

So far, I really enjoy mech interp research! It has short feedback loops which fits with me better, and there’s some major progress that’s been made in a relatively short period of time! There seem to be many things to do and I’m looking forward to my time at MATS :)

Otherwise, I’ve been working on reproducing the Stanford LLM AI Sims thing, but a little more general and with open source LLMs. I may make a post sometime on all the stuff I’ve learned in the process of doing that

Comments

Post a Comment

Popular posts from this blog

My prediction of AGI in 2021-2023

In 2020 I made a prediction on Facebook that AGI was only 1-3 years away. Probably time to follow up on that. At the time, Gwern’s “You can only prove the presence of an ability, not the absence of it” seemed solid. Someone would say “GPT-3 can’t do X”, and then later someone would find a prompt where it could reliably do X! There was a lot of talk about becoming a good “prompt engineer”: being good at wording things in a way that cause language models to reliably do what you want. Relatedly, these days my favorite way to think about language models is as a Simulator (Janus popularized this framing here  https://generative.ink/posts/simulators/ ). Inside them is all these different persons/personalities, each with different abilities (try character.ai to experience this). Prompt engineering pulls out a specific simulated person and inhibits many others. If you get the wrong person, it’ll fail at things other simulated people are very capable of! But honestly, I think ChatGPT proved Gwe
Read more

OpenAI Scholars Project

My blog post is split up into three parts: Literature Review (incomplete, this is a very large field of study): Lenses of Analysis on Opinion Spread . This post is an older version that contains more detailed discussion and why I didn't include some things (like advertising), for those that are interested. Experiments and Analysis Section: Training models on their own outputs . Speculative Section:  What we'll (probably) see before we build artificial cultures . This is just some research ideas and thoughts I had after doing lots of reading, take them with a grain of salt.
Read more

Inductive Linguistic Biases

Image
(unfinished blog post, posting for reference) Introduction Humans have "inductive biases". When humans see data that they could draw multiple conclusions from, they often pick one of those conclusions. There are certain kinds of conclusions that humans are more likely to come up with, and those conclusions are our inductive biases. For example, humans tend to assume the sequence (1,2,3) will be followed by (4,5,6,7,...) even though there are many other conclusions you could come to. These inductive biases are very important for understanding how humans can learn from so little data: many mechanisms in our society involve learning to coordinate , instead of learning the "correct" behaviour. Often these norms determined based on the previous generation's "best guesses", and since you share the same inductive biases, you are likely to make those same guesses, and so can learn to coordinate easier. This means that if you can create AI models that have ve
Read more