Open-ended concept creation through scaffolding

(The scholars program is over, but I'm going to continue doing independent AI research for at least a year, as I continue to build my research chops. My posts might be less frequent, as I also spend some of my time now doing game design, see my other blog)

Most of my research direction is trying to determine if it's possible to create a language model with very little (or no) training data. Ideally, it should still have scaling laws. If we could do this, we could arbitrarily scale our models to continually increase capabilities, and it wouldn't be limited by the presence of data.

Yet, in some ways this is non-sensical. The set of all distributions is very large, so a single model cannot "learn" them. Thus, it must focus on learning a subset of all distributions. Somehow we'd need a way of specifying a distribution that isn't too large to be intractable, while still containing human language and most of the things we care about.

And even if we did that, the system wouldn't have any idea what some concept like "bread" corresponds to. Even putting aside the details of assigning words and sounds to abstract concepts, "bread" as an abstract concept only makes sense in our particular world with our particular physics. And even in our universe under our physical laws, evolution could have happened in a different way where bread has no comparable analogy (unless bread-like substances experience convergent evolution). This is especially true with cultural references like "Elvis". A system may be capable of having an understanding of the concepts we associate with Elvis, in the same way that we may understand an Alien world's celebrities and some of the ideas they portray without understanding much of the cultural particulars. But there's still a significant amount of information that is not universal, and is particular to our world.

Thus, my research project is faced with two choices:

1. Try to find some way of encoding enough of the "human ideas" that make up the ideas we care about into the open-ended problem environments. Carefully make the problems not too general but also not too specific so they capture the right scope of things we care about.

2. Find a way to automatically embed those human ideas as part of the learning process, and just include a separate "boostrapping into open-endedness" process on top of that.

It's possible that as we make progress in understanding how language models encode information, 1 may become more tractable. But for now I have yet to find great paths to trying to do that.

That leaves us with 2. While I dislike it because it sorta goes against the spirit of what I was trying to do, I think there's a context where it's admissible: Learning to create systems that build ideological scaffolding, doing open-ended concept creation through trying to predict the outcomes of a set of learned rules that have emergent behavior. These rules can be pre-specified (my preference), or learned from data.

To explain, I'm going to loosely define a "concept" as a set of associations that one builds from data. If we look at the way a neural network forms concepts (at least for images), it's by composing lots of smaller parts.

There's a question of where concepts come from. If a person grew up in a box with no outside stimulus, would they form the same concepts as us? If they had paper, maybe, given many thousands of years. But maybe not, there's many theories that suggest that brains function as adaptive organisms to their environment. Lacking stimulus can lead to serious cognitive deficiencies, because the brain routes the places that would be used for things like reasoning and language to being used for other things instead.

I'm aware that this question of "where do concepts come from" is a heavily studied and debated question (also see Epistemology), and I still have a lot more reading to do before I can say concrete things here. But here's my current working mental model:

- People form concepts by making associations, and grouping stimulus into categories

- They go a step further and build generative, causal models of their observations. The goal of these generative models is to produce more data to build better models. Yet, by itself, just producing more data and training on that would lead into the collapse problems my project encountered, so there needs to be some kind of feedback. The feedback comes in the form of data being helpful if it leads to the production of generative models that are smaller, but still succinctly explain the observed data (this is a high level view, I think the process needs slack and other related insights in the quality diversity and open-endedness literature).

However, I don't think this is everything. By itself, this seems like it should eventually converge on a working model of the environment, and then stop and do nothing else.

Yet, consider mathematics. In mathematics, we can set up a simple set of rules, and perfectly understand how those rules will be applied to any given object. However, understanding of the rules is not sufficient to give us a full understanding of the implications of those rules. Very simple sets of assumptions lead to huge mathematical worlds. Somehow, in trying to make a unifying theory for a small set of concepts and they ways they interact, we are left with a huge amount of knowledge and plenty of open questions, even after hundreds of years of effort.

One way to understand this is to consider the Curry-Howard correspondence, where our mathematical systems are actually leading to the construction of programs. And with programs, you can understand what'll happen at every step, but understanding what the entire program will do is uncomputable.

A view I find more insightful is thinking of NP-Complete problems. Sometimes, we can be told an idea by someone else or come up with an idea, and the idea will "click" in our head and feel very right to us. This is a setting where our brain was very capable of recognizing what the right solution was, but had difficulty producing it. NP-Complete problems also have a particular structure where it is very difficult to tell how "close" your solution is. You can have something that is almost there, but it turns out you've reached a dead end and need to undo lots of your work. Thus, a better way to start trying to solve the problem is to build heuristics: in this setting, I can do X, in this setting I can do Y. You build these layers of heuristics on top of each other after gathering lots of data, and start to be capable of solving simple problems, or coding programs for certain tasks, or proving certain theorems. You're still not capable in general, but this scaffolding process helps you get somewhere. Eventually you start to make higher order heuristics that use heuristics as primitives, then you build on those higher order heuristics, etc.

I think that a similar thing is happening with our mental concepts. We have some initial set of concepts. These concepts interact with each other to cause non-trivial implications, and we try to make a good mental model of what's going on. In doing so, we start doing scaffolding building: gathering lots of data, building associations, rules, and heuristics, and then trying them out and seeing what works. This process is open-ended, because having a proper solution to it would be solving all NP-Complete problems.

These ideas are all very fuzzy, and I'm aware that I haven't said anything concrete here. I'm still feeling my way around the territory, so that's kinda what I have to do. Now that I'm at this place, I think there's some interesting research questions about if it's possible to have continuous forms of these non-trivial interactions emerge from a set of initially simple rules, or if learned distributions exhibit non-trivial interactions.

But we can start with discrete rules that we know have non-trivial interactions. In particular, what I'd like to do is the following:

1. Have neural networks learn some mathematical rules from a set of data (say, simulating conway's game of life, a set of axioms and applying deduction rules, etc.)

2. Have the neural networks output data adhering to these rules, to try and form predictions about the outcomes that they've implemented. Use that data to refine their predictions.

It might be necessary to freeze the part of the model that implements the rules in order to prevent them from being changed, but I'm also interested in seeing if there's a way to make it so it wants to keep the rules there (due to some particular way the loss is implemented)

This is sort of analogous to using more compute at test time.

It's worth pointing out that if the goal of the model is to form better long term predictions of what happens given the rules it understands, it should also be capable of doing that by just being given more data. In other words, this is just a form of data-augmentation on sequence data. Still, it provides the opportunity for a model to initially learn some generative model, and then "ponder" the implications of that generative model to improve it's predictions. In theory, eventually you could pivot entirely to the "ponder" approach once you ran out of data, and this should give you open-ended improvement. Eventually the model would cap out at it's capacity, but then we can scale up the models, so I don't think that's a fundamental concern.

So this is roughly the research direction I'll be pursuing next :)