"Smarter" is ill-defined.
Are you smart if you see that cognitive scientists had something figured out, connected it to something in your field of chemistry and made a breakthrough treatment for some disease?
Does any of our existing intelligence build on completely unknown information?
You could argue that any new solution/emergent intelligence is an association of existing information in new ways.
It also brings us to what is brute forcing? Humans are brute forcing stuff most of the time. You will have 10 stupid thoughts for every good thought. 10 stupid ideas for every good idea, you will discard most of them so fast that you won't think of that you did that. It's very rare, if not near impossible, that a non-mathematician with no prior knowledge, goes into a room on day 1 of deciding to think about a conjecture they don't know anything about, and solve it on the blackboard. They need to use their collect associations, formalize their axioms, think of first principles (known information), and associate it.
So to answer your question you need to answer:
1) What does "smarter" refer to? Is it accumulation of information about the universe?(if yes, a database is smart) Is it an efficient, abstracted representation of accumulated information? Is it prediction of the future state of the universe given a learned abstraction?
Or going further, is it solving tasks useful for the human in the world? Is it acting as an agent with it's own goals and interests in the universe? What if the goals are hard for humans to understand?
Does it count as smart if you make an evolutionary approach that spawn 1000 instances and the most fit one survives and spawns 1000 mutations of itself?
2) What does "genuinely new approaches" refer to? Is it fine to use math axioms to prove things? Does interpolation between two existing solutions count as new? At what granularity of concept does novelty happen (Does using two higher abstraction concepts, like combining some new, modern neural network architecture with a database count as a new thing?), even if the underlying lower abstraction of the same information can be expressed as storage, latency and compute, and is not novel?
---
As you can imagine, this is all not really defined or agreed upon. Here are some examples of projects that are set up to drive these ideas:
The Hutter Prize poses that intelligence is defined by compression, so if you manage to compress the English Wikipedia efficiently, you are closer to better intelligence (prize.hutter1.net)
Compute: Intel CPU
The ARC-AGI foundation poses that being able to be goal-oriented and understand higher level abstractions is intelligence: arcprize.org
Compute: NVIDIA T40s
Modded NanoGPT competition poses that intelligence is minimizing entropy/loss over a dataset in the least amount of steps: kellerjordan/modded-nanogpt
Compute: 8x NVIDIA H100
---
All of this also exists in the context of the compute medium. It could be, that you need a certain capacity of memory/computation to perform whatever we have defined as intelligent tasks. Hutter Prize takes the bet of that to make intelligence, you don't need a lot of computation, an Intel CPU laptop and 50GB RAM should be enough.
A lot of the big labs are instead taking the bet on that you just scale everything, make trillion-sized models and things will fall out of that automatically, for this you will need a very different compute medium (data centers) compared to (a laptop).
Mentions
"Smarter" is ill-defined. Are you smart if you see that cognitive scientists had something figured out, connected it to something in your field of chemistry and made a breakthrough treatment for some disease? Does any of our existing intelligence build on completely unknown information? You could argue that any new solution/emergent intelligence is an association of existing information in new ways. It also brings us to what is brute forcing? Humans are brute forcing stuff most of the time. You will have 10 stupid thoughts for every good thought. 10 stupid ideas for every good idea, you will discard most of them so fast that you won't think of that you did that. It's very rare, if not near impossible, that a non-mathematician with no prior knowledge, goes into a room on day 1 of deciding to think about a conjecture they don't know anything about, and solve it on the blackboard. They need to use their collect associations, formalize their axioms, think of first principles (known information), and associate it. So to answer your question you need to answer: 1) What does "smarter" refer to? Is it accumulation of information about the universe?(if yes, a database is smart) Is it an efficient, abstracted representation of accumulated information? Is it prediction of the future state of the universe given a learned abstraction? Or going further, is it solving tasks useful for the human in the world? Is it acting as an agent with it's own goals and interests in the universe? What if the goals are hard for humans to understand? Does it count as smart if you make an evolutionary approach that spawn 1000 instances and the most fit one survives and spawns 1000 mutations of itself? 2) What does "genuinely new approaches" refer to? Is it fine to use math axioms to prove things? Does interpolation between two existing solutions count as new? At what granularity of concept does novelty happen (Does using two higher abstraction concepts, like combining some new, modern neural network architecture with a database count as a new thing?), even if the underlying lower abstraction of the same information can be expressed as storage, latency and compute, and is not novel? --- As you can imagine, this is all not really defined or agreed upon. Here are some examples of projects that are set up to drive these ideas: The Hutter Prize poses that intelligence is defined by compression, so if you manage to compress the English Wikipedia efficiently, you are closer to better intelligence (prize.hutter1.net) Compute: Intel CPU The ARC-AGI foundation poses that being able to be goal-oriented and understand higher level abstractions is intelligence: arcprize.org Compute: NVIDIA T40s Modded NanoGPT competition poses that intelligence is minimizing entropy/loss over a dataset in the least amount of steps: kellerjordan/modded-nanogpt Compute: 8x NVIDIA H100 --- All of this also exists in the context of the compute medium. It could be, that you need a certain capacity of memory/computation to perform whatever we have defined as intelligent tasks. Hutter Prize takes the bet of that to make intelligence, you don't need a lot of computation, an Intel CPU laptop and 50GB RAM should be enough. A lot of the big labs are instead taking the bet on that you just scale everything, make trillion-sized models and things will fall out of that automatically, for this you will need a very different compute medium (data centers) compared to (a laptop).
@jon_barron Appendix A7 in the blogpost! it's a new kind of momentum we used to set a Muon nanogpt record: modded-nanogpt#369 tldr: instead of tuning an exponential decay \beta for momentum we can specify desiderata (mean lag, recency bias, etc) and directly solve for the optimal momentum coefficients max ent = maximum entropy, which is the optimization objective slop = because i kind of hate this method, i don't understand why it works, and i want every reader to understand that it's a research object that i don't necessarily endorse
@beyarkay this is because you have to include the logs in the PR from the rules of the repo. it's not actual code modded-nanogpt#367
Record PR: modded-nanogpt#367 Blog post: brunborg.com
I like to point to KellerJordan/modded-nanogpt which does what took OpenAI weeks on a specialized computer cluster in 2018-2019, in a manner of minutes and for less than a dollar worth of on-demand GPU compute
GPT-2(small)相当のNanoGPTを高速化するRTAが、今のベストスコアで8xH100で1分切ってるので、やろうと思えば割と自分だけのデータセットで自作LLMするハードルはだいぶ下がった気がする kellerjordan/modded-nanogpt
Geez, GPT-2-Medium can now be trained on 221 seconds... modded-nanogpt#380 (And small in 9.65 seconds)