Whenever dealing with a new set of problems apriori, it’s useful to take a step back and decompose the overarching field into it’s primitives, and understand which of them are essential to performing in the regime.
If you relentless poke problems long enough they will reveal wonders. It takes considerable patience to solve a single problem over several hours. It’s also easy to get stuck and not know where to go next. Usually, we have an inductive bias indicate the next best action. Yes this comes with experience but even some without experience can develop a suitable hill climbing algorithm to solve any new problem they’ve never seen before.
It’s like a muscle that needs to be developed with physically demanding exercise.
-
Frame the target
- Why does repeatedly multiplying by a small number make an RNN’s gradient disappear?
- Why must an injective linear map have a trivial kernel?
- What observation would show that my proposed robot controller fails?
- “What remains hard even if everything else works?”
-
Shrink the problem
- Work backwards from the solution
- Create an auxiliary problem (Polya, G. 2014)
-
Make one justified move
Knowing where to start is sometimes the most difficult part. If there is no inductive bias forming as you read the question then you need to hill climb.
- What exactly am I claiming?
- Why should that follow?
- What is the simplest example?
- Can I construct a counterexample?
- What changes if I remove one assumption?
Explore several steps ahead when the consequences are clear. When uncertainty dominates, choose the next step that would teach you the most.
-
Challenge the result
How could this be wrong?
- Try a counterexample or an extreme case. For math problems test the boundary conditions
-
Compress and continue
-
Reflect and update weights
- Try a different solution. Tackle the problem from a different angle.