Posts tagged training-data
How much data does a transformer need to learn repetition?
We systematically degrade the repetition signal in the training data — token by token, and row by row — and find a critical threshold below which induction heads cease to form. Even 10% of tokens in repeated sequences is enough.
Token distribution drives repetition learning
We surgically replace the tokens inside repeated sequences with random tokens, while keeping the sequence structure fixed to investigate the impact on repetition performance.
Is natural language special for learning repetition?
We reverse all tokens in the Pile dataset and find that a transformer trained on completely unnatural data still learns to repeat sequences — suggesting linguistic structure is not required for induction head formation.
Repetition is surprisingly ubiquitous in tokenized natural language
55% of tokens in the tokenized Pile dataset are part of repeated sequences, defined as either A or B in ...AB...AB, and we characterise the structure of those repetitions in detail.