Publications
Like flowers that bloom in unexpected places, every story unfolds with beauty and resilience, revealing hidden wonders.
Cable: Context-aware Biases for Length Extrapolation
Cable improves the model’s extrapolation ability significantly with a negligible burden in time and memory compared to the vanilla transformer. Furthermore, compared to existing RPE methods, our approach maintains nearly identical training time and GPU memory usage, while its inference overhead remains either negligible or comparable, depending on the sequence length.
Cable Length Extrapolation
A Cable model trained on T=1024 can extrapolate on T=8192, achieving a better performance (PPL=22.22) compared to the sinusoidal model (PPL=22.81) trained on T=8192.
Cable: Runtime and Memory Overhead
Cable improves the model’s extrapolation ability significantly with a negligible burden in time and memory compared to the vanilla transformer. Furthermore, compared to existing RPE methods, our approach maintains nearly identical training time and GPU memory usage, while its inference overhead remains either negligible or comparable, depending on the sequence length.
Got questions?
Feel free to reach out.