mirror of
https://github.com/wassname/NALU-pytorch.git
synced 2026-08-04 12:23:42 +08:00
72 lines
2.3 KiB
Markdown
72 lines
2.3 KiB
Markdown
Quick experiment: What if we used asinh instead of log in Neural Arithmetic Logic Units? Idea from [reddit user fdskjfdskhfkjds](https://www.reddit.com/r/MachineLearning/comments/94833t/neural_arithmetic_logic_units/e3u974x/)
|
|
|
|
|
|
| | NAC_exact | NALU_sinh | Relu6 | None | NAC | NALU |
|
|
| :------ | :-------- | :-------- | :----- | :---- | :---- | :----- |
|
|
| a + b | 0.133 | 0.530 | 3.846 | 0.140 | 0.155 | 0.139 |
|
|
| a - b | 3.642 | 5.513 | 87.524 | 1.774 | 0.986 | 10.864 |
|
|
| a * b | 1.525 | 0.444 | 4.082 | 0.319 | 2.889 | 2.139 |
|
|
| a / b | 0.266 | 0.796 | 4.337 | 0.341 | 2.002 | 1.547 |
|
|
| a ^ 2 | 1.127 | 1.100 | 92.235 | 0.763 | 4.867 | 0.852 |
|
|
| sqrt(a) | 0.951 | 0.798 | 85.603 | 0.549 | 4.589 | 0.511 |
|
|
|
|
It appears that NALU_sinh is better at division than NALU.
|
|
|
|
|
|
# Neural Arithmetic Logic Units
|
|
|
|
[WIP]
|
|
|
|
This is a PyTorch implementation of [Neural Arithmetic Logic Units](https://arxiv.org/abs/1808.00508) by *Andrew Trask, Felix Hill, Scott Reed, Jack Rae, Chris Dyer and Phil Blunsom*.
|
|
|
|
<p align="center">
|
|
<img src="./imgs/arch.png" alt="Drawing", width=60%>
|
|
</p>
|
|
|
|
## API
|
|
|
|
```python
|
|
from models import *
|
|
|
|
# single layer modules
|
|
NeuralAccumulatorCell(in_dim, out_dim)
|
|
NeuralArithmeticLogicUnitCell(in_dim, out_dim)
|
|
|
|
# stacked layers
|
|
NAC(num_layers, in_dim, hidden_dim, out_dim)
|
|
NALU(num_layers, in_dim, hidden_dim, out_dim)
|
|
```
|
|
|
|
## Experiments
|
|
|
|
To reproduce "Numerical Extrapolation Failures in Neural Networks" (Section 1.1), run:
|
|
|
|
```python
|
|
python failures.py
|
|
```
|
|
|
|
This should generate the following plot:
|
|
|
|
<p align="center">
|
|
<img src="./imgs/extrapolation.png" alt="Drawing", width=60%>
|
|
</p>
|
|
|
|
To reproduce "Simple Function Learning Tasks" (Section 4.1), run:
|
|
|
|
```python
|
|
python function_learning.py
|
|
```
|
|
This should generate a text file called `interpolation.txt` with the following results. (Currently only supports interpolation, I'm working on the rest)
|
|
|
|
|
|
|
|
| | Relu6 | None | NAC | NALU |
|
|
|---------|----------|----------|----------|--------|
|
|
| a + b | 4.472 | 0.132 | 0.154 | 0.157 |
|
|
| a - b | 85.727 | 2.224 | 2.403 | 34.610 |
|
|
| a * b | 89.257 | 4.573 | 5.382 | 1.236 |
|
|
| a / b | 97.070 | 60.594 | 5.730 | 3.042 |
|
|
| a ^ 2 | 89.987 | 2.977 | 4.718 | 1.117 |
|
|
| sqrt(a) | 5.939 | 40.243 | 7.263 | 1.119 |
|
|
|