Neural Arithmetic Logic Units

[WIP]

This is a PyTorch implementation of Neural Arithmetic Logic Units by Andrew Trask, Felix Hill, Scott Reed, Jack Rae, Chris Dyer and Phil Blunsom.

API

# single layer modules
NAC(in_features, out_features)
NALU(in_features, out_features)

# stacked layers
MultiLayerNAC(num_layers, in_dim, hidden_dim, out_dim)
MultiLayerNALU(num_layers, in_dim, hidden_dim, out_dim)

Experiments

To reproduce "Numerical Extrapolation Failures in Neural Networks" (Section 1.1), run:

python failures.py

This should generate the following plot:

To reproduce "Simple Function Learning Tasks" (Section 4.1), run:

python function_learning.py

This should generate a text file called interpolation.txt with the following results. (Currently only supports interpolation, I'm working on the rest. Also getting nans which I'm investigating.)

	Relu6	None	NAC	NALU
a + b	0.002	0.000	0.000	1.399
a - b	0.046	0.000	0.000	0.224
a * b	83.012	99.590	98.822	12.237
a / b	2245.560	2888.195	2765.908	nan
a ^ 2	76.126	99.106	99.559	nan