From 315aea0616c2bce9b9e4e61bf337439b1e8009dc Mon Sep 17 00:00:00 2001 From: wassname Date: Sun, 12 Nov 2023 13:13:24 +0800 Subject: [PATCH] checkout --- journal.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/journal.md b/journal.md index 2d351b0..97b9d41 100644 --- a/journal.md +++ b/journal.md @@ -190,6 +190,16 @@ We add them, plus positions, and add them into one of embed_dim Now how does the paper describe it? https://openreview.net/pdf?id=vhFu1Acb0xb +# 2023-11-12 12:58:27 + +Hmm Ok so they have a seperate encoder and decoder and the transformer just represents the dynamics model? But where are the encoder and decodr? + +Hmm so one image turns into a sequence of tokens, how? +- decoding it just using the tokenizer.embeddings then tokenizer.decode, and it's in the world model + + +OK it looks like I don't actually need a VAE here! The encode decode is already made and is decoupled from the dynamic model. So lets start from scratch, loading minstral... + # IDEAS