The idea of using "auto-regression" transformers to predict the future is actually a very amazing thing. In a time-series streaming video input, the point is to use the most recent several seconds of video (the context window) to predict the next several seconds (the likely or possible futures). Humans do this all the time. As you drive down the road, your attention is drawn here and there and you imagine/form predictions of what might happen next. You also begin or prepare appropriate actions for possible futures. A noisy car is zig-zagging, coming up from behind or passing; you might speed up, slow down, change lanes, etc. A person standing in the median or on the curb might step into the street. You prepare. Etc. An autonomous car that can do this sort of thing as well, changing its predictions and plans constantly with every new video frame, would be amazing. Wait for HW5/AI5!
You are viewing a single comment's thread from: