Co-moderated a very interesting q&a with Inceptive co-founder and “Attention is all you need” co-author Jakob Uszkoreit this morning.

Key takeaways of a session that could have easily went on for hours on a very decent level of technical depth:

• The supreme importance of computational efficiency in ML (see Sutton’s “bitter lesson”).
• The relative unimportance of ideas of how/why stuff works, compared to empirical findings.
• The importance of a good fit of your idea with the computational affordances (see #1 above). Current computers are digital, but AI would much better run on some digital-analog hybrid (yet to be developed)
• The original idea behind the famous paper: As language is roughly but not precisely tree-structured (subject, predicate, object etc.), create an architecture that can parse language by considering, hierarchically, any pair of words to consider all possible edges in that imperfect, possibly cyclic tree; do this in a fashion that is efficiently implementable on parallel digital hardwareThanks for all who contributed, including the 100 participants, Pascal Kaufmann and Mindfire for all the logistics and creating the overall occasion with the Swiss AI Award, ZHAW digital & ZHAW Centre for Artificial Intelligence (CAI) co-organizing, ETH AI Center for hosting, Benjamin F. Grewe for co-moderating!

Link to original article