Transformers: Causal Language Modeling Versus Mask Language Modeling
Introduction In this post, we will investigate the differences between Causal Language Modeling (CLM) and Masked Language Modeling (MLM) What is Causal Language Modeling? Causal Language Modeling (CLM) is a type of language modeling task where the model is trained to predict the next word in a sequence, given the previous words in the sequence; the model is trained to generate a sequence of words that makes sense in a given context....