In machine learning we often need to build models that predict multiple outputs for a single instance (we can point to the large areas of multi-label classification, and multi-target regression, involving a applications in diverse domains: text categorization, image labeling, signal classification, time-series forecasting, recommender systems, …). There is a common assumption through much of this literature that one should model the outputs together due to the presence of dependence among them. Intuitively this makes sense, but others argue, often convincingly, that modeling the outputs independently is sufficient. Much of this discrepancy can be resolved after knowing which loss metric(s) are under consideration, however there is a more interesting story to tell since empirical and theoretical results sometimes contradict each other, and years of activity in these areas still do not give us a full picture of the mechanisms behind the relative success (or lack thereof) of modeling outputs/labels/tasks together. In exploring what it means for one label to be `dependent’ on another, we take a path through some old and some new areas of the literature. We come across interesting results which we then take into the area of transfer learning, to challenge the long-held assumption that the source task must be similar to the target task.