Treffer: Building efficient and expressive distributed computing systems

Ph.D. ; Distributed computing has become the norm and serve as the backbone in the computing infrastructures used by many organizations. More and more applications are built by distributed programming frameworks, allowing them to achieve scalability, availability, and fault tolerance. ; One critical challenge in distributed computing is to balance the expressiveness of the framework and the efficiency of the computing system, which is the main focus of this thesis. On one hand, there are existing systems that are designed for a specific class of applications (e.g., Machine Learning), but are not applicable or not efficient for other applications. This could create significant context switch overhead in the data pipeline due to the extra cost incurred by data movement and data transformation. On the other hand, there are systems that aim at generality, but lack efficiency for many performance-critical applications due to the over-simplified programming interfaces. ; The above dilemma serves as the background and the primary motivation of this thesis, which explores the design space of both the programming interfaces and the underlying distributed computing engine, and aims at achieving better expressiveness and efficiency. Our proposed solution, Husky, addresses the problem with a novel design of a computing framework that not only expresses existing computing paradigms concisely, but also allows them to be composed together. It can even support more flexible computing patterns that are beyond the expressiveness of existing frameworks. In addition to task execution efficiency, this thesis also explores the design space in task scheduling, and proposes useful abstractions that generalize resource sharing among tasks of different natures, in order to further enhance overall task throughput in the Husky computing system. ; 分佈式計算已經成為了很多公司與組織使用的計算基礎設施中的普遍標準和主要基礎。越來越多的應用正在遷移到分佈式框架，以實現更好的規模可伸縮性，可用性，和容錯性。 ...