联邦学习（Federated Learning, FL）作为一种新兴的分布式机器学习方法，已经引起了大量研究的关注。要系统地理解联邦学习的相关研究，建议遵循以下结构化的阅读图谱，以便逐步加深对其原理、应用和挑战的理解。

1. 基础与概念性论文

这些论文介绍了联邦学习的基本概念、目标、以及经典算法，是了解联邦学习的起点。

• Konečnỳ, J., et al. (2016). “Federated Learning: Strategies for Improving Communication Efficiency” arXiv

  • 介绍了联邦学习的概念，提出了最早期的算法（如FedAvg），并讨论了如何优化通信效率。

• McMahan, H. B., et al. (2017). “Communication-Efficient Learning of Deep Networks from Decentralized Data” arXiv

  • 这篇论文提出了经典的Federated Averaging (FedAvg) 算法，系统阐述了在分布式环境下训练深度学习模型时的通信效率问题。

• Yang, Q., Liu, Y., Cheng, Y., Kang, Y., Chen, T., & Yu, H. (2019). “Federated Learning” ACM Transactions on Intelligent Systems and Technology (TIST)

  • 详细综述了联邦学习的基本框架、挑战、技术和应用，适合作为综述性的阅读材料。

2. 隐私保护与安全性

联邦学习的一个重要目标是确保数据的隐私和安全，这一领域的研究为其提供了理论基础和技术手段。

• Bonawitz, K., et al. (2017). “Practical Secure Aggregation for Federated Learning on User-Held Data” arXiv

  • 讨论了如何在联邦学习中实现安全聚合（Secure Aggregation），即确保服务器无法知道单个客户端的模型更新内容，以保护用户隐私。

• Geyer, R. C., Klein, T., & Nabi, M. (2017). “Differentially Private Federated Learning: A Client Level Perspective” arXiv

  • 探讨了如何将差分隐私（Differential Privacy）应用于联邦学习中，以确保用户模型更新时的隐私。

• Zhao, Y., et al. (2018). “Federated Learning with Non-IID Data” arXiv

  • 讨论了在非独立同分布（non-IID）数据的情况下，如何在联邦学习中实现模型训练，这是实际应用中的重要挑战之一。

3. 优化与效率

联邦学习中的通信和计算效率问题是该领域的关键研究方向，许多研究尝试通过各种方法优化模型训练过程中的资源消耗。

• Li, X., et al. (2020). “Federated Optimization in Heterogeneous Networks” arXiv

  • 讨论了在客户端计算能力和网络资源异质性情况下如何进行联邦优化。

• Kairouz, P., et al. (2021). “Advances and Open Problems in Federated Learning” arXiv

  • 这篇论文对联邦学习的现状、挑战以及未来的研究方向进行了系统性综述，覆盖了通信效率、模型性能、隐私保护等多个方面。

• Chen, M., et al. (2020). “Joint Learning and Communication Optimization for Federated Learning over Wireless Networks” arXiv

  • 探讨了如何在无线网络环境下优化联邦学习中的学习效率和通信效率。

4. 系统实现与工具

要更好地理解联邦学习在实际中的应用和系统架构，可以参考一些开源框架和实际实现案例。

• Google AI. “Federated Learning for Mobile Keyboard Prediction” Blog Post

  • 这是联邦学习最早的实际应用之一，讲述了Google如何使用联邦学习提升手机键盘的预测能力。

• TensorFlow Federated (TFF): GitHub

  • TensorFlow Federated是Google推出的一个开源框架，用于实现联邦学习的系统实验。通过阅读其文档，可以深入理解联邦学习的具体实现细节。

5. 联邦学习在各领域的应用

联邦学习在诸多行业中都具有广泛的应用，了解这些应用有助于扩展对联邦学习实际意义的认识。

• Rieke, N., et al. (2020). “The Future of Digital Health with Federated Learning” arXiv

  • 探讨了联邦学习在医疗健康领域的应用，特别是在跨医院数据无法集中共享的情况下如何训练模型。

• Hard, A., et al. (2019). “Federated Learning for Mobile Keyboard Prediction” arXiv

  • 描述了联邦学习在智能手机上如何用于改善键盘输入的预测性能。

6. 联邦学习的挑战与未来研究方向

对于未来的研究，联邦学习还面临许多挑战，比如系统异质性、模型性能与隐私保护的平衡等。

• Wang, J., et al. (2021). “Federated Learning: Challenges, Methods, and Future Directions” arXiv
  • 这篇论文对联邦学习面临的主要挑战进行了分析，如数据不均衡、通信成本、模型性能等，并提出了一些未来的研究方向。

联邦学习论文阅读图谱总结：

1. 基础理论：先了解联邦学习的基本框架和经典算法。
2. 隐私与安全：深入研究数据隐私保护和安全机制。
3. 优化与效率：关注如何优化联邦学习中的通信与计算。
4. 系统实现：通过工具和实际案例理解系统实现细节。
5. 应用领域：了解联邦学习在不同领域的实际应用。
6. 挑战与未来方向：展望联邦学习的未来挑战和潜在研究方向。

通过这个图谱，你可以系统地了解联邦学习的关键领域，并逐步深入到各个具体问题的解决方法与研究前沿。

• Reference
• Types
  • Supervised learning
  • Unsupervised learning
  • Reinforcement learning
    • Action space
    • Policy
    • Episode
    • Horizon
    • Return
    • Value function
    • Q function

Reference

Deep-Reinforcement-Learning-With-Python

Types

---

Supervised learning

In supervised learning, the machine learns from training data. The training data consists of a labeled pair of inputs and outputs. So, we train the model (agent) using the training data in such a way that the model can generalize its learning to new unseen data. It is called supervised learning because the training data acts as a supervisor, since it has a labeled pair of inputs and outputs, and it guides the model in learning the given task.

Regression

Quantitative response
predict a quantitative variable from a set of features

Classification

Categorical response
predict a categorical variable

---

Unsupervised learning

Similar to supervised learning, in unsupervised learning, we train the model (agent) based on the training data. But in the case of unsupervised learning, the training data does not contain any labels; that is, it consists of only inputs and not outputs. The goal of unsupervised learning is to determine hidden patterns in the input. There is a common misconception that RL is a kind of unsupervised learning, but it is not. In unsupervised learning, the model learns the hidden structure, whereas, in RL, the model learns by maximizing the reward.

---

Reinforcement learning

Action space

The set of all possible actions in the environment is called the action space. Thus, for this grid world environment, the action space will be [up, down, left, right]. We can categorize action spaces into two types:

• Discrete action space
  When our action space consists of actions that are discrete, then it is called a discrete action space. For instance, in the grid world environment, our action space consists of four discrete actions, which are up, down, left, right, and so it is called a discrete action space.
• Continuous action space
  When our action space consists of actions that are continuous, then it is called a continuous action space. For instance, let’s suppose we are training an agent to drive a car, then our action space will consist of several actions that have continuous values, such as the speed at which we need to drive the car, the number of degrees we need to rotate the wheel, and so on. In cases where our action space consists of actions that are continuous, it is called a continuous action space.

Policy

A policy defines the agent’s behavior in an environment. The policy tells the agent what action to perform in each state.
Over a series of iterations, the agent will learn a good policy that gives a positive reward.
The optimal policy tells the agent to perform the correct action in each state so that the agent can receive a good reward.

• Deterministic Policy
  deterministic policy tells the agent to perform a one particular action in a state. Thus, the deterministic policy maps the state to one particular action

• Stochastic Policy
  maps the state to a probability distribution over an action space.

  • Categorical policy
    when the action space is discrete
    uses categorical probability distribution over action space to select actions
  • Gaussian policy
    when our action space is continuous
    the stochastic policy uses Gaussian probability distribution over action space to select actions when the action space is continuous

Episode

The agent interacts with the environment by performing some action starting from the initial state and reach the final state. This agent-environment interaction starting from the initial state until the final state is called an episode. For instance, in the car racing video game, the agent plays the game by starting from the initial state (starting point of the race) and reach the final state (endpoint of the race). This is considered an episode. An episode is also often called trajectory (path taken by the agent)

• Episodic task
  As the name suggests episodic task is the one that has the terminal state. That is, episodic tasks are basically tasks made up of episodes and thus they have a terminal state. Example: Car racing game.
• Continuous task
  Unlike episodic tasks, continuous tasks do not contain any episodes and so they don’t have any terminal state. For example, a personal assistance robot does not have a terminal state.

Horizon

Horizon is the time step until which the agent interacts with the environment. We can classify the horizon into two:

• Finite horizon
  If the agent environment interaction stops at a particular time step then it is called finite Horizon. For instance, in the episodic tasks agent interacts with the environment starting from the initial state at time step t =0 and reach the final state at a time step T. Since the agent environment interaction stops at the time step T, it is considered a finite horizon.
• Infinite horizon
  If the agent environment interaction never stops then it is called an infinite horizon. For instance, we learned that the continuous task does not have any terminal states, so the agent environment interaction will never stop in the continuous task and so it is considered an infinite horizon.

Return

Return is the sum of rewards received by the agent in an episode.

Value function

Value function or the value of the state is the expected return that the agent would get starting from the state $s$ following the policy $\pi$

Q function

implies the expected return agent would obtain starting from the state $s$ and an action $a$ following the policy $\pi$.