肿瘤微环境中细胞间通讯（CCC）的分析有助于揭示癌症进展和药物耐受的潜在机制。目前，单细胞RNA-Seq数据大规模可用，提供了前所未有的预测细胞间通讯的机会。基于已知分子相互作用（如配体、受体和细胞外基质）推断细胞间通讯方面已经取得了许多成就和应用。但是，先验信息并不充分，只涉及细胞间通讯的一小部分，因此会产生许多假阳性或假阴性结果。为此，我们提出基于改进的层次变分自编码器（HiVAE）的模型，充分利用单细胞RNA-seq 数据，自动估计 CCC。具体而言，HiVAE 模型用于分别学习已知配体-受体基因和单细胞RNA-seq数据中所有基因的细胞潜在表达，进而进行级联集成。随后，利用转移熵衡量基于学习表示的两个细胞之间信息流的传递，这被视为有向的通讯关系。实验在人类皮肤疾病数据集和黑色素瘤数据集的单细胞RNA-seq数据上进行。结果表明，HiVAE 模型在学习细胞表示方面是有效的，而转移熵可用于估计细胞类型之间的通讯得分。 © 作者 2023 年。由牛津大学出版社出版。保留所有权利。请发送电子邮件至：journals.permissions@oup.com 以获取权限。

Analysis of cell-cell communication (CCC) in the tumor micro-environment helps decipher the underlying mechanism of cancer progression and drug tolerance. Currently, single-cell RNA-Seq data are available on a large scale, providing an unprecedented opportunity to predict cellular communications. There have been many achievements and applications in inferring cell-cell communication based on the known interactions between molecules, such as ligands, receptors and extracellular matrix. However, the prior information is not quite adequate and only involves a fraction of cellular communications, producing many false-positive or false-negative results. To this end, we propose an improved hierarchical variational autoencoder (HiVAE) based model to fully use single-cell RNA-seq data for automatically estimating CCC. Specifically, the HiVAE model is used to learn the potential representation of cells on known ligand-receptor genes and all genes in single-cell RNA-seq data, respectively, which are then utilized for cascade integration. Subsequently, transfer entropy is employed to measure the transmission of information flow between two cells based on the learned representations, which are regarded as directed communication relationships. Experiments are conducted on single-cell RNA-seq data of the human skin disease dataset and the melanoma dataset, respectively. Results show that the HiVAE model is effective in learning cell representations, and transfer entropy could be used to estimate the communication scores between cell types.© The Author(s) 2023. Published by Oxford University Press. All rights reserved. For Permissions, please email: journals.permissions@oup.com.