Abstract: Accurate identification of retinal vessels is essential for assisting doctors in screening early fundus diseases. Diabetes, hypertension, and cardiovascular disease can cause abnormalities of the retinal vascular structure. Retinal vessel segmentation maps can be quickly obtained using the automated retinal vessel segmentation technology, which saves time and cost of manually identifying retinal vessels. Aiming at the problem of incomplete and inaccurate extraction of fine retinal vessels, this paper explored the design of a multitask convolutional neural network and the topological relationship of retinal vessels. A cascaded retinal vessel segmentation network framework guided by a skeleton map was proposed. The auxiliary task of skeleton extraction was used to extract vessel centerlines, which could maximally preserve topological structure information. SAFF cascaded the two modules by remaining embedded between their feature layers. This process could effectively fuse the structural features with the vessel local features by learning pixel-wise fusion weight and thus enhancing the structural response of features in the vessel segmentation module. To obtain a complete skeleton map, the skeleton map extraction module introduced a graph-based regularization loss function for training. Compared with the latest vessel segmentation methods, the proposed approach wins the first place among the three public retinal image datasets. F1 metrics of the proposed method achieved 83.1%, 85.8%, and 82.0% on the DRIVE, STARE, and CHASEDB1 datasets, respectively. Ablation studies have shown that skeleton map-guided vessel segmentation is more effective, and graph-based regularization loss further improves accuracy of the retinal vessel segmentation compared to the vanilla network. Moreover, the framework generality is verified by replacing the skeleton map extraction and vessel segmentation modules with various convolutional networks.