The role of microstructural defects is critical for very-high cycle and ultra-high cycle fatigue (VHCF, UHCF). In high strength and purity steels for machine construction internal and surface penetrating inclusions are found to be common initiation sites, while for example in metal additive manufacturing prepared parts further pore and crack like defects are found to dominate the number of cycles to damage initiation and ultimately, to short crack growth and failure. In current work we present a fully damage coupled crystal plasticity based micromechanical model and methodology for evaluation of VHCF and UHCF behavior of metallic materials. To capture the damage accumulation process relevant for the respective micromechanisms of fatigue, we apply the approach to modeling of experimental test results where the microstructural models are created from data obtained by reconstructing fatigue damage initiation sites by way of focused ion-beam serial sectioning and x-ray tomography characterization. The results demonstrate what are the critical material features necessary to properly capture the underlying micromechanisms in defect containing metallic microstructures. Especially, the inclusion to microstructure interactions with respect to plastic slip and stress state in the adjacent metallic microstructure and the role of the interface imperfection are presented and discussed in detail. The approach yields results where the damage nucleation process can be captured by micromechanics and inferences with respect to VHCF and UHCF material performance can be made, also with respect to validation of the methodology. In addition to comparison of experimental and modeling results, coarse graining methods methods are discussed and addressed. The results provide a direct workflow how to develop, validate and exploit micromechanical methods to address fatigue associated engineering problems with far improved detail and accuracy of prediction than earlier.