Axion-like particles (ALPs) can form a network of cosmic strings and domain walls that survives after recombination and leads to anisotropic birefringence of the cosmic microwave background (CMB). In addition to studying cosmic strings, we clarify and emphasize how the formation of ALP-field domain walls impacts the cosmic birefringence signal; these observations provide a unique way of probing ALPs with masses in the range $3H_0\lesssim ma \lesssim 3H_\text{cmb}$. Using measurements of CMB birefringence from several telescopes, we find no evidence for axion-defect-induced anisotropic birefringence of the CMB. We extract constraints on the model parameters that include the ALP mass ma, ALP-photon coupling $A\propto g_{a\gamma\gamma f_a}$, the domain wall number $N_\mathrm{dw}$, and parameters characterizing the abundance and size of defects in the string-wall network. Considering also recent evidence for isotropic CMB birefringence, we find it difficult to accommodate this with the non-detection of anisotropic birefringence under the assumption that the signal is generated by an ALP defect network.