Always strange to my eye see that white Cape soil, it's red up north. I know that it needs large quantities of compost and it takes years to get it productive but the raised beds struggle in our climate and even more in yours. The tree will certainly suck up all the moisture in the soil and apparently Yucca are also really bad that way.
What looks like a Karee seems to manage well: that's one that needs constant trimming or it gets out of hand