I suspect, if the shadow can return the tennis ball like a 'yellow whiz', that the wind is going to blow the memories back too, in spite of the speaker's best efforts. It's a bit of a Sisyphean struggle to be 'with no mind', and I think your poem shows us such a struggle. Your gentle lines here reflect the 'quiet hours' and nostalgia for a simple memory very well. Sometimes when I read a poem, a good one, I am inspired to return to my own poetry; this one inspired me to write. :)