In this section of the song, it seems clear that the narrator is using the memory of their loved one to bury their love, hence "The brightness of the sun, will give me just enough to bury my love, in the Moondust".
They use their past memories of their loved one to reconcile in their own thoughts so that the narrator can finally find peace in his loneliness.
As previously mentioned, the narrator's affection for their loved one exceeds all else, precisely the reason why the narrator is willing to "bury their love" to see that their love is happy and well.
I believe this song to be a message to all that suffer a dilemma in their love life; to 'give the world' to your loved in exchange for your own.
Only those that you love the most will you find yourself burying your emotions for; so that you can ensure their wellbeing in spite of your own.