If the poor beggar does not get it from you, Allah is going to provide him. Feel honoured that Allah honoured you, by allowing you to give him. What I am saying is give, say thank you to the one you gave and say Alhamdulillah to Allah.
You are not doing anybody but yourself a favour.
Also keep in mind, do not belittle any good deed:
لاَ تَحْقِرَنَّ مِنَ الْمَعْرُوفِ شَيْئًا
Ibn Umar radhiallahu 'anhuma used to carry sugar, which would possibly be interpreted today to mean like candy. They would say why would you give this in charity? He said because I love it and Allah said:
لَن تَنَالُوا الْبِرَّ حَتَّىٰ تُنفِقُوا مِمَّا تُحِبُّونَ.. آل عمران: ٩٢)
By no means shall you attain Al-Birr (piety, righteousness, etc; it means hei Ilah's Reward, i.e. Paradise), unless you spend (in Allah's Cause) of that which you love. (Surat Aali Imraan: 92)
[Gems of Ramdan]

@FadingShadow

Imam Ahmad (rahimahullah) said:

Friday is a Great Day, in which prayers are performed (salat al-jumu’ah) and (many) opportunities are given to the servants of Allah for supplication (on this day), so no one should neglect it (and one must be eager to seek out the benefits of this day).

[Masaa’il of Imam Ahmad (rahimahullah), vol. 2, pg. 98]

وَفَرْحَةٌ عِنْدَ لِقَاءِ رَبِّهِ
One who fasts will be overwhelmed with happiness when he meets Allah, because of the deed of fasting.

On The Judgment Day, there will be people who are drenched in their sweat from their sins (based on their level of sins). Some will have darkened faces.

آل عمران: ١٠٦)
يَوْمَ تَبْيَضُّ وُجُوة وَتَسْوَدُّ وُجُوةٌ...
n the Day (i.e. The Day of Resurrection) when some faces will become whit nd some faces will become black. (Surat Aali Imraan: 106)

Some people will be walking on their faces.
... وَنَحْشُرُهُمْ يَوْمَ الْقِيَامَةِ عَلَىٰ وُجُوهِهِمْ عُمْيًا وَبُكْمًا وَصُمًّا...
* الإسراء: ٩٧
nd We shall gather them together on the Day of Resurrection on their face blind, dumb and deaf. (Surat al-Israa': 97)

Some will be blind, even though in the life before they could see.
... وَنَحْشُرُهُ يَوْمَ الْقِيَامَةِ أَعْمَىٰ ««طه: ١٢٤)
We shall raise him up blind on the Day of Resurrection. (Surat Taha: 124)

Then comes those who fasted - they will be excited and overwhelmed with joy when they are meeting Allah. People are in terror and misery, yet those who fasted are bragging and they cannot wait to meet Allah because the Prophet sallallahu 'alayhi wa sallam said:
وَفَرْحَةٌ عِنْدَ لِقَاءِ رَبِّهِ

Someone who has not smelled the life of Ihsaan with Allah (which means worshipping Allah as if you see Him) may hear this, clap one hand over another in shock and say you people are crazy. I understand fasting and I will do it because I have to, but you are telling me I can enjoy it? Not many can understand that, because thirst and hunger are on one side and delight and enjoyment are on the other side. You are combining between opposites. It is like putting two
negative magnets or two positive magnets next to each other – they will not
stick.
[Gems of Ramadan]

@FadingShadow

Pleasure in worship is almost like a miracle - not many understand it and not many attain it. Only those who Allah blessed with the level of Ihsaan will inderstand it, comprehend it and apply it. Make your intention, your purpose and your aim the right cause - for Allah, and loving and pleasing Allah. Your aim here is your weapon, so march forth with your fasting and you will see delight and pleasure in your fasting.
[Gems of Ramadan]

@FadingShadow