Guilt and any other emotion that has the capacity to make you move away from God is not of God. Adam and Eve experienced guilt.
And they heard the voice of the Lord God walking in the garden in the cool of the day: and Adam and his wife hid themselves from the presence of the Lord God amongst the trees of the garden. Genesis 3:8
The Holy Spirit within us will convict us of our wrong. But it is Satan who will try to wrack you with guilt to pull you away from God. The Holy Spirit convicts us to bring us to a place of repentance or regret for something wrong we've done. Adam and Eve were not sorry for what they did and could only offer excuses when God questioned them.
And the man said, The woman whom thou gavest to be with me, she gave me of the tree, and I did eat. And the Lord God said unto the woman, What is this that thou hast done? And the woman said, The serpent beguiled me, and I did eat. Genesis 3:12-13
But, when we stray against the will of God and go to Him according to His word and with godly sorrow and true repentance, He is faithful to forgive.
Now I rejoice, not that ye were made sorry, but that ye sorrowed to repentance: for ye were made sorry after a godly manner, that ye might receive damage by us in nothing. 2 Corinthians 7:9
King David, after the realization that he was the one that the prophet Nathan spoke of, even recognizing the emotion he felt when he didn't know it was him, was immediately convicted; not quilted, of the wrong he had committed.
King David went before God with godly sorrow. This prayer is after King David had sinned with Bathsheba and his subsequent plan to have her husband killed. Unlike his predecessor King Saul and unlike Adam and Eve, he was accountable for his actions and acknowledged to God in front of Nathan, that his sin was against God.
And David said unto Nathan, I have sinned against the Lord. And Nathan said unto David, The Lord also hath put away thy sin; thou shalt not die. II Samuel 12:13
Nathan, the prophet, shows him the errors of his way and delivers God's punishment for the deeds he had committed.
David had indeed sinned against God. Nathan tells him,
Howbeit, because by this deed thou hast given great occasion to the enemies of the LORD to blaspheme, the child also that is born unto thee shall surely die. II Samuel 12:14
The account does not share with us if King David was already aware of the ramifications of his sin or the condition of his heart when he set in motion his plan to cover Bathsheba's pregnancy. But, we do know, that once Nathan came to him, he did not hesitate to admit his sin. And even at that time, Nathan told him that God had put his sin behind him although God would still chasten him. King David's prayer is a model of how we humbly and with godly sorrow come before the Lord acknowledging our wrong; accepting full responsibility for our actions and asking God's forgiveness.
Have mercy on me, O God, according to your steadfast love; according to your abundant mercy, blot out my transgressions. Wash me thoroughly from my iniquity, and cleanse me from my sin! For I know my transgressions, and my sin is ever before me. Against you, you only, have I sinned and done what is evil in your sight, so that you may be justified in your words and blameless in your judgment.
David lost his infant son as chastisement for the sin he committed, but he continued to serve the Lord and was considered a man after God's own heart.
And when he had removed him, he raised up unto them David to be their king; to whom also he gave their testimony, and said, I have found David the son of Jesse, a man after mine own heart, which shall fulfil all my will. Acts 13:22
Sometimes, we don’t want the chastisement of God, but on any given day, consider His chastisement better rather than to have Him say, ‘Depart from me you worker of iniquity.’ His word lets us know that He does chasten those whom He loves; not to harm but for our betterment.
If ye endure chastening, God dealeth with you as with sons; for what son is he whom the father chasteneth not? But if ye be without chastisement, whereof all are partakers, then are ye bastards, and not sons. Hebrews 12:7-8
The Lord hath chastened me sore: but he hath not given me over unto death. Psalms 118:18
God tells us in His word that when we come to Him in sincerity that He is faithful to forgive us.
If we confess our sins, he is faithful and just to forgive us our sins, and to cleanse us from all unrighteousness. 1 John 1:9