Here's what John Calvin says about Psalm 119:13. This selection is also very insightful and helpful.
Verse 13: With my lips I declare all the rules of your mouth.
David shows here the accord which ought to be betwixt God and us: to wit, that God has instructed us, and when we have heard that which has been said to us, that everyone should say “Amen,” and that there be a sweet harmony and accord betwixt him and us, without jar or contradiction.
Behold here the summary of what is meant in this verse. Note that David declares that he contents not himself alone with following of God and cleaving to him: but that he labors and desires as much as lies in him, to stir up his neighbors likewise, and to bring it so to pass, as that God might be served with a common accord, throughout the whole world. He then that will have a true zeal to honor everywhere, and seek by all means possible that he can, to redress those which are wickedly given, to stir up those which are cold and negligent, to strengthen those which are weak and feeble, to entertain those which are already in a good earnestness, and to make them more earnest.––And indeed, these are the very things which all the faithful seek to do.––For the Lord our God teaches us upon this condition, that every one of us think not only particularly of himself, but that we should also have a mutual care one of another, and whensoever we shall have this zeal, let us understand, that God has created the whole world upon this condition, that we all should be obedient to him.
But we must well notice the order that is here set down: For David does not begin with this sentence, “That with his lips he has told of all the Judgments of the Lord’s mouth.” For he has said before, that he "has hid them in his heart." The thing which David said to be hid in his heart, he soon after declares it with his mouth, wherein he shows that every one of us ought to begin with himself. When we mean to instruct and teach our neighbors, we must not say to them, “Go you before:” but, “Come next after me, or else hard by me:” and having care all to go the right way, we should all at once labor to go together to our God, to be conjoined to him in true faith. And I speak it to this end, because we shall see many who in this behalf can very well prattle and babble, and would seem to be the greatest doctors in the world. But what of all this? Let us in the meantime look into their lives and conversation, and we shall find in them nothing but infection and stench, nothing else but mocking of God in all they go about and do. We must therefore follow the order which David here holds: to wit, that the word of God be his as a Treasure in the bottom of our heart: and afterward when as we shall have this affection, let us endeavor ourselves to draw others also thereto, and go altogether with one accord to honor our Lord God; and that he which hath been better taught than any of his neighbors have been, let him confess that he is so much the more beholden and bound unto God, to do that which is here showed unto us by David.
For although we are not all Prophets as he was, yet for all that, this was spoken to us all in general, “Admonish ye one another.” Saint Paul (1 Thess. 5:14, Col. 3:16, and Heb. 3:13) speaks it to all the faithful and to all Christians. And that we should also know that this belongs and pertains to us: and chiefly as I have before said, that they which are most earnest, should lay forth the grace which was given unto them for the common building up of the Church, and instruction of their neighbors.
When I first decided to memorize Psalm 119, I thought it would be a good idea to have some good Puritan and Reformed books to read alongside so as to help me understand and apply it all in a better way. In John Calvin's Sermons on Psalm 119, he exposits every verse very profitably. I came to what he has to say about this well-known verse, and thought it was worth sharing.
Verse 11: I have stored up your word in my heart that I might not sin against you.
When David speaks after this manner, “I have hid your word or promise in my heart,” he well declares, that if we have but only a wandering knowledge, that it will not hold us in, but that the Devil will by and by win upon us to oppress us, with temptations, and in the end to cast us down headlong. What must we then do? It is not enough that we have been at church, and heard what hath been there said to us, and that every one of us has mumbled to himself some one thing or another, but the word of God must be settled in us and be hid in our heart, to wit, that it may there be residing and continually abiding: and to have received it with such an affection, as that it be as it were imprinted in us. If this be not so, sin will reign in us, for it hath natural habitation with us: For all our senses are wicked and corrupt, all our wills and desires are enemies to God, unless God’s word be well hidden in our hearts.
Moreover, we are to understand, that David did not here vaunt himself of his own power and strength, as though he were in admiration thereof: but the spirit of God speaking by his mouth, intends to give us a mirror, wherein we must be confirmed, to wit, that we must not have only our ears beaten with the Doctrine of salvation, and receive it in our brain: but that it should be hidden in our heart, to wit, that we should lay it up as in a Treasure house.
For this saying, to hide, signifies that David did not study to be ambitious to set forth himself, and to make a glorious show before men: but that he had God for a witness of that secret desire which was before him. He never looked to worldly creatures, but being content that he had so great a Treasure, he knew full well that God who had given it him, would so surely and safely guard it, as that it should not be laid open to Satan to be taken away.
Saint Paul, in 1 Timothy 1:19, also declares to us, that the chest wherein this treasure must be hid, is a good conscience. For it is said, that many being void of this good conscience, have lost also their faith, and have been robbed thereof. As if a man should forsake his goods and put them in an open place, without shutting of any door, it were an easy matter for thieves to come in and to steal and deprive and spoil him of all: Even so, if we leave at random to Satan the Treasure which GOD hath given unto us in his word, if it is not hidden in this good conscience, and in the very bottom of our heart as David here speaks, we shall be spoiled thereof.