My immediate family has done this for years In fact, a usual conversation goes like: "don't you remember, I borrowed that from you..." "O yeah, but it's not mine, I borrowed it from ...."
OR
"I thought I loaned that to ....." "you did, and I borrowed it from her/him" "Do you need it back?" "Na, haven't needed it all these years, but if I need it I know where it is"
_________________________
"I was curious. Since I'm not a cat, that's not dangerous." - Greg House
In fact, a usual conversation goes like: "don't you remember, I borrowed that from you..." "O yeah, but it's not mine, I borrowed it from ...." 
