So stop doubting yourself and write it. Finish it. Ship it.
Even if no one reads it except you and your close family and friends, think about this.
You did it for the few people who matter—truly matter in your life.
And in that way you gave them love. Your children. People who will have history holding in their hands—your history.
You don't know what kind of impact that will have on them. You don't know if that one sentence you don't even remember writing will forever change their lives. You don't know if simply the act of writing and completing your book will give courage to one of them to write one too.
For that alone it's worth doing it. Worth writing it the best way you can.
Then you can rest easy. You gave it all you had.