No they werent related.
In the comics wolverine does have a relative named Dog Logan who is his brother and bully. Thats who X-men: origins of wolverine based their Sabertooth off of.
In the comics, sabertooth isnt his brother and they arent related in any way. Sabertooth just has a long standing grudge against wolverined and made a vow to torture him once a year.
I believe it was actually because wolverine killed one of sabertooth's family members, but i dont recall.
Anyways, not family but long standing rivals and they were both part of weapon x