“When I signed autographs after the show people would flat-out say, ‘Why are you with her?

Her music is s—.'”

“I wanted all of it out of my life.”

(EXCLUSIVE, Premium Rates Apply) LAS VEGAS - NOVEMBER 09: Singers Avril Lavigne and Deryck Whibley attend the grand opening of KOI Las Vegas at The Planet Hollywood Resort & Casino on November 09, 2007 in Las Vegas, Nevada.

Avril Lavigne and Deryck Whibley.Chris Polk/WireImage

Chris Polk/WireImage

Whibley said that the intense disapproval from Sum 41 fans weighed heavily on him during the relationship.

“I got a lot of flack for dating a pop star.

Sometimes at our shows, people would bring signs that read, ‘Avril Sucks!’

(EXCLUSIVE, Premium Rates Apply) Avril Lavigne and Deryck Whibley (Photo by Kevin Mazur/WireImage for People Magazine)

Avril Lavigne and Deryck Whibley.Kevin Mazur/WireImage

When I signed autographs after the show people would flat-out say, ‘Why are you with her?

Her music is s—.'”

Lavigne’s listeners weren’t much better.

“Her fans thought I was ugly and not good enough for her,” Whibley remembered.

“All of this made us retreat into our own relationship.

Us against the world, f— em all.”

The disapproval became so dramatic that Whibley feared for his safety with good reason.

They suggested that the band bring on professional security that dealt with this kind of stuff."

Whibley said that the media also treated him with immense cruelty.

Lavigne’ or ‘Avrils troll of a husband.’

Paparazzi would taunt me with Avrils success, trying to get me to react on camera.

As if my life could get any worse!"

Kevin Mazur/WireImage

The musicians’ relationship also faced resistance from Whibley’s Sum 41 bandmates.

That didn’t stop her from picking up the negative vibes, though.

“I just said, ‘Of course the guys like you!

Theyre just tough to get to know at first, but theyll come around.'”

“We all knew how much he hated Avril and that relationship,” the singer wrote.

“He had outright blamed it for a lot of the failure of theUnderclass Heroalbum.

Walking Disaster: My Life Through Heaven and Hellis out now.