1) It didn't have the sense of humor the Avengers had. It took itself too seriously.
2) For the most part, we got to know these characters in other movies and allow the audience to get comfortable with them. So when Avengers came along, they didn't have to redo all the introductions, they just dove straight into character development. In LXG, they had to introduce the characters and explain new concepts and then get into any character development.
3) Sean Connery bullied the director to make the movie mostly about Alan Quartermain. You had a director who couldn't stand up to the lead actor and the actor ended up taking control of the movie. This is always bad. See Blade III for another example for this.
Robert Downey Jr. tried to do the same thing and make the movie mostly about Iron Man. Joss Whedon did have to adapt his method of working with actors to make RDJ happy, but in the end Whedon was able to convince Downey to trust him as a director. (It also helps that Downey has more artistic integrity than Connery.)