Super Smash Bros is seen as a joke and non-competitive to much of the traditional fighter base for several reasons.
1: The characters - As beloved as characters like Mario, Luigi, Kirby, Pikachu, and others are; they do not fit the standard perception of what constitutes a fighter. Though this begins to delve into the murky waters called "semantics", the general perception is not colorful and cheery. Look at Street Fighter, Vs. Series, King of Fighters, BlazBlue, and others. These characters are generally, though sometimes more loosely in regards to the Vs. series, vastly different in design than many Nintendo characters. They perceive colorful and sprightly to mean immaturity and childish, not something a "mature individuals" would be associated with. Many Nintendo characters are pinned with that "kiddy" label, which of itself is not untrue, but it is the connotation implied that is what many have as an issue. To sit here and follow this logic is to bring into discussion what even constitutes a fighter? And that is too much time and brain power wasted for triviality that changes nothing at the end of the night. The summation of my rambling is this, Nintendo characters are supposedky kiddy. So therefore the game must be kiddy too.
2: Variables - Most traditional fighters rely solely on technical skill such as spacing, timing, and gauge building. They don't have over arching variables that can affect the entire course of a match. What do I mean by this? Simply the Super Smash Bros series is more dynamic in nature than most fighters. While skill remains the primary basis of the game, there are more outside influences to add to this the equation. Items, stages, and random tripping are all components that can drastically change entire matches, regardless of a player's skill. The better player can lose to somebody highly under skilled simply because of a mistimed jump, an unseen Bob-Omb, stage hazard appearing at an inopportune time, random tripping (ughhhhhh), etc...
Granted, the competitive Smash community has made great strides in mitigating as much of these hazards and variables as possible (though Brawl made this harder to do with some of the mechanic decisions by the developers). Still, many competitive "elite" players scoff at the game because it even offers these features (which are entirely optional).
3: Xenophobia - Let's be frank here, the game isn't Street Fighter. It isn't made to be like Street Fighter and it won't ever be. Can it take cues and ideas from those games? Sure, absolutely. But anyone trying to make it "Nintendo Street Fighter" is going to live in a very sad state for a long time. Most gamers spend so much of their time devoted to perfecting techniques, stringing combos, judging frames, and so many other details. When something is out there radically different then that it is met with some skepticism. It doesn't fit the mold so therefore it doesn't count. It is like saying an orange isn't a fruit just because it isn't part of the apple family. It may not be an apple, but it's still a fruit regardless. It may be more lighthearted but that doesn't disqualify it as being a fighter. Traditional fighter? No. A fighter none the less.
4: It is designed for greater access - Super Smash Bros is successful for one reason beyond just the massive crossover it is. The formula works. We have had three games now, soon to be four, that all have built upon the foundation laid in Super Smash Bros 64. Everyone can jump right in and begin playing, so because of that the fanbase is generally dispersed between families and children. Whereas much of the fighting game community is segregated and inclusive to only a niche group. Most people jump in only to hop back out or go on to play something else. Because Smash Bros is different in offerimg an easier option for those wanting the fighter experience without much of the self-mutilating toil that will occur trying to dive deeper into the fighting community. This causes, again, the perception that the series is "simplistic" or "under developed" when many competiive Smash players can inform them that it carries quite a bit of depth. There is a very valid meta-game too it. Getting people to that point seems to be a much shorter route in Super Smash Bros.
This doesn't cover everything, only a few points that seem to be the most common to me. My points may not have been fully developed as they could have or on point. If not forgive me, it is a little late. That would explain my rambling.