My feelings on selout have changed from back when it was the "hot new thing." I currently feel that solid colored outlines of various brightness with minimal internal AA tends to make for more attractive outlines. I'm probably thinking the same thing Indigo is. But really, skill and perception are the biggest factor in determining how good an outline looks regardless of the style. I didn't have very good perception of the technique I was trying to implement on the old blue board, and a lot of my sprites came out harsh and unattractive because I went overboard with selout.
I did and still do prefer selout to simply outlining or not outlining a large fighter/brawler type sprite. Capcom and their ilk used the technique very effectively, but they KNEW their sprites where always going to be overlayed on some sort of background. It's when you put those sprites on a light or arbitrary (i.e. transparent) background that things start looking ugly. Also, they didn't abuse the technique and use it on every single line in a sprite. Study some Alpha 3 sprites and you will see they have a lot of solid outlines all over the place. Like sarcasm, it is a subtle art to employ, go over board and you're just going to sound like a cynical asshat.
Here, the floor is the brightest area, and their feet end up looking the harshest. The upper body area, however fares much better. I'm noticing that Cody's back and Sodom's crescent blur nicely into the background.
And now when we throw them on black, they look fine all around.
Now throw them onto white, and the outlines in places are harsh...jarring. Sodom's entire left outline now has huge issues. And the sprites are getting that cardboard cutout effect.
Now on transparent. Does it look good to you? What if this was the default SMF theme? These sprites where not designed to be used with arbitrary background colors.
Now put them on a medium gray background. The sprites still have some crispness to the edges, but not to the point of being jagged. I have a strong suspicion that Capcom artists drew their sprites on a color close to this medium gray. Or perhaps they had someone who ran a bunch of color tests on their backgrounds and found a good average shade from them. Using this they could guage how dark to go in the outlined areas and avoid making them too harsh.
How dark you go with the pixels you're inserting on the edges makes a HUGE impact on the aesthetic value of the sprite. Good selout does not equal internal aliasing to black. This is the mistake I and many of us made when we where first figuring out the theory back in the swoo.net days. Check out this thread
which shows a prime example of this mistake. Actually this thread says pretty much everything I'm saying here, just not as explicitly.
Her pants hurt me on white. I can deal with the other variations.
Also, like most things in life, you can't get something for nothing:
These look great. The characters blend into the backgrounds and get darker around the edges, but it's not just a simple 1 pixel border everywhere. They are breaking the outline in places to get some extra bulk to the forms which, in my opinion, is a much more elegant solution than straight colored outlines.
But when they get thrown in front of bright sand or sky, things get harsher. Breaking the outline makes lines on Terry's thighs harsher than if they had been outlined, and the broken lines call out single dark pixels on Mai's legs. This is the price you pay, and obviously Capcom and SNK where willing to deal with it.
Someone mentioned Keneth Fejer, and how he employs the technique well. Yes, he does use it well, because he keeps it subtle, he understand how to use contrast, and he knows when to just use an outline:
Here's his original.
And here's a butchered version using ZOMGSELOUT!!!
When these professional pixel pushers where doing the art, they where assuming that the character would be on a semi-dark background ~90% of the time. Every now and then they might jump in front of a bright blue sky or a bunch of light red motion trails, but for the most part they would be surrounded by trees and buildings and such. I think they realized that by throwing a few (well placed) slightly darker pixels around the outlines, they'd be able to get that anti-aliased effect on the sprite edges, and it could work well for just about every environment. Keep in mind there probably was an army of artists working on different characters, and as a result there are some genuinely bad examples in these games where one of the guys working on a set of frames just didn't quite get it.
Capcom's artists developed their technique by making games, which is much different than posting a sprite on a forum every now and then for a hobby. Because of this, I think it's very difficult to really pick up the technique and use it effectively unless you are actually making a game. This is also why I think having a selout challenge with the requirement of a transparent background is fundamentally flawed. You gain no advantage using the technique this way. Matter of fact, your sprite almost certainly will look worse off because of it. A better idea would be to have the creator of the challenge make a few different backgrounds, like a forest, street corner, and a cave. Then have the contestants make a single sprite using the technique, and see if that sprite can work on each background.
*sigh* no that's just outlining, with color variation according to the lightsource, which is fine.
...Selout is putting broken outlines around the edges of the sprite which are darker than what they're outlining, with the express intent of having them not melt into a similarily-coloured background...
If you mean selout as defined by my crappy old tutorial, and the many bad attempts since it's inception, then yes, you are correct. But look at the above examples and you'll see they do respect the lightsource. Cody and Mary's hair, Terry's cap, Mai's arms--they avoid those darker pixels. But even then, they are breaking the outlines.
I believe it's an advanced technique, one that requires lots of practice to become competent at, and one that's really easy to screw up. Despite this, I think it can prove useful to everyone. At the very least, try it a few times to understand what it is and why you see it in so many games. Then you can integrate it, reject it, or whatever. Just don't scoff at it and call it worthless without any real proof.
P.S. I never really agreed that selout prevents sprites from getting lost in similarly colored backgrounds. Motion, the basic silhouette, and internal details play a much larger role than the outline. Like the hypothetical white ninja in a snow level--using dark pixels around his head and shoulders is just going to turn out ugly. It's better to either use outlines, no outlines, or just avoid the situation altogether and not put a white ninja on a snow level.