Thursday, June 22, 2017

RetroArch Tone-mapping LUT Shader

Reshade has long had a shader, LUT.fx, that enables users/designers to do tonemapping and other color adjustments without touching any code. Instead, they can do all of their adjustments in an image editing program, such as Photoshop or GIMP, which many people are familiar with already. While my image-adjustment and color-mangler shaders can be used to accomplish those same tasks (Pokefan531 did a great job modifying them to produce his handheld color shaders, after all), they're awkward to work with, since an artist has to make all of his/her changes by twiddling esoteric values in the shader settings menu.

So, I decided to port the Reshade shader to RetroArch so our users could get in on the fun. I ran into some unexpected behavior with the direct port, though, and decided to adapt another similar shader instead. This one ended up having the same weird issue, which I think is related to undefined behavior, but I put a stupid workaround in the shader to mostly deal with it.

Anyway, here's how you use it:

First, take a screenshot that you want to use as your reference (I'm going to use the Sonic the Hedgehog title screen) and then take one of the passthrough palette textures that come with the shader (they're the png files located in reshade/shaders/LUT, named for their color depth). Then, in Photoshop or GIMP or Paint.NET or whatever, open your reference screenshot and paste the palette image down below it:
I find the easiest way to do this is to go to the image menu > canvas size and then anchor it from the top and increase the canvas Y-axis measurement by the height of the palette texture (in my example, I'm using the '16' texture, so I made the image 16 px taller). Paste the palette image in there, move it to the bottom-left corner and then merge the palette layer with the screenshot layer.

Next, do whatever it is you need to do to make the picture look like you want it. That includes brightness/contrast, hue/saturation, indexed color, different lossy colorspace conversions (such as by switching to CMYK colorspace and then back to RGB). I'm going to do a simple hue shift in my example because it's easy to spot:
Once you have it all set, we need to isolate the palette from the screenshot. I think the easiest way to do this is to go back to image > canvas size, anchor from the bottom left and enter the size of the palette (the width of the passthrough palettes is the height squared; I used the 16-bit palette, so it's 16 * 16 = 256 px). Save that image under a new name (I called mine 'hedgehog-palette.png'; if you're using a different palette depth from the default 16, it's probably a good idea to put that into the filename somewhere so you don't forget it) and then drop it into your reshade/shaders/LUT directory with the other palette images.

Now, open the LUT shader preset (cgp/glslp/slangp file) in a text editor and change the line that points to the palette (probably line 7, but YMMV):
SamplerLUT = shaders/LUT/16.png
becomes
SamplerLUT = shaders/LUT/hedgehog-palette.png
Save and exit and then fire up RetroArch, load a core and some content and then load up the shader. It should apply those same color transformations to the game image, like so:
If you used a different bit-depth palette from the default 16, your colors may look crazy and messed up at first, in which case you need to go back into the quick menu > preview shader changes and then change the "LUT Size" runtime parameter to match.

This shader is available from the online updater and/or git in GLSL, Cg and slang shader formats.

Monday, June 19, 2017

RetroArch shaders on Shovel Knight

I recently played through the awesome retro-styled platformer Shovel Knight for the first time and, while the pixel art is incredible, I kept thinking "I bet this would look even more amazing on a CRT." The game runs at a higher resolution on PC, so getting the low-res scanline-y look I crave wouldn't really be possible (or at least would be a significant hassle) through a 31 khz PC monitor. However, thanks to j_selby's out-of-the-blue Citra-libretro core, we can now run 3DS games via RetroArch--including Shovel Knight--and apply all sorts of fun shaders to the output!

Here are a couple of shots (click to embiggen):
The old favorite, cgwg's CRT-Geom
xBR-lvl2

artifact-colors; horizontal scaling is a little wonky

crtglow-gaussian

ntsc-320px-gaussian-scanlines

Thursday, June 1, 2017

New NTSC Shaders

NTSC simulation/emulation is a tough nut to crack. There's a lot of math involved, and the results are very dependent on games/content having the correct resolution or else the effect falls apart. Themaister's NTSC shader does a fantastic job with both 256- and 320-pixel-wide content, which covers most modern-ish retro consoles, including S/NES, Genesis, PS1 and N64, and it handles content of arbitrary resolutions pretty well. Nevertheless, it's always good to have variety, so I was excited to find some other shaders that included different takes on the NTSC signal problem.

Artifact Colors

This shader is based on Flyguy's awesome "Apple II-Like Artifact Colors" shadertoy and is the most impressive/magical of the shaders I'm going to cover here. Where this shader excels is in reproducing the NTSC "artifact colors" that certain old computers depended on before full-color interfaces were a thing. You can find some great explanations of the phenomenon at the 8088 mph demo writeup and this post at nerdlypleasures.

This splitscreen image demonstrates just how mind-boggling this effect can be, taking a 1-bit black and white image and ending up with bright colors:
RGB output on the left, composite artifact output on the right. Those colors are all generated by the signal modulation :O
And here's an animated gif that cycles between the typical limited-palette RGB output and the full-color artifact-color version (half RGB and half composite artifact back and forth):
Notice how the magenta and black stripes turn to a rich brown
In porting this shader, I tried to make runtime parameters out of as many variables as possible because it's such a cool thing to see how they affect the emergent colors in real-time. One such parameter is the F_COL variable, which can be used to basically have another color palette that you can switch to.

I also made a preset that pairs it with T. Lottes' CRT shader, which is perfectly suited to CGA content:
I called this one c64-monitor in the 'presets' subdirectory of the shader repos
I made the presets force a 640-pixel width, since that seems to be the sweet spot for this shader, which isn't surprising seeing as many of the games that relied on artifact coloring used a 640x200 mode. I'm not very familiar with any of the classic computer cores that could take advantage of this shader, but I'd love to hear comments from anyone who gives them a shot.

This preset also looks pretty darn good for general use, though it does a strange green/magenta strobe thing on Sonic's waterfalls and probably other pseudo-transparency:

CRTSim

This one from J. Kyle Pittman / PirateHearts is similar to the CRT effect in You Have To Win the Game:
It doesn't try to be accurate, it just looks nice and runs fast. While it does the whole bevy of old TV effects, including a really nice per-channel color persistence that can be used to reproduce the characteristic red smear of a failing CRT, my favorite part of it is the LUT-based NTSC effect. It uses a simple texture of diagonal colored lines:
This looks gray, but it's actually red, green and blue diagonal lines
and mixes it in with the main image based on differences in brightness between each pixel and its neighbors. For such a simple concept, the result is very convincing, and it even does a reasonable job of "passing" the NTSC crosstalk test-ROM (with certain settings):
Also of interest with this shader is the choice of permissive public domain licensing, so people can integrate it into their games and other programs without fear of licensing conflicts.

Both of these shaders are available in RetroArch's regular GLSL and slang/Vulkan formats. On-topic but not pictured are two NTSC shaders derived from MAME's NTSC shader implementation, one multipass and one single-pass. I haven't gotten them to work properly in GLSL format, only slang, and all of my Vulkan screenshots have their red and blue channels swapped for some reason right now, so I couldn't share any shots of them. You can get a taste from their shadertoy implementation, though.

Wednesday, April 5, 2017

Upscaling shaders for pre-rendered backgrounds

In the Playstation 1 / Nintendo 64 era, it was a common practice to pair low-poly 3D models with prerendered photo-realistic 2D backgrounds to give the illusion of a fully 3D-rendered game. This technique was used extensively in the Resident Evil series and Squaresoft RPGs (Final Fantasy 7, 8 and 9 and Chrono Cross), and when we emulate those games, we run into some issues related to the background.

If we use an emulator that supports increased internal resolution, the 3D polygon elements look sharp and crispy but the prerendered scenes are super-pixelated, and the contrast between the two can be jarring and distracting:
An unsettling juxtaposition.
To avoid that, you can stick to the native resolution and let everything look consistently pixelated and then use a full-screen post-processing shader remove some of the rough edges. Unfortunately, many of the algorithms we use for upscaling representational, cartoony pixel art don't always do a good job with the prerendered scenes, as the subtle gradients and natural, random dithering can trip up the pixel comparison and pattern detection methods that work so well in other applications:
Only a few pixels get smoothed; most are skipped
There are some good upscaling algorithms that are designed to deal with this sort of application, though, and we'll take a look at some of the ones available in shader form. We'll be comparing 4x upscales from two native-res captures from Final Fantasy 7:


ScaleFX-Hybrid

Sp00kyFox's scaleFX algorithm does a stellar job on cartoony stuff, and he tweaked the algorithm a bit to better handle small details. It has a rough, stipple texture on edges, unlike the regular version, and maintains details that the regular version smooths away:

It also handles gradients better than the regular scalefx, with reduced posterization.

NNEDI3

This algorithm, which utilizes a "predictor neural network architecture and local neighborhood pre-processing" (whatever that means), was recently ported to RetroArch's slang shader format by a fellow that goes by the name ZironZ. He hardcoded the huge swaths of weights derived from the neural network into the shader passes, which makes for relatively long compile times but respectable speeds once the task is done. He provides a handful of presets, some of which are much too demanding for my modest GPUs, but the basic preset runs in real-time and looks very nice:

This algorithm compares well with the popular waifu2x algorithm, which is also neural net-based but cannot run anywhere close to real-time (these shots are using the 'photo' preset with 'medium' denoising):
waifu2x, photo, medium denoise; no shader form available

waifu2x, photo, medium denoise; no shader form available
While all upscalers have a "signature" in the way their output looks (characteristic swirls or burrs on objects, etc.), much of waifu2x's magic mojo comes from its denoising routine, which gives images a surreal, painterly look when maxed out:
waifu2x, photo, max denoise; no shader form available
waifu2x, photo, max denoise; no shader form available

We can't reproduce that look exactly with real-time shaders, but we can get closer by adding a pre-pass of bilateral blur, which denoises the image by blending neighboring pixels that are close in color/intensity.

Fast-bilateral-NEDI

This combines Xin Li et al's NEDI algorithm (not to be confused with the aforementioned NNEDI3 from tritical) with a fast-bilateral pre-pass to get some of that smooth, surreal look at the cost of lost detail:

The fast-bilateral shader includes a runtime parameter to control the strength of the blending, and I modified it to go all the way up to 2.0 (default value is 0.4 with a max of 1.0), which looks nice but nearly wipes out the tile mosaic:

Fast-bilateral-super-xBR

Hyllian's xBR algorithm is now legendary in emulation circles for its ability to upscale pixel art, and the super-xBR variant is tuned to handle images and photos. The pre-pass of fast-bilateral cleans up some of the images' own built-in noise:

Again, cranking up the bilateral-blur to to 2.0 has an interesting if not useful effect:
And, just for fun/comparison, here's what waifu2x looks like in "artwork" mode (that is, a mode that throws away even more detail) with the denoising at maximum:
waifu2x, artwork, max denoise; no shader available
waifu2x, artwork, max denoise; no shader available
Hyllian's 3D detection shaders

Another alternative strategy involves using an initial shader pass to identify which elements are upscaled 3D models vs which ones are 2D textures. This lets the shader work on HUD elements and backgrounds without getting thrown off by the increased internal resolution:
super-2xbr-3d-6p-smoother (I think) 
jinc2-sharper-3d (I think)

Sunday, March 5, 2017

Raspberry Pi 240p Composite Output

Big news for emulation on Raspberry Pis of any generation: a new firmware update has enabled 240p output through the composite video-out port (a dedicated port on older models and a combo 3.5 mm jack on newer models).

From Raspbian and its derivatives, you can run from a terminal:
sudo rpi-update
and it will automatically install the new firmware. For Lakka, the updated firmware will make its way into the main release at some point, but in the meantime, you'll need to mount your SD/microSD card on another machine and then find the 'boot' directory (for Lakka, this will be located in the ~500 MB partition that holds the system files rather than the bigger partition that holds your games) and make a backup in case anything goes wrong (not likely, but better safe than sorry).

Next, you'll need to download the new firmware files from github (here's a direct link to a zipped download) and copy them into the boot directory, overwriting what's already there. I believe you only really need a subset of the files (bootcode.bin, fixup.dat, fixup_cd.dat, fixup_db.dat, fixup_x.dat, start.elf, start_cd.elf, start_db.elf and start_x.elf), so if it acts strangely, maybe try cutting down to just those files.

In your config.txt, you can use the sdtv_mode directive to change to the new modes (either NTSC or PAL):
sdtv_mode=16    # progressive NTSC
sdtv_mode=0x10  # progressive NTSC (using hex notation)
sdtv_mode=18    # progressive PAL
sdtv_mode=0x12  # progressive PAL (using hex notation)
Also, make sure your config.txt isn't forcing HDMI output via hdmi_force_hotplug=1 (comment it out, if so) and you might want to force composite output, just to be sure, by adding hdmi_ignore_hotplug=1.

Raspbian and derivatives can now also change modes at runtime using the tvservice program:
$ tvservice -c "NTSC 4:3" ; fbset -depth 8 ; fbset -depth 32; tvservice -s 
Powering on SDTV with explicit settings (mode:0 aspect1)state 0x40001 [NTSC 4:3], 720x480 @ 60.00Hz, interlaced
for interlaced mode or:
$ tvservice -c "NTSC 4:3 P" ; fbset -depth 8 ; fbset -depth 32; tvservice -s 
Powering on SDTV with explicit settings (mode: 16 aspect:1)state 0x40001 [NTSC 4:3], 720x480 $ 60.00Hz, progressive
for 240-line non-interlaced mode.

Now, you'll notice that the above line still says 720x480 and applications will still see a 480-line resolution (that is, emulators will want to run at 2x scale). That's because the firmware still acts like normal 720x480 (or 576 lines for PAL) and the composite encoder just displays only half the lines.

This is actually to our benefit because it allows us to easily go back and forth at runtime and has the potential for on-the-fly switching (though RetroArch/Lakka doesn't support that and probably never will). It also allows emulators to show interlaced content without freaking out (that is, a "true" 240p resolution would cut off half of the picture, while this setup will just blank out every other line, which is much less obtrusive).

I have some pics here but, hilariously, the one from the new firmware turned out the worst:
RPi 240p Composite

Analogue Nt - RGB-PPU YPbPr

RPi 480i Composite
If you look closely, you can see that the RPi 240p shot has gaps between each scanline (easier to see on the green waves of the hill than on Luigi), while the 480i shot does not. What the still image can't show is that the 240p image is rock-solid stable, while the 480i image flickers sickeningly. Either way, the RPi composite output obviously does proper NTSC encoding, so blending of dithering and so on happens automatically without needing a shader, which is nice. EDIT: the NTSC encoding is (obviously, in retrospect) using the RPi's full 720 horizontal res timing rather than a console's native timing, so effects that depend on NTSC artifacting may or may not look like you would expect. For example, here's what the "tvpassfail" NTSC test ROM *should* look like:
NTSC Pass/Fail test ROM with maister's NTSC 256 px shader
But this is what the RPi's composite output actually shows:
NTSC Pass/Fail test ROM via RPi composite output
There's still crosstalk, of course, but it's spread across more pixels, which breaks the test. Perhaps unsurprisingly, it looks closer to the 320 px NTSC shader output:
NTSC Pass/Fail test ROM with maister's 320 px shader
With this new firmware, the RPi3 is now arguably the best/easiest/cheapest way to do proper 240p emulation on a typical NTSC standard-def television. Previously, a softmodded Wii was the undisputed champ in this area, as it was the only way to do 240p without putting a series of adapters and converters in the mix. The Wii still has an advantage insofar as it can output a high-quality YPbPr signal (i.e., comparable to the Analogue Nt shot above) but the RPi3 has the ability to emulate more consoles, including great Playstation 1 emulation via PCSX-ReARMed and decent N64 emulation via the core formerly known as GLupeN64 (now renamed to just Mupen64Plus-libretro). The Pi2SCART add-on from ArcadeForge can get a cleaner, "pixel perfect" output from emulators on the RPi, but at ~$50, it costs significantly more than the RPi itself, costs as much as a used Wii and requires adapters to work with standard NTSC televisions. So, in my opinion, RPi is the way to go for cheap/easy "240p" emulation on a regular ol' U.S. TV, while the Wii is a step up in visual quality but more of a hassle (and fewer games/consoles available to emulate). Pi2SCART wins in both respects but also adds significant cost.

Friday, January 20, 2017

Pixel Art Upscaling Test Image

At the RetroArch/libretro forums, we get a lot of requests for shader previews because there are a lot of them and many people--particularly new users--get overwhelmed and don't know where to even start. So, we've batted around the idea for a while but there were some complications, namely that it's hard to get a good representative preview without managing giant, full-res images, and it's difficult to show of the effects of some shaders while still making the previews directly comparable to one another.

To handle these issues, I decided to make a small test image that contains many of the important features for a retro-focused upscaling algorithm to handle (this is enlarged; it's actually 64x64):
It was a fun exercise trying to pack as many tests as possible into this small canvas. There are two gradients--red-to-green and black-to-white--to test banding, several styles of text, some space invaders to test effects on isolated pixels surrounded by another color, the blended/antialiased doom-guy head, a variety of slopes that go from 1:1 up to 6 pixels of run to a single pixel of rise, the pesky circle-C copyright symbol, some parallel lines and a checkerboard to test de/dithering response/false-positives and a multicolored square that turns into a green-and-beige checkerboard in the presence of NTSC signal modeling, like this:
For the preview repo, I ran the test image through each shader preset at 8x integer scale (or slightly above that for the sharp-antialiased shaders). I don't have previews available for all of the presets yet at the time of this writing, but most of the popular ones are covered. These previews will hopefully make their way into RetroArch at some point, but in the meantime you can browse them at the repo.

Thursday, January 19, 2017

Shaders for Sharpest Pixels

As readers of my blog can probably guess, I usually favor scanline and CRT effects on my retro games but there are a lot of folks who prefer sharp-edged pixels. For many of these folks, integer scaling on the Y-axis and 1:1 pixel aspect ratio (PAR) with nearest neighbor (NN) sampling is gospel, but that can lead to weird display aspect ratios (DAR) on many systems, including S/NES, where the PAR is naturally non-square*. This issue is further exacerbated if you want to run your games at 4:3, which is the nominal aspect ratio of CRT displays and is probably most like what people saw playing retro games in their natural habitat.

If you get away from 1:1 PAR/integer with NN sampling, you end up with a lot of problems caused by uncertainty regarding where in the texel (that is, the texture's pixel) the upscaled pixel is actually coming from. This manifests as "shimmering" during what should be smooth scrolling and ugly, uneven pixel sizes on what should be smooth slopes:
The solution for this is to anti-alias the pixel edges, weighting the resulting pixels against their neighbors, and there are a handful of shaders that do precisely this. For my comparison shots, I've zoomed in much further than I usually do and I'm using an NES shot to accentuate the effects. I also limited the interpolation to the X-axis to make it easier to see the effect.

PIXELLATE

Originally written by Fes, this is the OG sharp anti-aliasing shader and has been ported everywhere. It takes four texture samples a small distance from the current pixel and averages them together.
Since the averaging is happening in gamma-adjusted colorspace, it favors dark pixels just a bit, so I added a runtime option to the slang port to do the interpolation in linear colorspace instead:
This avoids the darkening but also leads to "floating" pixels sometimes when an upscaled pixel is flanked by light pixels, as occurs behind/below Mario's ear. So, pros and cons /shrug.

AANN

jimbo1qaz took another stab at the same concept and wareya added some bits to allow for interpolating in "pseudo-perceptual" colorspace. It ends up being slightly darker than pixellate via gamma curve, surprisingly enough:

SHARP-BILINEAR

Themaister took a different approach with this one. It prescales the image to a desired integer scale (I added a default option to automatically choose the largest integer that would fit on the screen) using NN scaling and then use bilinear scaling to go the rest of the fractional scale factor. In this shader, there's no averaging of multiple samples, so the gamma status doesn't matter. It ends up landing somewhere between AANN and pixellate via gamma curve:
This shader is very lightweight because it only samples the texture once and then leverages the GPU's own scaling hardware, which works essentially for free. Note: this effect is the exact same concept as the "prescale" option that appears in a variety of emulators.

There are a few other shaders worth mentioning that I didn't include here, like Inigo Quilez' "improved texture filtering" (we just call it 'quilez' in the repos), which is significantly sharper than plain bilinear scaling while still providing evenly sized pixels, and aliaspider's scaling swiss army knife, GTU, which can produce a similar anti-aliasing effect when pushed to a very high internal resolution.

These shaders are available in Cg, GLSL/slang and quark/GLSL formats.

*This statement is hilariously contentious and causes some people to flip their shit.

Analytics Tracking Footer