Hey guys, apologies for being a bit absent lately. Life's been busy. Glad @cybercritic brought up the issue of precision.
@JoergZdarsky - what was the root cause of your circular artifacts? Also, how did you end up fixing the planet culling? I know we (in PM) discussed setting the mesh's bounds explicitly in Unity, but you weren't initially having success with that.
I think you're producing tons more triangles than necessary, so am glad you're going towards a Normal Map. I came up with a formula a couple years ago to calculate the LOD distances needed to maintain a worst-case 'screen-space density' of vertices, based upon the client's screen resolution and camera's field of view. It makes a lot of assumptions, but is what I use for a starting point:
Pmax = Target maximum # of pixels between vertices (regardless of LOD)
S = Vertex Spacing in World Units at highest LOD (smallest spacing)
W = Screen Width
H = Screen Height
FOVh = Horizontal Field of View
N = Level of Detail, runs from 0 to LODMax, with LOD 0 being the highest detail
D = Distance at which to transition to LOD N in order to maintain PMax
D = 2 * H *S / ( 2 * tan(.5 * FOVh*(H / W)) * Pmax * (2 ^ N))
So, as an example, if you wanted to maintain a resolution of less than 10 pixels between each vertex, and at LOD 0 your vertices were .1 world units apart (i.e. 10 cm):
(W = 1920, H = 1080, FOVh = 90):
LOD 0: D = 22.83468146
LOD 1: D = 45.66936292
LOD 2: D = 91.33872585
LOD 3: D = 182.6774517
LOD 4: D = 365.3549034
LOD 5: D = 730.7098068
LOD 6: D = 1461.419614
LOD 7: D = 2922.839227
LOD 8: D = 5845.678454
Please note there are tons of assumptions in this formula. i.e. you're always facing perpendicular to the patch's normal direction, slope is 0, etc. But the assumptions are made to cover the worst case scenario, so it will error on the side of giving better resolution.
The problem with calculating the vertex positions to an extremely fine detail, calculating their normals, and then "baking" that into the normal map is that you're running a massive number of noise computations - say 8 octaves of noise at a 512x512 resolution - that's 16x the density of your terrain mesh! This will be incredibly slow to calculate.
You might want to explore using a tiled higher-frequency noise (i.e. perhaps 16x higher) to generate a normal map which 'perturbs' the underlying normal data you've already calculated in Kernel 2. Your Normal Map will still be coupled to the terrain (due to using position as input to the noise generator), but will only need ~3 octaves to calculate each texel. You may need 2 calculations per texel for this perturbation vector (i.e. x perturbation and y perturbation).
Alternatively, you could generate several noise-based tileable normal map textures and perturb the calculated vertex normals with these. This would be even cheaper.
To be honest though, I'm still struggling with a good solution for this myself. Would be curious to hear from others in this thread!