I've done a lot of investigating and testing, and your thinking is close.
First this video does have a SAR of 2:3 and should play with DAR 32:27 as it specifies, not with 16:9 as I first suggested.
Note however that ffmpeg shrinks the width, rather than increasing the height as you are doing. Serviio should apply the same scale=iw*sar:ih,setsar=1 as does ffmpeg and output 853x720 as per the following from the recorded stream. You do not need to compute the absolute values for each ffmpeg command because ffmpeg will do the math and set the square pixels per the formula.
I hope you will change Serviio to use the formula.
Input #0, mpegts, from 'D:\Azureus Active Downloads\avc-1500-mp2-2-3.mpg':
Duration: 00:01:25.07, start: 1.389089, bitrate: 1581 kb/s
Program 1
Metadata:
service_name : Service01
service_provider: FFmpeg
Stream #0:0[0x100]: Video: h264 (High) ([27][0][0][0] / 0x001B), yuv420p, 12
80x720 [SAR 2:3 DAR 32:27], 25 fps, 25 tbr, 90k tbn, 50 tbc
Stream #0:1[0x101]: Audio: mp2 ([3][0][0][0] / 0x0003), 44100 Hz, stereo, s1
6, 128 kb/s
[src @ 044f5d00] w:1280 h:720 pixfmt:yuv420p tb:1/90000 sar:2/3 sws_param:
[Parsed_setsar_1 @ 051f3b80] a:1/1
1.37 A-V:-1348[Parsed_scale_0 @ 051f31c0] 27w:1280 h:720 fmt:yuv420p sar:2/3
-> w:853 h:720 fmt:yuv420p sar:2560/2559 flags:0x4
Secondly while this stream is encoded to play 32:27, it is actually a 16:9 stream. I don't know why a few SOPCAST streams have a SAR of 2:3 but Sopcast does play them correctly as 16:9 because it uses the equivalent of scale=iw*max(1,sar) rather than dar to size its playback window. In order to get Serviio to also play these streams as 16:9, I added mediabrowser online transcoding rules to get mpegts-h264(high) streams with non-square pixels (which singles out these streams) and transcode them to flv with DAR=16:9. Unfortunately this did not work and I have reported the details in this post: Erroneous parameters generated when using DAR=16:9
viewtopic.php?f=7&t=7495Note that uses a livestream from russia to illustrate the problem rather than a SOPCAST Stream.