Page 1 of 1

Hierarchical web resources

PostPosted: Wed Feb 22, 2012 11:17 am
by mmatula
First, let me thank you for this excellent piece of software!

I'm working on a plugin for some Czech channels which I will publish soon. For now I have Czech TV iVysilani working. However as far as I can see, Serviio currently supports only "flat" web resources - i.e. I give it a web resource URL and that is essentially a list (rather than a tree) of media. So, people could register a web resource for a given TV show and they would see all the episodes offered in the list. What I would really love to be able to implement is a plugin serving the complete site content. I.e. I'd like to be able for the plugin to serve a tree structure, so that I could offer something like:

- Alphabetical
-- A
-- B
--- <B-Movie 1>
--- <B-Series 1>
---- Episode 1
---- Episode 2
---- ....
-- C
...
- By Genre
-- Movies
--- <Movie 1>
--- <Movie 2>
--- ...
-- Series
--- <Series 1>
---- Episode 1
---- Episode 2
---- ...
-- Documentaries
-- Sports
-- Music
...
- Live Streams
-- CT1
-- CT2
-- CT24
-- CT4 - Sport

Is it possible already and I just did not get how I could implement that? If not, would it be possible to expose SPI that would allow me to do this?
Thanks in advance!

Re: Hierarchical web resources

PostPosted: Wed Feb 22, 2012 11:26 am
by will
Not at the moment sadly as it would be very useful for what your describing for other sites as well, but there is an open ticket for it https://bitbucket.org/xnejp03/serviio/i ... -to-create

Re: Hierarchical web resources

PostPosted: Wed Feb 22, 2012 12:22 pm
by zip
Will add it at some point. The problem is that Serviio will have to look at eah of these files with ffmpeg -i so if you returns the structure of the whole site it'll take a long time to index, lots of bandwith and probably raised eyebrows from the content provider.

The plugin should be intelligent enough to somehow use the maxItemsToRetrieve value - it works nicely in a flat structure, but in hierarchical it doesn't make much sense.

Re: Hierarchical web resources

PostPosted: Wed Feb 22, 2012 1:57 pm
by Illico
zip wrote:The plugin should be intelligent enough to somehow use the maxItemsToRetrieve value - it works nicely in a flat structure, but in hierarchical it doesn't make much sense.

For me the maxItemsToRetrieve
- in hierarchical stucture (with virtual folder for each category) may be a general variable for maximum Items to retrieve in "each end of tree".For example max 5 files in each category folder (So you'll have 5x NbFolder "ffmpeg -i" ).
- in a flat structure it may be the global maximum item to retreive. ex: All 5 files flat depending on plugin algo : 5 files from the first category, or 1 file for each first 5 categories.

Re: Hierarchical web resources

PostPosted: Wed Feb 22, 2012 9:37 pm
by mmatula
zip wrote:Will add it at some point. The problem is that Serviio will have to look at eah of these files with ffmpeg -i so if you returns the structure of the whole site it'll take a long time to index, lots of bandwith and probably raised eyebrows from the content provider.

The plugin should be intelligent enough to somehow use the maxItemsToRetrieve value - it works nicely in a flat structure, but in hierarchical it doesn't make much sense.


Hm. I see. Well in that case it does not seem workable - it would definitely be a *lot* of bandwidth (just crawling the metadata pages) for some sites. I was hoping for the web resources the work could be done lazily, since the metadata are provided by the plugin and do not need to be collected ahead of the time. Also the indexing is done on the provider side, so not sure what kind of indexing is needed by serviio. Am I being too naive?

Re: Hierarchical web resources

PostPosted: Thu Feb 23, 2012 12:07 am
by zip
Serviio has to know about the file type before playback so that it can tell the TV what file it's serving and if it should be transcoded or not. If all the files are exactly the same format we could get around it by giving each item the same cacheKey - in that case it's only run FFmpeg -i once and reuse the information for all the remaining items. It depends on the content source though and if the plugin dev can guarantee they only use a single file type (and the resolution is the same, for example).

Re: Hierarchical web resources

PostPosted: Thu Feb 23, 2012 7:57 am
by mmatula
Can the responsibility for determining the type of the stream be on the plugin side in this case? There could be a mandatory requirement for hierarchical plugin to provide this kind of information. I am guessing, most of the online TV services would publish just a few different types of streams and the plugin should be able to easily find out which type a particular video is using. The plugin guide can document what kind of info serviio needs and the plugin developer can run the FFmpeg for each different type of stream during the plugin development to get the exact info it needs to return to serviio depending on the stream type. Then during the runtime serviio could ask for the type information only at the time when TV requests content of a particular folder (i.e. only after I navigate to Alphabetical->B->B-Series 1 folder when it tries to load all the episodes). Since the plugin would not have to run FFmpeg on any video at that point, as it would know the stream type from some hints in the metadata provided by the provider and map that to the information it got from FFmpeg for that type in advance during the development time, this should not take long. Would that be possible, or is there a reason why this kind of info needs to be provided to serviio way before user/client requests the content of a particular folder?

Re: Hierarchical web resources

PostPosted: Thu Feb 23, 2012 9:21 am
by zip
this would be possible by using the cacheKey as I said before. So all the ontent with exactly the same technical metadata would share the same cacheKey - that way Serviio runs ffmpeg -i only for the cacheKeays that don't exists in the cache yet, instead for each file (which is normally the case as they all have unique cache keys).

Re: Hierarchical web resources

PostPosted: Thu Feb 23, 2012 10:02 am
by Illico
zip wrote:So all the ontent with exactly the same technical metadata would share the same cacheKey

What are informations attached to a cacheKey ?
Video/Audio Codec, resolution, frame rate... OK may be the same.
But what about Resource URI, Title, Thumbnails, there are different for each files ?

Re: Hierarchical web resources

PostPosted: Thu Feb 23, 2012 10:30 am
by zip
cacheKey is only used to lookup technical metadata (result of ffmpeg -i) of the content in the technical metadata cache. The other data is stored in a different (non persisted) cache

Re: Hierarchical web resources

PostPosted: Thu Feb 23, 2012 10:43 am
by Illico
zip wrote:cacheKey is only used to lookup technical metadata (result of ffmpeg -i) of the content in the technical metadata cache. The other data is stored in a different (non persisted) cache

Ok, make sense now.
Thanks