TMSID Preservation overkill or worth exploring?

[gcobb]

Stuart, I understand, but disagree. It isn't just my time, I hope to make the changeover easier for all UK tivo owners. I realise I might fail!

Baggsey,

Thanks for the info. I am aware of thetvdb and I do plan to integrate it into my query code. As you say, the problem is that it isn't good data for non-US shows. Thanks for the pointer to the lookup code example (I have looked at the API but I haven't written any code to access it yet so this will be useful).

I didn't notice (in my initial cursory look) an API to send updates to thetvdb -- do you know whether they would be interested in accepting updates from this UK data gathering?

Graham

[Tcm2007]

Please bear in mind that there will a permanent knock on effect in complexity going forward, in that it will be necessary to check if a TMSID is already allocated before allocating one to a new programme, which would not be needed if we just start again.

[poppadum]

Please bear in mind that there will a permanent knock on effect in complexity going forward, in that it will be necessary to check if a TMSID is already allocated before allocating one to a new programme, which would not be needed if we just start again.

But once the emulator has seen a series and allocated a TMSID, then it surely it will need to do the lookup when it sees an episode of that series again (at least that's the way that Wktivoguide seems to work - it maintains an episodes file which maps series titles to IDs). So I'm not clear why it makes much difference whether we start with an empty list of TMSIDs or pre-seed the list with TMSIDs we already know?

[gcobb]

Please bear in mind that there will a permanent knock on effect in complexity going forward, in that it will be necessary to check if a TMSID is already allocated before allocating one to a new programme, which would not be needed if we just start again.

Not sure I follow that there is any permanent complexity issue. If the programme is genuinely new and the user/process/system knows that (somehow?) then a new ID can be allocated in either scheme. If the programme is not new and has been broadcast since the changeover then it must have the same ID it had last time it was broadcast. So every programme has to be looked up to see if it has already been allocated an ID since the changeover (in fact, in my mind, not having an ID allocated already is the defintiion of new, isn't it?). What ongoing complexity is added by preloading that database (beyond the one time complexity of the preloading, of course)?

Am I missing something?

Graham

P.S. Thanks to a couple of volunteers, I have data on about 15000 TMSIDs so far. I am making a few more changes but will open up the data collection and export shortly to see if we can get some more (I haven't added thetvdb lookups yet, though).

[PPS. Didn't see poppadum's post until I had entered this one. Sorry for the duplication]

[Tcm2007]

OK, I've reviewed the complex set of Perl scripts which maintain the SQL database which holds all the data.

Changing the way those scripts work from a sequential allocation of ids to a look-up is non trivial. Worse, as the look up could only be based on the title text, it would significantly degrade the EPG - the OzTivo system goes to great lengths to ensure that similarly named programmes are only put on the same id if they are genuinely the same. Re-using the old TMSIDs would break that.

I don't think this one is going to fly, sorry.

[Tcm2007]

Please bear in mind that there will a permanent knock on effect in complexity going forward, in that it will be necessary to check if a TMSID is already allocated before allocating one to a new programme, which would not be needed if we just start again.

But once the emulator has seen a series and allocated a TMSID, then it surely it will need to do the lookup when it sees an episode of that series again (at least that's the way that Wktivoguide seems to work - it maintains an episodes file which maps series titles to IDs). So I'm not clear why it makes much difference whether we start with an empty list of TMSIDs or pre-seed the list with TMSIDs we already know?

The TMSid is not the same as the server database's ID; tmisid is just a field in the series and showing record, there's no looking up.

The main problem, beyond modifying the scripts to do that lookup, is that the lookup would be a crude text match on title, and none of the sophistications of the OzTivo scripts to refine beyond that and ensure new showings are matched to a series or one-off showing would work.

We'd therefore be permanently degrading the epg quality for the sake of a minor one-time inconvenience.

In my view it's not a good trade.

Not my decision though!

[Tcm2007]

Below is the data held in the showing table. If you are collecting sufficient data to populate ALL these fields then the pre-loading might work. If not, it would require substantial reengineering.

Code: Select all

#%     Column    |   Type   |              Modifiers
#% --------------+----------+----------------------------------------------
#%  id           | integer  | not null
#%              default nextval('public.newprogram_id_seq'::text)
#%  title        | text     | not null
#%  seriesid     | text     |
#%  description  | text     |
#%  matchpartial | boolean  | default false
#%  starrating   | smallint |
#%  year         | integer  |
#%  season       | integer  |
#%  episodenum   | integer  |
#%  advisory     | text     |
#%  castlist     | text     |
#%  director     | text     |
#%  tvrating     | text     |
#%  genre        | text     |
#%  tivoid       | integer  | not null
#%  modifiedby   | text     | default 'Unknown'::text
#%  modifiedon   | integer  |
#%  version      | integer  | default 1
#%  tivopseries  | integer  |
#%  searchtitle  | text     |
#%  notes        | text     |



tivoid is tmisid

And for a series:

Code: Select all

#%      Column      |  Type   |              Modifiers
#% -----------------+---------+------------------------------------------------
#%  id                | integer | not null
#%                                 default nextval('public.series_id_seq'::text)
#%  title             | text    | not null
#%  description       | text    |
#%  matchpartial      | boolean | default false
#%  castlist          | text    |
#%  director          | text    |
#%  genre             | text    |
#%  advisory          | text    |
#%  tvrating          | text    |
#%  tivoid            | integer | not null
#%  nexttivoepisode   | integer |
#%  modifiedby        | text    | default 'Unknown'::text
#%  modifiedon        | integer |
#%  version           | integer | default 1
#%  allowdups         | boolean | default false
#%  searchtitle       | text    |
#%  partialregexp     | text    |
#%  partialtype         | integer |
#%  epsasprogsstation | text    |
#%  notes             | text    |

[gcobb]

Useful lists, thanks. I will certainly update my data collection to keep as much information as I have (I was ignoring things that didn't seem necessary like descriptions and actor lists). That may keep some options open.

[irrelevant]

Code: Select all

#%  castlist     | text     |
#%  director     | text     |

Text ??!! Not a separate table of people, with a many-many relationship table?

But in any case ... My personal thoughts are that we want to make this as easy as possible for the user. If retaining the Tmsid would allow a TiVo to continue working without the need for a complete rebuild, or losing all the season-passes, then I think it's worth some added complexity.

Obviously it's hard to quantify but one of my justifications when doing personal projects is that if adding a feature takes an extra couple of days to implement, but saves me an hour a week thereafter, then I'm obviously going to see the saving after a few weeks. If that couple of days saved a couple of dozen people that hour, then it's a no-brainer.

(Case in point: I started this project to save me having to load a file into an emulator, take a snapshot, crop it, and save the GIF. Two minutes for one file. No problem at all. But what if you have many hundreds ....)

I don't know exactly how hard it would be to do the relevant lookups with any degree of reliability, but I do feel that the benefits - almost no work being needed on the TiVos themselves - should be seriously considered. Season passes are the big plus point on the TiVo, it would be a big turn-off for people to be told they had to re-create them, and might be enough to put them off persisting with the TiVo, especially given the work involved doing it through TV & remote.

Pre-populating the database with as much as we can would be enough, I think, just to cater for peoples' existing season passes. We don't necessarily need to find Tmsids on new programmes/series, as nobody will have them on their TiVo already. We can just allocate anew for those.

[Tcm2007]

Well I won't be implementing or using it, so I'm pretty neutral!

From what I've seen, the OzTivo system has grown into something that works well, but is therefore very complex. Simply making it work in the UK in the time period available is likely to be challenging, so adding extra features is likely to be low on the to do list.

Especially as this is an extra feature which has to be implemented before you start using and testing a system in anger; the historic database will have to be in place before you start EPG processing. The system does not allow for a partial database (just Title and TMSID say) with the rest of the data added later; it's all done at once so either you'd need to re-engineer that, or source a lot more data.

Plus as I say, the extra matching refinements won't work.

[poppadum]

Well I won't be implementing or using it, so I'm pretty neutral!

From what I've seen, the OzTivo system has grown into something that works well, but is therefore very complex. Simply making it work in the UK in the time period available is likely to be challenging, so adding extra features is likely to be low on the to do list.

That's a fair point.

Is it possible for those of us who are interested to have a look at the code to start getting our heads around how it works? I'm slightly concerned that time's ticking on. I've been working on XMLTV parsing of the Radio Times data and converting it to a tivo-esque format; it works quite nicely as an input to Wktivoguide to build slices, but it would be nice to start thinking about data gathering for the emulator.

[gcobb]

I have been reading through the code (the wonders of a long flight!) and I have three ideas which I think would work. Comments and thoughts are welcome. In descending order of my preference, here they are...

1) Add code to new_series (Series.pm), or possibly to its caller, to lookup the title (in a new and separate table) and, if it is found, use the TMSID associated with that title (instead of assigning a new tivoid) and mark it as used, so it won't be matched again.

This avoids any problems with the clever series matching stuff -- it would only be invoked when the code has already decided this is a new series, never seen before, and (by only using each TMSID once) it avoids accidentally merging two series. Of course, if there are two series with the same title it might assign the TMSID to the wrong series but there is not a lot we can do about that (and the same problem exists with some Tribune data today). It has no impact on anyone who does not have an SP for that TMSID.

The downside is that if this action is later undone (say the program is later manually reassigned to be a part of a previously existing series) the TMSID will have been marked as used and will never be assigned. But that is probably not worth worrying about.

2) Deal with the problem manually, using the crowdsourcing capabilities.

Unfortunately I can't find any code or web pages to do with the crowdsourcing (anyone know where they are?) so I don't know if there is an option to change the tivoid using that capability. If there is, then the lookup could be done as part of the crowdsourcing web page and a user could be given all the available information to manually decide whether to use the old TMSID.

In this case, a human being is making the decision and they are quite likely to guess correctly. I am not sure what impact (if any) there would be on people who had already downloaded the slice with the automatically assigned tivoid.

3) Deal with the issue entirely on the Tivo: create a script who's job is to watch for series appearing in the database which seem to match existing season passes and modify the TMSID for the season pass to use the ID for the new series.

This seems to suffer from the problem Stuart mentioned of bypassing all the clever series matching stuff and it cannot be overridden by crowdsourcing correcting the data. The advantage is that only users who wanted to preserve their SPs would be hit. But if a script exists to preserve SPs I would expect most people to be using it.

Graham

[tivofantastic]

Great to see people trying to work out a solution for preserving season passes...

If it can be done without any downsides then it has to be a good thing - like many people I have hundreds of season passes collected over years for programs that aren't currently being aired, and of course it would be great if these season passes could be ported.

However, if the solution is not fully compatible with the OzTivo way of doing things then personally I would do without. I think we stand to gain far more by staying closely aligned to the OzTivo software framework so that any further development efforts and improvement incorporated by the OzTivo team can be quickly and painlessly incorporated into the UKTivo setup (and vice versa).

Also, I think anything that impacts performance negatively is also to be avoided. The TiVo hardware is somewhat long in the tooth and not so powerful - processing additions to my season pass list or re-ordering my season passess already takes quite long enough (even with a CacheCard and 32MB of RAM installed on my TiVo). Even adding a new season pass can take around 30 seconds, which is long enough.

So, on balance, I am with tcm2007. Personally, knowing that tcm2007 has developed some of the cleverest hacks for the TiVo (e.g. EndPad, DailyMail, Sync Now Playing, Conflict Resolution between 2 networked TiVos) I am inclined to trust tcm2007 to make a good call on this one - especially as he doesn't appear to have a vested interest either as I think he has now moved away from using TiVo in favour of MMC.

Don't get me wrong - if it's possible to come up with a solution that doesn't impact either performance or our ability to stay fully aligned with the OzTivo system then go for it. But, if not, I am prepared to trade the loss of a few episodes of my old season passes (until I spot that they are back on air so allowing me to re-enter the season pass) in exchange for a better TiVo platform into the future. Just my vote and thoughts - thanks again to everyone trying to make the alternative EPG a reality.

[irrelevant]

I think option 1 sounds the best - it can be made to slot in without upsetting the existing code, and should, I think, work well. The matter of Tmsids not being released again on a reallocation can be addressed fairly easily, either by modifying the change routine to release the number again, or just having a standalone utility that cross-checks the database against the list of known programmes; that would need to be run occasionally, but would mean no further Oztivo code needs to be amended.

[healeydave]

I think option 1 would be acceptable too.

The OzTivo code was tweaked and refined many years ago and has been static for some time now that the system is live and has been running for several years.
It is my belief that Warren is unlikely to do any further development of the system (as it is) unless something significant happens

Also, the changes that Graham is proposing sound like they would even be cross compatible anyway (e.g IF,THEN, ELSE).

Rather than post the link to the TVGuide screens (where the Crowd Sourced Editing is possible), because you have to register etc. , I have grabbed some screen captures.