Alpha Tester - Getting hideous reboots

[groovyclam]

I seem to be getting five different types of death in the kernel log:

May 25 03:55:02 (none) kernel: Tmk Assertion Failure:
May 25 03:55:02 (none) kernel: AssertValidPointer, line 2021 ()
May 25 03:55:02 (none) kernel: Tmk Fatal Error: Thread TmkTaskManager <146> died due to signal -2

or

May 25 04:36:13 (none) kernel: Tmk Assertion Failure:
May 25 04:36:13 (none) kernel: AssertValidPointer, line 2021 ()
May 25 04:36:13 (none) kernel: Tmk Fatal Error: Thread Scheduler <147> died due to signal -2

or

May 25 05:12:22 (none) kernel: Tmk Fatal Error: Thread Prioritizer <143> died due to signal 11
May 25 05:12:22 (none) kernel: NIP 0x1b96cf0 link 0x1b946e4 ctr 27618432
May 25 05:12:22 (none) kernel: R00 0x6c29202c R01 0x7fffedd0 R02 0x00000100 R03 0x6c292028
May 25 05:12:22 (none) kernel: R04 0x00000002 R05 0x00000001 R06 0x00000000 R07 0x7fffed50
May 25 05:12:22 (none) kernel: R08 0x00000000 R09 0x7fb73854 R10 0xdeadbeef R11 0x7ff2b1f0
May 25 05:12:22 (none) kernel: R12 0x42200092 R13 0x01da9208 R14 0x00000000 R15 0x00000000
May 25 05:12:22 (none) kernel: R16 0x018bdd8c R17 0x00000000 R18 0xffffffff R19 0xffffffff
May 25 05:12:22 (none) kernel: R20 0x01840000 R21 0x018d250c R22 0x00000000 R23 0x00000000
May 25 05:12:22 (none) kernel: R24 0x00000000 R25 0x01b4f780 R26 0x00000001 R27 0x00000002
May 25 05:12:22 (none) kernel: R28 0x00000000 R29 0x00000000 R30 0x7fb76f54 R31 0x00000000

or

May 25 07:57:05 (none) kernel: tcp_keepalive: call keepopen(0x808c9d20)
May 25 07:57:59 (none) kernel: About to check tasks for tty to shutdown
May 25 07:57:59 (none) kernel: This one's (159) goin' down
May 25 08:01:43 (none) kernel: Tmk Fatal Error: Thread TmkTaskManager <146> died due to signal 11
May 25 08:01:43 (none) kernel: NIP 0x1b96cf0 link 0x1b946e4 ctr 28875344
May 25 08:01:43 (none) kernel: R00 0x6c6c2904 R01 0x7fffee30 R02 0x00000100 R03 0x6c6c2900
May 25 08:01:43 (none) kernel: R04 0x00000002 R05 0x7fb4010c R06 0x00000000 R07 0x00000006
May 25 08:01:43 (none) kernel: R08 0xdeadbeef R09 0x7fb61934 R10 0xdeadbeef R11 0x7fb618b8
May 25 08:01:43 (none) kernel: R12 0x42200032 R13 0x01da9208 R14 0x00000000 R15 0x00000000
May 25 08:01:43 (none) kernel: R16 0x018bdd8c R17 0x00000000 R18 0x00000001 R19 0x7ffff1f8
May 25 08:01:43 (none) kernel: R20 0x00000000 R21 0x00000000 R22 0x00000000 R23 0x00000000
May 25 08:01:43 (none) kernel: R24 0x00000000 R25 0x01b4dcb0 R26 0x00000001 R27 0x00000002
May 25 08:01:43 (none) kernel: R28 0x00000000 R29 0x00000000 R30 0x7fbe5890 R31 0x00000000

or

May 25 08:39:07 (none) kernel: tcp_keepalive: call keepopen(0x80818d20)
May 25 08:39:42 (none) kernel: Tmk Assertion Failure:
May 25 08:39:42 (none) kernel: AssertValidPointer, line 2021 ()
May 25 08:39:42 (none) kernel: Tmk Fatal Error: Thread ContextMgr event handler <149> died due to signal -2

[gcobb]

Thanks very much for being willing to help. I think we have seen one other report of a Tivo which did Guided Setup seeing crashes and we need to work out what we can do to prevent it. So, until we can reproduce the problem on our test tivos your help is invaluable!

The crashes all seem to be corrupted pointers in the task scheduling data structures.

As Stuart suggested, that suggests to me that it is most likely a problem with the amount of work the Tivo is being asked to do (instead of something like a corrupted database or a disk error, for example). For example, a memory allocation failing and not being checked, or too many threads running at once exposing a bug where a lock is not being taken out when it should be. That seems consistent with the fact that you did not do a Clear and Delete Everything. And with the fact that a couple of the logs suggest that a scheduling-related activity was going on at the time of the crash. But this is all speculation of course -- the real cause could be quite different.

How many SPs and wishlists do you have set up? Does anyone know any way to tell (e.g. with a short TCL script) how many showings this Tivo has in its database?

Have you had a go at seeing if you can reproduce the crash predictably (for example, forcing a reschedule by changing the priority of a SP, or scrolling through the schedules by time)?

[groovyclam]

Have you had a go at seeing if you can reproduce the crash predictably (for example, forcing a reschedule by changing the priority of a SP, or scrolling through the schedules by time)?

I haven't found a reliable way yet of reproduction ( arf! ) although I get a "feeling" that calling up the live listings overlay ( on top of live TV ) and scrolling through the channel data can make it more prone to happen sooner rather than later.

The weird thing is it can go to a reboot *immediately* after a reboot as well.

How many SPs and wishlists do you have set up?

12 SPs, 13 people wishlists ( directors/actors ) and 9 search term wishlists -> all wishlists are on autorecord.

I recreated the SPs after the AltEPG import through the "normal" TiVo GUI method but I restored the wishlists via TiVoWeb from a backup I did whilst still using Tribune data.

I also have several manual recordings in the todo list for channels without AltEPG listings.

[healeydave]

Seems to be a number of old messages relating to "invoking rule 834: rebooting system".

Old threads seems to suggest memory problems, possibly config, not hardware.

Incidentally, this is the first report of its kind I have heard off, maybe it is related to excess data as stuart suggests!?, coupled with an extensively used system.
When I say extensive, I mean perhaps in cases where someone is making thorough use of it with numerous wish lists etc.
My Tivo's are not being worked by any stretch of the imagination and the ones I've tested in a similar way to your (only doing GS without C&D), haven't really been used past leaving on live tv and doing the odd download.

[FliesOpen]

I looked into Rule 834 many years ago. The rule states: "if any process (program) dies and it's been accessing MFS (the TiVo database), TiVo should reboot to maintain system integrity".

So it's not a clue, it's a symptom.

I'm intrigued by the number of "VERY LATE" messages - they suggest a hung process.

[groovyclam]

As I said above - not having guide data for a channel is actually very much outside the norm ( this isn't a slam at the people setting up the guide data! ).

Tribune used to fill the channels that they didn't give details for with those generic 2 hours programme blocks.

AltEPG not actually having ANY programme records in a channel they they have in a lineup is quite weird.

Is it an easy job to populate the channels that AltEPG has in its lineups, but doesn't do listings for, with generic 2 hours programmes for a couple of days so they at least have some records in the database ?

I can easily imagine some process looking for programme records for a channel getting its knickers in a twist when no programme records at all are pulled out.

-----

The only other thing that I think is weird in my case is putting people wishlists back in when those people might not have "people" records ( and might never have had them after I cleared down the old prog data and todo and redid guided setup from AltEPG ).

I am going to delete my people wishlists and see if that makes a difference where the reboots are concerned.

[groovyclam]

By the way - it's been up for 9 hours now just recording Tennis off EuroSport all day.

My connection for listings happened about 11pm last night and it rebooted several times through the night until about 9am this morning.

To me that suggests it freaks out whenever it has to process listings ( or work out the todo ).

[gcobb]

To me that suggests it freaks out whenever it has to process listings ( or work out the todo ).

Can you force a scheduling event (change an SP priority) and see if it happily runs through that? It will be interesting to see if it crashes before it gets its next dose of new listings.

[groovyclam]

I deleted my people wishlists yesterday evening and moved an SP up a few notches and the TiVo did indeed reboot before the listings download ( but not immediately ) about half an hour later.

It has also rebooted 10 ( ten!! ) times throughout the night after the listings download. The first reboot happened 2.5 hours after the listings download finished being processed. They then happened in quick succession.

It is now 7:15 am and it seems to have settled down and been up for 3 hours.

------

One thing that I have done in the past that I don't know whether or not makes a difference is that I was using the hack to record SKY radio channels via a secondary faux cable-box "SKY RADIO" lineup and had used the following tcl script to hack the RF input to be mapped onto the SCART. I have no idea how to "undo" this so I presume it is still in action:

#!/tvbin/tivosh

set db [dbopen]

RetryTransaction {
set setupObj [db $db open /Setup]
foreach signalSrc [dbobj $setupObj get Source] {
set sigType [dbobj $signalSrc get SignalType]
if {$sigType != 3} {
dbobj $signalSrc set Connector 7
}
}
}

dbclose $db

How can I undo this as I am now using the AltEPG "SKY + RF lineup" without anything plugged into the RF. Or will doing the AltEPG guided setup have sorted this out ?

[gcobb]

I don't know whether the Sky Radio stuff will have been undone but I would think so as I assume the Source stuff would have been recreated.

I would like to understand the main differences between your setup and the setup on my test box, on which I did GS and have not seen any crashes. Can you please try running the tcl script in the attached ZIP file and paste the results here? It counts how many Programs and Series you have on your system to see if it is very different from mine.

The script is called count-db.tcl and should be run using:

Code: Select all

tivosh count-db.tcl

It will take a few minutes but it should be reporting progress as it goes. You may see a few errors. errTmBackgroundHoldoff can be ignored. If you see other errors I would be interested in knowing what you see (not all the errors -- but the different error codes and whether there are a lot or just a few would be useful). My test tivo does report about 20 errors with code 0x0003001a so seeing errors doesn't necessarily indicate a real problem.

I am looking to see if I can do a similar count of Showing objects but I haven't found them in the database yet.

count-db.zip

(534 Bytes) Downloaded 289 times

[gcobb]

Which Programme Source (Aerial + Cable, etc) did you select when you did the Guided Setup? I want to try again to reproduce your setup.

[Automan]

My Tivo is a standard tivo image and I did a clear & delete everything....

Satellite and RF line-up picking free to air channels

Code: Select all

Visit TC.co.uk for upgrades and info !
Filesystem on / set to READONLY - type rw to make READ/WRITE
TiVo: {/var/hack} % tivosh count-db.tcl
Warning - could not open object 879695: can't open object (errDbNotFound)

Warning - could not open object 879696: can't open object (errDbNotFound)

Examined 500 records
Examined 1000 records
Examined 1500 records
Examined 2000 records
Examined 2500 records
Examined 3000 records
Examined 3500 records
Examined 4000 records
Examined 4500 records
Examined 5000 records
Examined 5500 records
Examined 6000 records
Examined 6500 records
Examined 7000 records
Examined 7500 records
Examined 8000 records
Examined 8500 records
Examined 9000 records
Examined 9500 records
Examined 10000 records
Examined 10500 records
Examined 11000 records
Examined 11500 records
Examined 12000 records
Examined 12500 records
Examined 13000 records
Examined 13500 records
Examined 14000 records
Examined 14500 records
Examined 15000 records
Examined 15500 records
Examined 16000 records
Examined 16500 records
Examined 17000 records
Examined 17500 records
Examined 18000 records
Examined 18500 records
Examined 19000 records
Examined 19500 records
Examined 20000 records
Examined 20500 records
Examined 21000 records
Examined 21500 records
Examined 22000 records
Examined 22500 records
Examined 23000 records
Examined 23500 records
SwModule - 9
StationDay - 2350
Headend - 6
SoundFile - 18
Image - 262
IrBlastData - 176
IrFormat - 190
ResourceGroup - 49
IrTivoFormat - 88
PostalCode - 2
LoopSet - 12
Font - 2
Program - 14221
Genre - 207
Station - 692
Component - 901
Series - 3730
ComponentCode - 578
DataSet - 6
SwSystem - 1
VideoClip - 74
Total - 23576
TiVo: {/var/hack} %
TiVo: {/var/hack} %


Automan.

[Automan]

Just rebooted again

Don't normally get the bogus line....

Code: Select all

May 26 20:03:56 (none) TmkMediaswitch::Trace[155]: BOGUS AUDIO LENGTH: 31104
May 26 20:03:56 (none) TmkMediaswitch::Trace[155]: BOGUS EVENT LENGTH: 552991 SID:0xe0
May 26 20:03:56 (none) TmkMediaswitch::Trace[155]: BOGUS AUDIO LENGTH: 63072
May 26 20:07:18 (none) TmkAssertionFailure[158]: (AssertValidPointer, line 2021 ())
May 26 20:07:18 (none) Scheduler[158]: Tmk Fatal Error: Thread Scheduler <158> died due to signal -2
May 26 20:07:18 (none) Scheduler[158]: Tmk Thread Backtrace: 1a344ec 1a32a44 1b96114 1b9645c 1b95fd4 1b96bfc 1b9b30c 1b96d20 1b946e4 1b49064 1aa4508 1b96d20 1b3c9f4 1b3d0dc 1b3de98 1b3dafc 1b3d460 1b3d240 1b47878 1b4d7bc 1b19e40 1b1b100 1b435f4 1b42ecc 1b42e1c 1b44a0c 1b16a90 1af7774 1af5a2c 1af89f8 1af9404 1bdd77c 1bde5c4 1bf42a8 1b5b5f8 1bdd1b0 1b91bb4 1c0644c 1c063bc 1c23c84 1c23df4 1c27364 1c28374 1c28420 1c27504 1c06298 1c06148 1bf71a4 1c239b0 1c2764c 1c28374 1c28420 1c27748 1c2379c 1c24388 1c2400c 1bf
May 26 20:07:18 (none) Scheduler[158]: Tmk Fatal Error: Thread died due to signal -2
May 26 20:07:18 (none) Scheduler[158]: Invoking rule 834: rebooting system


Automan.

[mikerr]

I would advise deselecting channels that have no guide data in channels I receive,
as altEPG doesn't populate it with "to be announced" 2 hour chunks that tivo/tribune used to - its just totally blank.

[edit]ah, I see groovyclam beat me to it above

[gcobb]

Automan, thanks.

I don't suppose anyone can run my script on a Tivo which is set up to use Satellite + RF with the normal Tivo service (and has never connected to the AltEPG)?