Kernel panic - not syncing: stack-protector: Kernel stack is corrupted in: ... and a possible reason



  • Omega2+, from 0.1.5 b130 to 0.1.9 b150 several FirmWares, without dock / with Arduino Dock 2

    When your just unboxed Omega2+ is not working with its factory settings, there's no need immediately to think of a hardware error and/or no need to blame the unfortunate 3.3V power supply.
    You might only need to move a few meters away and hope this will help.

    This story is taking place in a flat of an apartment building built with reinforced concrete walls. There are lots of flats and lots of WiFi APs especially on CH1, CH6 and CH11. But there are a few completely empty CHs too.

    ‬My lovely Omega2... after the first steps...
    I've just started getting used to it when things have taken a very bad turn.
    I couldn't even start it. Whenever I tried, I got this message on the serial console (TX0):

    [   17.154192] EEPROM:Read from [factory] offset 0x0,length 0x400.
    [   20.579238]  ch1 bssid=e8:40:f2:f7:40:15
    [   20.583238]  ch1 bssid=9c:74:1a:ea:17:cc
    [   20.587215]  ch1 bssid=34:a2:a2:a2:ac:60
    [   20.591236]  ch1 bssid=78:54:2e:10:3a:3e
    [   20.595216]  ch2 bssid=00:27:19:d8:96:34
    [   20.599196]  ch10 bssid=04:27:58:33:44:60
    [   20.603279]  ch11 bssid=32:cd:a7:bb:90:88
    [   20.607345]  ch11 bssid=e8:de:27:d5:da:88
    [   20.611421] =====================================================
    [   20.617606] Channel 1 : Dirty = 322, False CCA = 0, Busy Time = 8646, Skip Channel = FALSE
    [   20.625999] Channel 2 : Dirty = 198, False CCA = 0, Busy Time = 9562, Skip Channel = FALSE
    [   20.634394] Channel 3 : Dirty = 144, False CCA = 0, Busy Time = 5270, Skip Channel = FALSE
    [   20.642788] Channel 4 : Dirty = 124, False CCA = 0, Busy Time = 3104, Skip Channel = FALSE
    [   20.651178] Channel 5 : Dirty = 104, False CCA = 0, Busy Time = 2560, Skip Channel = FALSE
    [   20.659572] Channel 6 : Dirty = 40, False CCA = 0, Busy Time = 2037, Skip Channel = FALSE
    [   20.667862] Channel 7 : Dirty = 64, False CCA = 0, Busy Time = 7737, Skip Channel = FALSE
    [   20.676167] Channel 8 : Dirty = 76, False CCA = 0, Busy Time = 22762, Skip Channel = FALSE
    [   20.684562] Channel 9 : Dirty = 88, False CCA = 0, Busy Time = 30515, Skip Channel = FALSE
    [   20.692957] Channel 10 : Dirty = 134, False CCA = 0, Busy Time = 28978, Skip Channel = FALSE
    [   20.701528] Channel 11 : Dirty = 172, False CCA = 0, Busy Time = 3165, Skip Channel = FALSE
    [   20.710000] =====================================================
    [   20.716180] Rule 1 CCA value : Min Dirtiness (Include extension channel) ==> Select Channel 6 
    [   20.724919] Min Dirty = 40
    [   20.727663] ExChannel = 0 , 0
    [   20.730676] BW        = 20
    [   20.900878] device eth0.1 entered promiscuous mode
    [   20.905758] device eth0 entered promiscuous mode
    [   20.940478] br-wlan: port 1(eth0.1) entered forwarding state
    [   20.946310] br-wlan: port 1(eth0.1) entered forwarding state
    [   22.939289] br-wlan: port 1(eth0.1) entered forwarding state
    [   24.809528] Kernel panic - not syncing: stack-protector: Kernel stack is corrupted in: 8701d944
    [   24.809528] 
    [   24.823293] Rebooting in 3 seconds..
    

    which isn't actually a Reboot, but a Halt/Poweroff state.

    It didn't take me long to figure out the problem was caused by the WiFi subsystem starting up.
    I only enabled the AP but the situation didn't improve.

    Lastly I noticed if a client successfully connects to the AP then the O2+ works without any problems, for as long as I want it to (of course there's no internet connection in this case).

    [   18.807205] EEPROM:Read from [factory] offset 0x0,length 0x400.
    [   22.138524]  ch1 bssid=e8:40:f2:f7:40:15
    [   22.142522]  ch1 bssid=9c:74:1a:ea:17:cc
    [   22.146576]  ch2 bssid=00:27:19:d8:96:34
    [   22.150588]  ch6 bssid=54:64:d9:fc:b0:86
    [   22.154570]  ch10 bssid=04:27:58:33:44:60
    [   22.158659]  ch11 bssid=04:27:58:34:0c:d0
    [   22.162728]  ch11 bssid=e8:de:27:d5:da:88
    [   22.166789] =====================================================
    [   22.172987] Channel 1 : Dirty = 182, False CCA = 0, Busy Time = 5687, Skip Channel = FALSE
    [   22.181380] Channel 2 : Dirty = 154, False CCA = 0, Busy Time = 7200, Skip Channel = FALSE
    [   22.189798] Channel 3 : Dirty = 112, False CCA = 0, Busy Time = 2908, Skip Channel = FALSE
    [   22.198205] Channel 4 : Dirty = 104, False CCA = 0, Busy Time = 1269, Skip Channel = FALSE
    [   22.206604] Channel 5 : Dirty = 96, False CCA = 0, Busy Time = 1126, Skip Channel = FALSE
    [   22.214905] Channel 6 : Dirty = 110, False CCA = 0, Busy Time = 2250, Skip Channel = FALSE
    [   22.223298] Channel 7 : Dirty = 96, False CCA = 0, Busy Time = 1234, Skip Channel = FALSE
    [   22.231604] Channel 8 : Dirty = 104, False CCA = 0, Busy Time = 2065, Skip Channel = FALSE
    [   22.240002] Channel 9 : Dirty = 112, False CCA = 0, Busy Time = 2292, Skip Channel = FALSE
    [   22.248403] Channel 10 : Dirty = 164, False CCA = 0, Busy Time = 2793, Skip Channel = FALSE
    [   22.256888] Channel 11 : Dirty = 172, False CCA = 0, Busy Time = 3890, Skip Channel = FALSE
    [   22.265359] =====================================================
    [   22.271551] Rule 1 CCA value : Min Dirtiness (Include extension channel) ==> Select Channel 5 
    [   22.280291] Min Dirty = 96
    [   22.283033] ExChannel = 0 , 0
    [   22.286037] BW        = 20
    [   22.680108] device eth0.1 entered promiscuous mode
    [   22.685069] device eth0 entered promiscuous mode
    [   22.717043] br-wlan: port 1(eth0.1) entered forwarding state
    [   22.722918] br-wlan: port 1(eth0.1) entered forwarding state
    [   24.718536] br-wlan: port 1(eth0.1) entered forwarding state
    [   29.342533] EEPROM:Read from [factory] offset 0x0,length 0x400.
    [   31.928473]  ch1 bssid=78:54:2e:10:3a:3e
    [   31.932471]  ch1 bssid=34:a2:a2:a2:ac:60
    [   31.936503]  ch1 bssid=9c:74:1a:ea:17:cc
    [   31.940508]  ch1 bssid=54:64:d9:fd:e5:3e
    [   31.944488]  ch1 bssid=e8:40:f2:f7:40:15
    [   31.948483]  ch2 bssid=00:27:19:d8:96:34
    [   31.952464]  ch6 bssid=54:64:d9:fc:b0:86
    [   31.956442]  ch9 bssid=c8:3a:35:4d:0b:70
    [   31.960438]  ch10 bssid=04:27:58:33:44:60
    [   31.964505]  ch11 bssid=04:27:58:34:0c:d0
    [   31.968672]  ch11 bssid=54:be:53:85:6b:12
    [   31.972738] =====================================================
    [   31.978942] Channel 1 : Dirty = 392, False CCA = 0, Busy Time = 5148, Skip Channel = FALSE
    [   31.987319] Channel 2 : Dirty = 250, False CCA = 0, Busy Time = 2994, Skip Channel = FALSE
    [   31.995712] Channel 3 : Dirty = 196, False CCA = 0, Busy Time = 1255, Skip Channel = FALSE
    [   32.004106] Channel 4 : Dirty = 176, False CCA = 0, Busy Time = 1233, Skip Channel = FALSE
    [   32.012500] Channel 5 : Dirty = 176, False CCA = 0, Busy Time = 1857, Skip Channel = FALSE
    [   32.020902] Channel 6 : Dirty = 134, False CCA = 0, Busy Time = 1358, Skip Channel = FALSE
    [   32.029297] Channel 7 : Dirty = 124, False CCA = 0, Busy Time = 34, Skip Channel = FALSE
    [   32.037498] Channel 8 : Dirty = 136, False CCA = 0, Busy Time = 37, Skip Channel = FALSE
    [   32.045715] Channel 9 : Dirty = 182, False CCA = 0, Busy Time = 1835, Skip Channel = FALSE
    [   32.054109] Channel 10 : Dirty = 186, False CCA = 0, Busy Time = 1116, Skip Channel = FALSE
    [   32.062590] Channel 11 : Dirty = 200, False CCA = 0, Busy Time = 3863, Skip Channel = FALSE
    [   32.071065] =====================================================
    [   32.077244] Rule 1 CCA value : Min Dirtiness (Include extension channel) ==> Select Channel 7 
    [   32.085983] Min Dirty = 124
    [   32.088832] ExChannel = 0 , 0
    [   32.091837] BW        = 20
    [   33.871986] device ra0 entered promiscuous mode
    [   34.362424] EEPROM:Read from [factory] offset 0x0,length 0x400.
    [   36.848468]  ch1 bssid=34:a2:a2:a2:ac:60
    [   36.852508]  ch1 bssid=9c:74:1a:ea:17:cc
    [   36.856488]  ch1 bssid=e8:40:f2:f7:40:15
    [   36.860495]  ch1 bssid=54:64:d9:fd:e5:3e
    [   36.864475]  ch2 bssid=00:27:19:d8:96:34
    [   36.868474]  ch6 bssid=54:64:d9:fc:b0:86
    [   36.872453]  ch9 bssid=c8:3a:35:4d:0b:70
    [   36.876432]  ch10 bssid=04:27:58:33:44:60
    [   36.880515]  ch11 bssid=04:27:58:34:0c:d0
    [   36.884577] =====================================================
    [   36.890773] Channel 1 : Dirty = 322, False CCA = 0, Busy Time = 4992, Skip Channel = FALSE
    [   36.899170] Channel 2 : Dirty = 218, False CCA = 0, Busy Time = 2652, Skip Channel = FALSE
    [   36.907549] Channel 3 : Dirty = 168, False CCA = 0, Busy Time = 525, Skip Channel = FALSE
    [   36.915854] Channel 4 : Dirty = 152, False CCA = 0, Busy Time = 72, Skip Channel = FALSE
    [   36.924071] Channel 5 : Dirty = 156, False CCA = 0, Busy Time = 36, Skip Channel = FALSE
    [   36.932288] Channel 6 : Dirty = 134, False CCA = 0, Busy Time = 1468, Skip Channel = FALSE
    [   36.940681] Channel 7 : Dirty = 104, False CCA = 0, Busy Time = 37, Skip Channel = FALSE
    [   36.948898] Channel 8 : Dirty = 112, False CCA = 0, Busy Time = 30, Skip Channel = FALSE
    [   36.957100] Channel 9 : Dirty = 154, False CCA = 0, Busy Time = 4389, Skip Channel = FALSE
    [   36.965493] Channel 10 : Dirty = 154, False CCA = 0, Busy Time = 1049, Skip Channel = FALSE
    [   36.973976] Channel 11 : Dirty = 130, False CCA = 0, Busy Time = 10088, Skip Channel = FALSE
    [   36.982539] =====================================================
    [   36.988731] Rule 1 CCA value : Min Dirtiness (Include extension channel) ==> Select Channel 7 
    [   36.997456] Min Dirty = 104
    [   37.000300] ExChannel = 0 , 0
    [   37.003305] BW        = 20
    [   37.199544] br-wlan: port 2(ra0) entered forwarding state
    [   37.205079] br-wlan: port 2(ra0) entered forwarding state
    [   39.198480] br-wlan: port 2(ra0) entered forwarding state
    [   61.191640] random: nonblocking pool is initialized
    
    
    
    BusyBox v1.25.1 () built-in shell (ash)
    
       ____       _             ____
      / __ \___  (_)__  ___    / __ \__ _  ___ ___ ____ _
     / /_/ / _ \/ / _ \/ _ \  / /_/ /  ' \/ -_) _ `/ _ `/
     \____/_//_/_/\___/_//_/  \____/_/_/_/\__/\_, /\_,_/
     W H A T  W I L L  Y O U  I N V E N T ? /___/
     -----------------------------------------------------
       Ω-ware: 0.1.9 b150
     -----------------------------------------------------
    root@Omega-5BE1:/# ifconfig
    apcli0    Link encap:Ethernet  HWaddr 42:A3:6B:00:5B:E1  
              inet6 addr: fe80::40a3:6bff:fe00:5be1/64 Scope:Link
              UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
              RX packets:0 errors:0 dropped:2 overruns:0 frame:0
              TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:1000 
              RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
    
    br-wlan   Link encap:Ethernet  HWaddr 40:A3:6B:C0:5B:E3  
              inet addr:192.168.3.1  Bcast:192.168.3.255  Mask:255.255.255.0
              inet6 addr: fd1d:48c4:7633::1/60 Scope:Global
              inet6 addr: fe80::42a3:6bff:fec0:5be3/64 Scope:Link
              UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
              RX packets:0 errors:0 dropped:0 overruns:0 frame:0
              TX packets:61 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:1000 
              RX bytes:0 (0.0 B)  TX bytes:11483 (11.2 KiB)
    
    eth0      Link encap:Ethernet  HWaddr 40:A3:6B:C0:5B:E3  
              inet6 addr: fe80::42a3:6bff:fec0:5be3/64 Scope:Link
              UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
              RX packets:0 errors:0 dropped:0 overruns:0 frame:0
              TX packets:115 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:1000 
              RX bytes:0 (0.0 B)  TX bytes:26412 (25.7 KiB)
              Interrupt:5 
    
    eth0.1    Link encap:Ethernet  HWaddr 40:A3:6B:C0:5B:E3  
              UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
              RX packets:0 errors:0 dropped:0 overruns:0 frame:0
              TX packets:61 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:1000 
              RX bytes:0 (0.0 B)  TX bytes:11483 (11.2 KiB)
    
    lo        Link encap:Local Loopback  
              inet addr:127.0.0.1  Mask:255.0.0.0
              inet6 addr: ::1/128 Scope:Host
              UP LOOPBACK RUNNING  MTU:65536  Metric:1
              RX packets:60 errors:0 dropped:0 overruns:0 frame:0
              TX packets:60 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:1 
              RX bytes:3588 (3.5 KiB)  TX bytes:3588 (3.5 KiB)
    
    ra0       Link encap:Ethernet  HWaddr 40:A3:6B:C0:5B:E1  
              inet6 addr: fe80::42a3:6bff:fec0:5be1/64 Scope:Link
              UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
              RX packets:5 errors:0 dropped:0 overruns:0 frame:0
              TX packets:2 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:1000 
              RX bytes:863 (863.0 B)  TX bytes:278 (278.0 B)
              Interrupt:6 
    

    It works and works all fine, up until I give the first command related to WiFi settings - for example:

    [   37.019560] br-wlan: port 2(ra0) entered forwarding state
    [   37.025098] br-wlan: port 2(ra0) entered forwarding state
    [   39.018470] br-wlan: port 2(ra0) entered forwarding state
    [   56.591081] random: nonblocking pool is initialized
    [  108.819888] jffs2: notice: (1741) jffs2_build_xattr_subsystem: complete building xattr subsystem, 0 of xdatum (0 unchecked, 0 orphan) and 0 of xref (0 dead, 0 orphan) found.
    
    
    
    BusyBox v1.25.1 () built-in shell (ash)
    
       ____       _             ____
      / __ \___  (_)__  ___    / __ \__ _  ___ ___ ____ _
     / /_/ / _ \/ / _ \/ _ \  / /_/ /  ' \/ -_) _ `/ _ `/
     \____/_//_/_/\___/_//_/  \____/_/_/_/\__/\_, /\_,_/
     W H A T  W I L L  Y O U  I N V E N T ? /___/
     -----------------------------------------------------
       Ω-ware: 0.1.9 b150
     -----------------------------------------------------
    root@Omega-5BE1:/# ifconfig
    apcli0    Link encap:Ethernet  HWaddr 42:A3:6B:00:5B:E1  
              inet6 addr: fe80::40a3:6bff:fe00:5be1/64 Scope:Link
              UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
              RX packets:0 errors:0 dropped:1 overruns:0 frame:0
              TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:1000 
              RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
    
    br-wlan   Link encap:Ethernet  HWaddr 40:A3:6B:C0:5B:E3  
              inet addr:192.168.3.1  Bcast:192.168.3.255  Mask:255.255.255.0
              inet6 addr: fd1d:48c4:7633::1/60 Scope:Global
              inet6 addr: fe80::42a3:6bff:fec0:5be3/64 Scope:Link
              UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
              RX packets:112 errors:0 dropped:0 overruns:0 frame:0
              TX packets:75 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:1000 
              RX bytes:16274 (15.8 KiB)  TX bytes:12419 (12.1 KiB)
    
    eth0      Link encap:Ethernet  HWaddr 40:A3:6B:C0:5B:E3  
              inet6 addr: fe80::42a3:6bff:fec0:5be3/64 Scope:Link
              UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
              RX packets:0 errors:0 dropped:0 overruns:0 frame:0
              TX packets:225 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:1000 
              RX bytes:0 (0.0 B)  TX bytes:46285 (45.2 KiB)
              Interrupt:5 
    
    eth0.1    Link encap:Ethernet  HWaddr 40:A3:6B:C0:5B:E3  
              UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
              RX packets:0 errors:0 dropped:0 overruns:0 frame:0
              TX packets:162 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:1000 
              RX bytes:0 (0.0 B)  TX bytes:28100 (27.4 KiB)
    
    lo        Link encap:Local Loopback  
              inet addr:127.0.0.1  Mask:255.0.0.0
              inet6 addr: ::1/128 Scope:Host
              UP LOOPBACK RUNNING  MTU:65536  Metric:1
              RX packets:100 errors:0 dropped:0 overruns:0 frame:0
              TX packets:100 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:1 
              RX bytes:5996 (5.8 KiB)  TX bytes:5996 (5.8 KiB)
    
    ra0       Link encap:Ethernet  HWaddr 40:A3:6B:C0:5B:E1  
              inet6 addr: fe80::42a3:6bff:fec0:5be1/64 Scope:Link
              UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
              RX packets:348 errors:215 dropped:0 overruns:0 frame:0
              TX packets:15 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:1000 
              RX bytes:51517 (50.3 KiB)  TX bytes:1371 (1.3 KiB)
              Interrupt:6 
    
    root@Omega-5BE1:/# cat /etc/config/wireless
    
    config wifi-device 'ra0'
    	option type 'ralink'
    	option mode '9'
    	option channel 'auto'
    	option txpower '100'
    	option ht '20'
    	option country 'US'
    	option disabled '0'
    
    config wifi-iface
    	option device 'ra0'
    	option network 'wlan'
    	option mode 'ap'
    	option encryption 'psk2'
    	option key '12345678'
    	option ApCliEnable '1'
    	option ApCliSsid 'yourwifissid'
    	option ApCliAuthMode 'WPA2PSK'
    	option ApCliEncrypType 'AES'
    	option ApCliPassWord 'yourpassword'
    	option ssid 'Omega-5BE1'
    
    root@Omega-5BE1:/# wifisetup add -ssid 'any_SSID' -encr WPA2PSK -password 'its_PASSWORD'
    > Restarting wifimanager for changes to take effect
    root@Omega-5BE1:/# [  418.998866] Kernel panic - not syncing: stack-protector: Kernel stack is corrupted in: 8771d944
    [  418.998866] 
    [  419.013078] Rebooting in 3 seconds..
    

    I have read everything (literally) in the manual and in the forum.
    I think I've tried all the settings and taken all the advice from the forum community (the ones related to the 3.3V power supply as well ;-) ) but nothing worked.

    I was very disappointed thinking everyone's O2+ is working apart from mine. Since I don't really have as much free time as I've already spent trying to make my O2+ work, I've finally decided to put aside my favourite toy.

    I wanted to give it one last try and turned it on when the miracle happened - it worked perfectly.

    The only thing that's changed by this time was that I've moved it to a different room of the flat, about 5 meters away from the previous place.
    I realized something must be interfering with the WiFi in the first room because when I move the O2+ back there, it goes straight to 'Kernel panic... ' and poweroff state again.

    Conclusions:

    Of course now I'm trying to find the source of the interference, but no luck so far.

    I think the Omega2+ should be able to manage these situations.
    At least the AP should be able to work fine if it's enabled.

    It would be amazing if the official Omega team could say something about this issue.
    Until then I am grateful for any advice, especially from those who were in a similar situation and managed to resolve it.
    What should I change on the wifi and how should I find what's causing the wifi interference.

    Many thanks for reading,
    gf.

    P.S.
    I think the absolute minimum would be the following if the kernel says:

    [ ...] Kernel panic - not syncing: stack-protector: Kernel stack is corrupted in: ...
    [ ...]
    [ ...] Rebooting in 3 seconds..

    then it should really do what it promised and Reboot.



  • These are interesting findings, they suggest there may be a bug in the wifi driver than can be triggered by variations in local circumstance.

    Unfortunately the wifi driver seems to be missing from the source release.

    However, the power possibility can't be entirely disregarded either - if it's on the edge, sometimes a combination of circumstances would cause failure, and other times it would barely survive.

    That reboot actually is a reset, not a shutdown, it is just then apparently getting stuck because the flash chip is still in a 4-byte address mode incompatible with the 3-byte addressing used to boot.



  • @Chris-Stratton
    Thanks Chris,

    I'm diligently investigating my problem.

    :mag_right: Ω :mag:



  • This post is deleted!


  • @Tiago-Couto said in Kernel panic - not syncing: stack-protector: Kernel stack is corrupted in: ... and a possible reason:

    exactly same problem here, someone has a solution?

    No, this is not the same as the problem you are reporting elsewhere. The subject of this thread is not failures in general, but a very specific one linked to wifi.

    Please stop making repetitive posts in multiple threads.



  • @Chris-Stratton

    These are interesting findings, they suggest there may be a bug in the wifi driver than can be triggered by variations in local circumstance.

    Hmmm I am wondering now if that is why out of the 3 APs I have both on G and N my Omega only sees one. I tried playing around with the channels encryption cyphers etc. But to no avail.

    Kinda frustrating.



  • Any news regarding this?


Log in to reply
 

Looks like your connection to Onion Community was lost, please wait while we try to reconnect.