Tempest ssh to guest intermittently fails, "GROWROOT: NOCHANGE: partition 1 is size 2078687. it cannot be grown" seen in guest console log
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| OpenStack-Gate |
New
|
Undecided
|
Unassigned | ||
Bug Description
Seen here for example:
2019-09-11 11:50:09.034110 | primary | sh: write error: No space left on device
2019-09-11 11:50:09.034181 | primary | Top of dropbear init script
2019-09-11 11:50:09.034251 | primary | Starting dropbear sshd: OK
2019-09-11 11:50:09.034376 | primary | GROWROOT: NOCHANGE: partition 1 is size 2078687. it cannot be grown
2019-09-11 11:50:09.034456 | primary | resize-rootfs already run per once
2019-09-11 11:50:09.034579 | primary | /run/cirros/
Note that this might not be the reason for the ssh failure into the guest, we could be hitting this in successful runs as well but only see this on ssh failure because that's when we dump the console log. Note that the network info was retrieved:
2019-09-11 11:50:30.311189 | primary | === network info ===
2019-09-11 11:50:30.311262 | primary | if-info: lo,up,127.0.0.1,8,,
2019-09-11 11:50:30.311377 | primary | if-info: eth0,up,
2019-09-11 11:50:30.311465 | primary | ip-route:default via 10.1.0.1 dev eth0
2019-09-11 11:50:30.311561 | primary | ip-route:
2019-09-11 11:50:30.311659 | primary | ip-route:
2019-09-11 11:50:30.311749 | primary | ip-route6:fe80::/64 dev eth0 metric 256
2019-09-11 11:50:30.311864 | primary | ip-route6:
2019-09-11 11:50:30.311952 | primary | ip-route6:ff00::/8 dev eth0 metric 256
2019-09-11 11:50:30.312068 | primary | ip-route6:
We should, however, attempt to get rid of that growroot error so it's not a red herring in debugging.
19 hits in 7 days, check and gate, all failures:
| description: | updated |
| Clark Boylan (cboylan) wrote | #1 |
| melanie witt (melwitt) wrote | #2 |
Note we only dump console logs during failures. It is possible that this happens on successful jobs too and isn't the cause of these failures (we just don't have that data).
That said I think fixing errors like this (the job in question should have a 1GB boot from volume disk) is likely to fix bugs and avoid distracting errors when debugging underlying issues.