gnome-terminal writes excessively to /tmp (affecting SSD drives)

See also http://ubuntuforums.org/showthread.php?t=1830432

Vte stores the scrollback buffer's contents on disk, that's what you see there. See https://bugzilla.gnome.org/show_bug.cgi?id=631685, https://bugzilla.gnome.org/show_bug.cgi?id=664611 (and maybe a few other mainstream Gnome bugreports) about discussions and rationale why this was chosen. Quick summary: Running out of disk space is much less likely and effects the system less badly than running out of RAM, and your RAM would eventually make it into the disk (swap) anyways. And this is the only reasonable approach that allowed to implement unlimited scrollback.

Note that vte-0.40 will compress and encrypt these files, shrinking its data size (and hence extending SSD lifespan) by a factor of ~3-4x and also invalidating the privacy / data leakage concerns. This version will hopefully make it into Ubuntu 15.10 Whichever Wildanimal :)

As for SSD lifespan: I'm not really up to date, but a quick search suggests that it's not really an issue. E.g. this random article http://betanews.com/2014/12/05/modern-ssds-can-last-a-lifetime/ says you'd have to write 574 GB/day for 10 years, this is pretty much the maximum vte can produce anyways (on my Core i3 computer, cat'ing my 42 MB test file takes ~8.5 seconds, that's 420 GB per 85000 seconds), so you'd have to continuously produce output that drives the terminal to its maximum throughput for years.

That being said, I'm really not familiar with SSD lifetime issues so you might have better insight why my estimation above was wrong. In that case could you please support it with data, rather than just a feeling that it's writing too much?

> As for SSD lifespan: I'm not really up to date, but a quick search
> suggests that it's not really an issue.

It's not as simple as that; for example manufacturers like Samsung will disclaim warranties after writing more than 75 TB for smaller disks. See https://www.samsung.com/global/business/semiconductor/minisite/SSD/global/html/support/warranty.html

With the max speed of vte measured on my computer (as I mentioned above), producing 75 TB of scrollback data takes about half a year. (With the compression, it'll be 1.5-2 years or so). Combine that with the typical load average of your terminals in the long run (including those times when the app isn't even running) hopefully being waaaaay below 1%, I still don't think SSD lifetime is a valid real-life issue.

@Egmont wrote:
"Running out of disk space is much less likely and effects the system less badly than running out of RAM, and your RAM would eventually make it into the disk (swap) anyways."

But I run out of ram space very rarely. It's highly elegant to keep volatile data in RAM unless RAM is full.

--------------------------------------------------------------------------------------------------------------------------------------------------------------
@Egmont wrote:
"still don't think SSD lifetime is a valid real-life issue."
My particular desktop SSD is at 19% lifetime (as measured by the vendor through SMART code 202 Perc_Rated_Life_Used or 173 Wear_Leveling_Count). So yes, real people can wear out an SSD.

--------------------------------------------------------------------------------------------------------------------------------------------------------------
Aside from that writing to disk is dramatically more system overhead, compared to RAM.
--
Bash history, which is NOT kept from overwriting, is a better use of disk writes.

> But I run out of ram space very rarely.

Maybe because you don't allow large (let alone infinite) scrollback.

> It's highly elegant to keep volatile data in RAM unless RAM is full.

How do you think gnome-terminal should handle that? Suppose you have 4 GB of ram, shall it consume up to 3 GB and then starting using the disk? Sounds not only hard to implement, but also why should it take away RAM from other apps?

That being said, I was not the one coming up with this design, see the mentioned links for people who have more experience than me on topics like memory fragmentation and OOM killer. I just pointed you to their choice, which I also happen to agree with.

> My particular desktop SSD is at 19% lifetime

After using it for how long, which apps, etc.? How much do you think gnome-terminal contributed to this? As I said, I'm happy to let you know that vte 0.40 will save a lot by compressing the contents.

My DSLR is at 25% of its lifetime. 25k pictures, 100k promised by the manufacturer. And it was definitely more expensive than a large SSD :)

> Aside from that writing to disk is dramatically more system overhead, compared to RAM.

In case of g-t/vte, the goal was not to reduce this overhead as much as possible, but to come up with a design that allows you to have freaking huge (practically infinite) scrollback buffers. Even with writing to disk, vte performs reasonably well among terminal emulators when cat'ing a large file, and disk writing speed is not the bottleneck, not even with HDD. And cat'ing a large file is really not the typical use case.

If you really feel like hacking: download latest vte (0.39.x or git), and modify src/vtestream-file.h _file_write() so that a static variable (that is a variable shared across all vte instances) is incremented by len every time, and printed to some file every now and then. Compile and install, overwriting Ubuntu's version of the lib. After restarting gnome-terminal, this will tell you how much data your g-t writes during a day. I'd be interested in seeing that number!

Also see the upstream links with discussions about putting these temporary files under some tmpfs, its pros and cons.

The attachment "Count the amount of data written to /tmp" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

>How do you think gnome-terminal should handle that? Suppose you have 4 GB of ram, shall it consume up to 3 GB and then >starting using the disk? Sounds not only hard to implement, but also why should it take away RAM from other apps?

The Linux virtual memory system is highly adept at swapping out content that's not in active use,
keeping RAM for what's hot. The global system for this works quite well.

I have seen the OOM killer killing an innocent process, and other developers have expressed similar concerns too. That being said, I'm not against storing the scrollback contents in memory as a possibility, see the upstream bugreport for details. But I still can't see the SSD wear-out issue justified.

The oom killer will only come along if some epic long files have been
cat'ed to the terminal,
and infinite scrollback is selected. Speed, battery life, SSD life, energy
use and privacy are all reasons to have the common case be no disk writes.

> Speed

Encryption added about 10% to the required CPU usage. Now, with in-memory scrollback, would you keep it encrypted or not? If so, you'd keep wasting CPU. If not, someone will come along and complain that it's been written to disk (swap) unencrypted and it leaks data. But other apps also can write their data to swap, and perhaps the whole swap partition should be encrypted (which is not free either), but it's out of vte's control. I really don't know the answer to this question.

> battery life
> energy use

Apart from encryption's CPU usage, do you have further information about it? What's the ratio between 1 second of CPU vs. 0.01 second of HDD usage? (More concretely: between numbers that correspond to VTE processing a given amount of scrollback vs. writing that amount of data to HDD or SSD?)

> SSD life

See above, I still believe it's a non-issue.

> privacy

Solved in vte-0.40.

I perfectly understand your feelings towards storing the scrollback in memory. What could push this feature request higher up on my priority list is if you could also support it with evidence (data).

It still looks to me that the typical amount that vte adds to cpu/energy usage, ssd life etc. are way below to be worried about. And if we're worried about cpu/energy, we should probably start the optimization somewhere else.

For start, could you please apply the patch and let me know how much data your g-t processes during let's say an average week? I have also installed it (replacing /tmp with my home dir, in case I'll reboot) and will share my numbers.

Swap is the equivalent of main memory: the security issues are no
different. Nobody can complain about swap written to disk, as that's a
risk for any piece of main memory at any time. Why turn things so far
upside down to support an edge case (infinite scrollback left open for long
periods of time)?

If it was for me, I probably would have dropped infinite scrollback support. But apparently some people find it really useful.

We already have a certain code. It was designed with multiple criteria in mind, including infinite scrollback support, efficient storing on disk, compression, encryption and so on - storing in memory wasn't really among these. Tons of code have been built up around these assumptions.

The main question is not how/why we got here, but how to move on from here without reimplementing everything from scratch in a totally different way (and also: why move on at all). I mean re-writing a huge amount of code is also a possibility, but given the current resources for vte development it's unlikely to happen in the foreseeable future. We should accept where we are and find the simplest way to extend that to keep the data in memory. As I stated in the upstream bug, I believe the brand new memfd support of Linux seems to be the easiest way. (Or, as a quick workaround, you can already make it store the scrollback under a tmpfs-mounted directory. That's pretty much what memfd would give you, without requiring a tmpfs mount point.)

> Why turn things so far upside down to support an edge case (infinite scrollback left open for long periods of time)?

I can't see why the time should matter here.

We either properly support infinite scrollback (not as an edge case; without risking heavy swapping and OOM), or we don't. If we do, its scrollback has to reside on disk. I hope this is clear so far. The decision was made: we do support it. Most of the scrollback code was written with this decision in mind.

Have you installed newest vte with the patch I attached, or any other means of measuring its write activity? I really do want to see numbers before we move on. The numbers so far still give me an impression that even for extremely heavy terminal users gnome-terminal would still be responsible for probably less than 1%/year of SSD wear-out. I'm waiting for counter-proofs. (Could you please also describe in words what's your average terminal activity like? E.g. do you have tasks running in the background that continuously produce data, do you run big compilations that print a lot, etc.?)

After 1 week of usage, my gnome-terminals have written a total of 48 MB. Out of this, about 40 MB was cating my favorite test file 4 times for speed measurement purposes, and the remaining about 8 MB was the normal tasks I performed, including management of several remote servers via ssh, compilation of several applications over and over again, and a complete dist-upgrade to Vivid alpha (not in screen as it's done by default, but actually writing to the terminal's scrollback) and dist-upgrading again a few times.

Assuming 75 TB lifetime as mentioned in comment 3 (which is about 1/10 of the actual lifetime measured in this stress-test: http://techreport.com/review/27909/the-ssd-endurance-experiment-theyre-all-dead), I would have to keep this usage pattern for 30 years and then it'd cause 0.1% of the SSD wear-out.

Of course your usage patterns might differ significantly and result in magnitudes higher numbers; let me know if that's the case.

Note: resizing with rewrapping is an issue, if you do it with a large scrollback then it produces a lot of output, it writes quicker than upon normal operation. This might be a concern; a workaround could be to disable rewrapping on resize, using a relatively small scrollback buffer, or disabling opaque resize in the WM. (The data is in the ballpark of 5–10 bytes per line on each resize, i.e. probably less than 1 MB if you have 100.000 lines. The real problem is that most WMs send many resize events during a manual resize.)

In the mean time, I've done a proof of concept in the upstream bug to store the scrollback in memory, and I'm planning to make it a mainstream feature.

Another week of heavily using gnome-terminal for everyday work, another 15 MB of data written. (I won't measure this anymore.)