Ubuntu
openssh package

openssh: want hpn-ssh for 20x speed improvement!

Bug #162253 reported by Joe Harrington
176
This bug affects 30 people
Affects Status Importance Assigned to Milestone
openssh (Debian)
Won't Fix
Unknown
openssh (Ubuntu)
Won't Fix
Wishlist
Unassigned
Declined for Gutsy by Brian Murray
Declined for Hardy by Brian Murray
Declined for Intrepid by Colin Watson

Bug Description

This is a wishlist item: hpn-ssh would be a big data-transfer speed improvement for many Ubuntu users, particularly scientific organizations like universities and observatories that transfer large amounts of data over long-haul networks, and large commercial users. hpn-ssh is a patch on openssh that just makes the TCP window variable in length, overcoming a static (and too-small) window size that is in openssh. No modifications are made to the security aspects of the code. Measured transfer rates are improved over some nets by a factor of 20. It's in wide use already by banks, the US government, many grid computing centers, and security companies. Even if it's not patched into openssh, making an alternative version available would be a service to the community. Other distros have this, but synaptic shows me nothing for Ubuntu. Here are some sites:

http://www.psc.edu/networking/projects/hpn-ssh/papers/hpnssh-gridnets2007.pdf
http://www.psc.edu/networking/projects/hpn-ssh/

Thanks for listening,

--jh--

See original description
Revision history for this message
In , Florian Weimer (fw) wrote : Re: Bug#292932: openssh-server: The high-performance patch from PSC should be included as standard.

* Colm Buckley:

> The high-performance patches from PSC
> (http://www.psc.edu/networking/projects/hpn-ssh/) should be included as
> part of standard SSH; these patches make an *enormous* difference when
> transferring large quantities of data over a high-bandwidth network.

Why is this patch not included upstream? Because of the no-encryption
part with has "issues"?

Revision history for this message
In , Mark Nipper (nipsy) wrote : openssh-server: both patches should be included

Package: openssh-server
Version: 1:4.1p1-6
Followup-For: Bug #292932

 There shouldn't be any security issues with including at least
the performance patch to allow for scalable buffer sizes.

 But it would be nice if the hpn11-none cipher patch could also
be included as there is probably good need of it by most people who use
scp. I'd even argue for documenting it officially as it seems pretty
obvious which situations call for its use (and which don't).

 Just my two cents. I'm going to rebuild my local packages with
the patches at least. :) It would be nice to see all Debian users
benefit from these though.

-- System Information:
Debian Release: testing/unstable
  APT prefers unstable
  APT policy: (500, 'unstable'), (500, 'stable'), (1, 'experimental')
Architecture: i386 (i686)
Shell: /bin/sh linked to /bin/bash
Kernel: Linux 2.6.12.4
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)

Versions of packages openssh-server depends on:
ii adduser 3.67 Add and remove users and groups
ii debconf [debconf-2.0] 1.4.57 Debian configuration management sy
ii dpkg 1.13.10 Package maintenance system for Deb
ii libc6 2.3.5-3 GNU C Library: Shared libraries an
ii libpam-modules 0.76-23 Pluggable Authentication Modules f
ii libpam-runtime 0.76-23 Runtime support for the PAM librar
ii libpam0g 0.76-23 Pluggable Authentication Modules l
ii libselinux1 1.24-4 SELinux shared libraries
ii libssl0.9.7 0.9.7g-1 SSL shared libraries
ii libwrap0 7.6.dbs-8 Wietse Venema's TCP wrappers libra
ii openssh-client 1:4.1p1-6 Secure shell client, an rlogin/rsh
ii zlib1g 1:1.2.3-3 compression library - runtime

openssh-server recommends no packages.

-- debconf information excluded

Revision history for this message
In , Mark Nipper (nipsy) wrote : openssh-server: actually, to provide something useful...

Package: openssh-server
Version: 1:4.1p1-6
Followup-For: Bug #292932

 In an attempt to provide something useful to this request
(rather than my me too post a moment ago), I just ran across this:
---
http://groups.google.com/group/mailing.unix.openssh-dev/browse_thread/thread/d6f419da2faae3ad/508e0f2fb1208e7d?lnk=st&q=%22darren+tucker%22+hpn-ssh&rnum=1#508e0f2fb1208e7d

Hopefully that comes through properly. Anyway, it's a discussion
between Darren Tucker (one of the OpenSSH contributors) and Chris Rapier
(one of the HPN-SSH contributors) about the performance patch with some
suggested changes by Darren. Assuming this all goes well ultimately,
upstream might very well be including these changes.

 But for the time being, it might be nice to grab the original
HPN-SSH patch and make Darren's recommended changes and include it in
the Debian package. In another post I saw by Darren at:
---
http://groups.google.com/group/comp.security.ssh/browse_thread/thread/91064fcec483b534/7d2ecfbc5bd0369b?lnk=st&q=%22darren+tucker%22+hpn-ssh&rnum=2#7d2ecfbc5bd0369b

he mentions that using the none cipher probably won't increase
throughput too much as the MAC (Message Authentication Code) is usually
the bottleneck. Although, this would change probably if the previous
performance patch were also included, assuming that Darren was referring
to the MAC being the bottleneck with the currently implemented OpenSSH
buffering scheme.

 So anyway, that's all I know at this point. Something to at
least keep an eye on as more of us are working on fast network segments
and even people on high latency and possibly even slow networks look to
benefit from this.

-- System Information:
Debian Release: testing/unstable
  APT prefers unstable
  APT policy: (500, 'unstable'), (500, 'stable'), (1, 'experimental')
Architecture: i386 (i686)
Shell: /bin/sh linked to /bin/bash
Kernel: Linux 2.6.12.4
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)

Versions of packages openssh-server depends on:
ii adduser 3.67 Add and remove users and groups
ii debconf [debconf-2.0] 1.4.57 Debian configuration management sy
ii dpkg 1.13.10 Package maintenance system for Deb
ii libc6 2.3.5-3 GNU C Library: Shared libraries an
ii libpam-modules 0.76-23 Pluggable Authentication Modules f
ii libpam-runtime 0.76-23 Runtime support for the PAM librar
ii libpam0g 0.76-23 Pluggable Authentication Modules l
ii libselinux1 1.24-4 SELinux shared libraries
ii libssl0.9.7 0.9.7g-1 SSL shared libraries
ii libwrap0 7.6.dbs-8 Wietse Venema's TCP wrappers libra
ii openssh-client 1:4.1p1-6 Secure shell client, an rlogin/rsh
ii zlib1g 1:1.2.3-3 compression library - runtime

openssh-server recommends no packages.

-- debconf information excluded

Revision history for this message
In , Mark Nipper (nipsy) wrote : openssh-server: okay, so here's the patch I used...

Package: openssh-server
Version: 1:4.1p1-6
Followup-For: Bug #292932

 Well, after downloading those two patches from:
---
http://www.psc.edu/networking/projects/hpn-ssh/

I realized that they were the same but one has the none cipher changes
thrown in. Blah.

 Anyway, I'm attaching the diff I used against Debian's 4.1p1-6.
It includes Darren's changes from my previously linked Usenet
discussion. Any other white space changes which are not in keeping with
"standard" coding policies are vim's fault. :)

-- System Information:
Debian Release: testing/unstable
  APT prefers unstable
  APT policy: (500, 'unstable'), (500, 'stable'), (1, 'experimental')
Architecture: i386 (i686)
Shell: /bin/sh linked to /bin/bash
Kernel: Linux 2.6.12.4
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)

Versions of packages openssh-server depends on:
ii adduser 3.67 Add and remove users and groups
ii debconf [debconf-2.0] 1.4.57 Debian configuration management sy
ii dpkg 1.13.10 Package maintenance system for Deb
ii libc6 2.3.5-3 GNU C Library: Shared libraries an
ii libpam-modules 0.76-23 Pluggable Authentication Modules f
ii libpam-runtime 0.76-23 Runtime support for the PAM librar
ii libpam0g 0.76-23 Pluggable Authentication Modules l
ii libselinux1 1.24-4 SELinux shared libraries
ii libssl0.9.7 0.9.7g-1 SSL shared libraries
ii libwrap0 7.6.dbs-8 Wietse Venema's TCP wrappers libra
ii openssh-client 1:4.1p1-6 Secure shell client, an rlogin/rsh
ii zlib1g 1:1.2.3-3 compression library - runtime

openssh-server recommends no packages.

-- debconf information excluded

Revision history for this message
In , Mark Nipper (nipsy) wrote : openssh-server: new patch to fix disconnect problem

Package: openssh-server
Version: 1:4.1p1-6
Followup-For: Bug #292932

 This patch is to replace my previous. It reverts one of
Darren's suggested changes which was causing disconnects on large
transfers and reverses the order in which I had added the -hpn string to
the version string.

-- System Information:
Debian Release: testing/unstable
  APT prefers unstable
  APT policy: (500, 'unstable'), (500, 'stable'), (1, 'experimental')
Architecture: i386 (i686)
Shell: /bin/sh linked to /bin/bash
Kernel: Linux 2.6.12.4
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)

Versions of packages openssh-server depends on:
ii adduser 3.67 Add and remove users and groups
ii debconf [debconf-2.0] 1.4.57 Debian configuration management sy
ii dpkg 1.13.10 Package maintenance system for Deb
ii libc6 2.3.5-3 GNU C Library: Shared libraries an
ii libpam-modules 0.76-23 Pluggable Authentication Modules f
ii libpam-runtime 0.76-23 Runtime support for the PAM librar
ii libpam0g 0.76-23 Pluggable Authentication Modules l
ii libselinux1 1.24-4 SELinux shared libraries
ii libssl0.9.7 0.9.7g-1 SSL shared libraries
ii libwrap0 7.6.dbs-8 Wietse Venema's TCP wrappers libra
ii openssh-client 1:4.1p1-6 Secure shell client, an rlogin/rsh
ii zlib1g 1:1.2.3-3 compression library - runtime

openssh-server recommends no packages.

-- debconf information excluded

Revision history for this message
Joe Harrington (joeharr) wrote :

This is a wishlist item: hpn-ssh would be a big data-transfer speed improvement for many Ubuntu users, particularly scientific organizations like universities and observatories that transfer large amounts of data over long-haul networks, and large commercial users. hpn-ssh is a patch on openssh that just makes the TCP window variable in length, overcoming a static (and too-small) window size that is in openssh. No modifications are made to the security aspects of the code. Measured transfer rates are improved over some nets by a factor of 20. It's in wide use already by banks, the US government, many grid computing centers, and security companies. Even if it's not patched into openssh, making an alternative version available would be a service to the community. Other distros have this, but synaptic shows me nothing for Ubuntu. Here are some sites:

http://www.psc.edu/networking/projects/hpn-ssh/papers/hpnssh-gridnets2007.pdf
http://www.psc.edu/networking/projects/hpn-ssh/

Thanks for listening,

--jh--

Revision history for this message
Mathias Gug (mathiaz) wrote : Re: [Bug 162253] openssh: want hpn-ssh for 20x speed improvement!

On Mon, Nov 12, 2007 at 07:16:37PM -0000, Joe Harrington wrote:
> This is a wishlist item: hpn-ssh would be a big data-transfer speed
> improvement for many Ubuntu users, particularly scientific organizations
>
> http://www.psc.edu/networking/projects/hpn-ssh/papers/hpnssh-gridnets2007.pdf
> http://www.psc.edu/networking/projects/hpn-ssh/
>

Has this patch been submitted to the upstream developers ?

 status triaged
 importance wishlist

Changed in openssh:
importance: Undecided → Wishlist
status: New → Triaged
Revision history for this message
Joe Harrington (joeharr) wrote :

Yes, see second-to-last question of:

http://www.psc.edu/networking/projects/hpn-ssh/faq.php

What's needed now is
1) a request to be made from Ubuntu pointing out to upstream that it is a significant issue they should address (either with this patch or some other way), and
2) meanwhile, either apply the patch, or make available alternative versions of openssh-client and openssh-server that have the patch applied, as other distros have.

Given the nature of security software, I can understand not wanting to apply the patch if upstream hasn't, but providing clearly-marked alternatives should not be similarly inhibited. It's a factor of 20 performance improvement we're talking about here.

Also, if you make alternatives available, please do not disable the None cipher. It's hard enough to enable that nobody will use it accidentally, but for LANs it can make a factor of 4 or more speed difference, which is a lot for those of us dealing with terabyte datasets and trying to back them up over the net.

Thanks again,

--jh--

Revision history for this message
In , Colin Watson (cjwatson) wrote : Re: Processed (with 1 errors): merging 292932 353303

reassign 292932 openssh
reassign 353303 openssh
merge 292932 353303
thanks

--
Colin Watson [<email address hidden>]

Revision history for this message
borneoo (borneoo-freemail) wrote :

This would be great to add hpn ssh package to repo !!
thx

Revision history for this message
In , Colin Watson (cjwatson) wrote : merging 292932 468222

# Automatically generated email from bts, devscripts version 2.9.26
merge 292932 468222

Bug Watch Updater (bug-watch-updater)
Changed in openssh:
status: Unknown → New
Revision history for this message
Piotr Czachur (zimnyx) wrote :

Yeah, one vote from me!

Revision history for this message
cowmix (mmarch-gmail) wrote :

Another vote here too!

VERY MUCH NEEDED.

Revision history for this message
Josh Lucien (katakaio) wrote :

Ditto. It's got my vote - this is the number one thing on my wishlist!

Revision history for this message
Shane R. Spencer (whardier) wrote : Re: [Bug 162253] Re: openssh: want hpn-ssh for 20x speed improvement!

I submished a patch for this on the debian side of things.. but i'd love
to see an official version based on the openssh tarballs and it's own
package full fledged packages.

Josh Ellis wrote:
> Ditto. It's got my vote - this is the number one thing on my wishlist!
>

Revision history for this message
Rebecca Menessec (aloishammer-deactivatedaccount) wrote :

I've looked around and seen some valid concerns about incorporating a large patchset like this into the Ubuntu packages, but:

1) It's real handy. I'm getting easy 2.5x boosts at home over a single-switch gig-e network.

2) Upstream (openssh.com) doesn't seem to have any burning interest that I can find, after... two years of availability?

3) Upstream (psc.edu) seems willing and able to maintain relatively regular patchset versions for several OpenSSH releases at a time. No release for 5.2 just now, but that doesn't seem like significant lag (yet), in context.

Revision history for this message
Joe Harrington (joeharr) wrote :

Ubuntu is a packager, not a developer. The right way to get this to happen is for someone (who would have to be more motivated than I am) to talk to the Debian SSH maintainers, or even the openssh developers themselves. Options include forking (bad), making a completely new package out of the patched ssh and getting that into debian in parallel with the regular ssh, making such a package and serving it elsewhere, or convincing debian to accept the patch. I don't think this is the venue for further discussion, as it's clear ubuntu will not do anything directly.

--jh--

Revision history for this message
Thierry Carrez (ttx) wrote :

It's not really a question of packager/developer, or Debian vs Ubuntu (especially since in this case the Debian OpenSSH maintainer happens to also be an Ubuntu OpenSSH maintainer). Debian and Ubuntu got burned in the past over OpenSSH patches, I'm pretty sure we won't integrate this in the openssh package until it makes it to OpenSSH upstream.

That doesn't prevent someone else from providing, as proof of concept, PPA packages of hpn-ssh. That could help in demonstrating how great it can be.

Revision history for this message
Jeremy Nickurak (nickurak) wrote :

Has anybody gotten these patches to apply to karmic's openssh? I've been trying to build a package for this, but so far no luck applying the patch.

Revision history for this message
Chuck Ritola (cobra176) wrote :

Jeremy: I've had success getting HPN-SSH patch to apply to openSSH.org's source downloaded into 9.10 and installing. Perhaps you make a package from those sources? Note: I found it critical to use the right HPN patch for the right SSH version. Also, the openssh.org soruce install will install the binaries to slightly different locations than the ubuntu package.

Revision history for this message
marcobra (Marco Braida) (marcobra) wrote :

Here something to play with http://forum.slicehost.com/comments.php?DiscussionID=3424

Hth

Revision history for this message
Richard Hansen (rhansen) wrote :

Unfortunately, the HPN patch breaks the DynamicForward feature in at least OpenSSH 5.0 (see <http://gnats.netbsd.org/45049>). I'm not sure how it fares with newer versions of OpenSSH.

From the bottom of the HPN web site (<http://www.psc.edu/networking/projects/hpn-ssh/>):

    Recent problems with buffer_append_space in HPN-SSH. If you are experiencing disconnects due to a failure in buffer_append_space please let us know. We're currently tracking some problems with this and we're trying to gather more information to help resolve it. You may want to try using -oHPNBufferSize=16384 to restrict the growth of the buffer. Let us know if that helps.

Bug Watch Updater (bug-watch-updater)
Changed in openssh (Debian):
status: New → Won't Fix
Revision history for this message
Robie Basak (racb) wrote :

It seems that upstream and Debian are Won't Fix. I don't think Ubuntu will deviate from this (due to the security maintenance nightmare it may produce) so I'll set this bug to Won't Fix for Ubuntu to make this clear.

If upstream or Debian's position changes, we can always change this.

Changed in openssh (Ubuntu):
status: Triaged → Won't Fix
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.