Pangolin performance via Newt

[steuerlexi]

Hey everyone,

First of all, thanks for the great work on Pangolin – I really like how clean and straightforward it is to set up.

I’m currently testing Pangolin in a fairly performant setup:
• Home connection: 1 Gbit fiber
• VPS: 2.5 Gbit throughput
• Service behind Pangolin: Pingvin Share (self-hosted file sharing tool)

The problem I’m running into is related to throughput performance. When I upload a file to Pingvin through Pangolin, I only get about 4 MB/s (≈32 Mbit/s), and the speed occasionally drops even further. The same happens when I try to download the file back – it’s consistently low and doesn’t reflect the bandwidth I actually have on either side.

So now I’m wondering:
Where’s the bottleneck? Is it:
• A known limitation or default setting in Pangolin?
• Related to WireGuard performance under load?
• Possibly something in the configuration (like MTU size, buffer sizes, etc.)?

I’d really appreciate if someone could share similar experiences, tuning tips, or diagnostic steps I might try.

Thanks in advance!
Alex

[steuerlexi]

Is nobody else experiencing this problem ?

[AstralDestiny]

Can't replicate it..
For me pangolin is on a dedicated host colo, Not the fastest speeds it's only 1Gbps/300mbps, But cheap for a remote system,
at the test network with Newt, Newt runs in a ipvlan so it skips all of the host's iptables and other routing and talks directly with the host's nic so it can go line speeds uninterrupted (ipvlan in l2), Goes out to the switch then hops over to the host running Pingvin Share, Can upload at 50Mbps which is the max for the network I was testing on from a friend's host and uploaded a 938MB file which uploaded consistently at 50Mbps. The Pingvin host took the connection at the nic level and bound it right to the container skipping the docker host's network stack entirely and Pingvin happily got the file at the max speed of the uploading test host. The host running Pingvin is using ipvlan in l3 instead of l2 but same still occurs it bypasses the docker host's network stack and well it's entire stack so it's just 2 containers talking directly to eachother across hosts so no limiting factors past kernel processing.

Can you tell me how far your vps is from your newt? and how your internal network is structured and if using hairpinning by chance? Where the connection comes from newt and instead of talking directly to say pingvin you put the host ip so it's doing containerA(newt) > Host(Same host) > ContainerB

Anyways let me know.

[Razdnut]

I'm having the same issue. Maybe the problem is with Traefik( or gerbil ??)since the issue persists even locally. Some people have fixed it by switching VPS. Temporarily, I solved it by enabling the Cloudflare Proxy, which tripled my upload speed.

[Razdnut]

I switched to Hetzner CAX11 (arm64), and in terms of performance, it seems to be one of the best. The speed has improved significantly. I can also play 4K HDR videos with a bitrate of 78000 kbps without interruptions and without making changes to buffering, with crowdsec added. Location: Nuremberg. VPS with a declared speed of 500 mbps (info here) tested with iperf3 at 430 Mbps. At home, I have a 2.5 Gbps fiber connection.
Suggestion: avoid installing newt with Docker. Install it directly. I've found that Docker tends to slow down the connection (there are many articles on this topic and on the effects of overlay2), especially in bridge mode. Personally, I created a newt.service that starts on server boot, and I believe this has improved both upload and download speeds.
If we consider that there are two layers in between, one with Docker and one from the tunnel, it’s not surprising that the speed decreases. Additionally, Traefik doesn’t excel in this aspect either. I am currently testing the new fastproxy, but it only supports HTTP 1.

[johanngrobe]

I don't know what happened, but suddenly my performance issues are gone. For my part it had nothing to do with pangolin nor newt. I had the same performance issues with a direct wireguard connection and tailscale. Hetzner Nürnberg works flawlessly as well. It seems to be my internet provider that had issues.

[steuerlexi]

I don't know what happened, but suddenly my performance issues are gone. For my part it had nothing to do with pangolin nor newt. I had the same performance issues with a direct wireguard connection and tailscale. Hetzner Nürnberg works flawlessly as well. It seems to be my internet provider that had issues.

very good to hear this. So Hetzner seems to be the best choice. What are you guys paying for the service at the moment?

[synologyy]

~5€

[steuerlexi]

I decided to explore more VPS services and revisited Ionos out of curiosity.

Ionos, in combination with my ISP "Deutsche Glasfaser," did not connect directly to the Ionos data center in Berlin but instead routed through Amsterdam, which significantly increased latency and reduced throughput. I also evaluated 1Blu and netcup. Here is my top 4 list:

1. Hetzner - 10ms
2. 1Blu - 10ms
3. netcup - 13ms
4. Ionos - 40-90ms

I decided to go with netcup as they hat the best offer with 2 euros

[alexandrescieux]

I’m also experiencing a severe performance issue with my Pangolin/Newt/Gerbil setup, and I can’t understand why.

It works flawlessly for serving low-bandwidth workloads (websites) — it's great, and I love it.
However, it struggles when it comes to streaming (4K video / FLAC audio) or downloading files.
I only get around 8–10 Mb/s (~1 MB/s), which causes long buffering times and very slow file downloads.

---

My setup is as follows:

• Home server

  • Ryzen 9 9955HX
  • 10 Gbit SFP+ port
  • Proxmox VE 8.4
  • Newt 1.3.1 in a dedicated VM, installed via Docker Compose following the documentation

• Router

  • GL.iNet Flint 2 with 2.5 Gb LAN ports for WAN and LAN
  • Running OpenWRT

• VPS (OVH VPS S - 2 vCPU / 2 GB RAM - Gravelines datacenter)

  • Pangolin 1.6.2 + Gerbil 1.0.0 + Traefik 3.4.4, installed via Docker Compose as per the documentation

---

What I’ve tried to identify the root cause:

Suspected CPU bottleneck (WireGuard overhead)

I initially suspected that Newt and/or Gerbil/Traefik were CPU-bound due to WireGuard overhead.
However, running docker stats on the VPS and checking the Proxmox dashboard ruled that out:
Pangolin/Gerbil/Traefik/Newt combined consistently stay under 10% CPU and 10% RAM usage, even when launching 5 streams or downloads in parallel.

---

Suspected bandwidth bottleneck

I ran speedtest-cli in different environments:

• On Proxmox (Debian): ~500 Mb/s up/down (symmetric sub-gigabit fiber from my ISP)
• Inside the Newt VM: ~500 Mb/s up/down (same as the hypervisor)
• On the VPS: ~450 Mb/s up/down (slightly below OVH’s advertised rate)

Everything seems to have a decent internet connection.

---

Suspected network throughput between components

I deployed iperf3 in server mode via Docker on the VPS (same conditions as Pangolin/Gerbil/Traefik), then:

• Ran iperf3 in client mode from Proxmox: 450 Mb/s up/down
• Ran iperf3 in client mode from the Newt VM directly (not via Docker): 450 Mb/s up/down
• Ran iperf3 in client mode from the Newt VM via Docker: 450 Mb/s up/down

So everything seems to have proper link speed internally despite the abstraction layers (VMs, Docker, VPS).

---

When monitoring netin/netout in the Newt VM via the Proxmox dashboard, I can clearly see the 1 MB/s cap on the graphs (not necessarily blaming Newt, but it aligns with the observed limit).

---

Given the above, it doesn’t appear to be a CPU, RAM, or bandwidth issue — which leaves us with either the Newt/Gerbil/Traefik codebase or their configuration.

Let me know if you need more details or logs — I’d be happy to help debug this further.

[alexandrescieux]

Happy to have contributed to the analysis!

Unfortunately, I'm not experienced enough in Go to make such a deep change to the networking codebase.
I'll be looking forward to seeing it included in a future release.
Bought a limited supporter key, keep up the good work !

For now, the current speed brute-forcing packets one by one from a Hetzner CAX11 is sufficient for a single-user Pangolin instance like mine. However, it could become a bottleneck in an enterprise setup.

[Zedifuu]

I've also been facing the same issue as the above I thought I was going mad! and ended up switching between a number of VPS providers to see if that was the issue. All facing similar results with iperf and transfer speeds.

I have been trying to get a basic wireguard setup between pangolin and Opnsense, in the meantime with not much luck, Basic Wireguard connections dont seem documented 🥲

Should this discussion above maybe be translated into an issue on the Newt repository so it's easier to find and not forgotten about?

[steuerlexi]

@oschwartz10612 @johanngrobe @synologyy @alexandrescieux

Following the ongoing discussions about performance bottlenecks within the Pangolin and Newt setup, it has become clear that a key limitation is the sequential processing of packets, which significantly restricts throughput speeds especially for high-bandwidth use cases like 4K video streaming and large file transfers.

While the core team has acknowledged that parallel processing is both desirable and feasible, it has not yet been implemented, primarily due to resource constraints and the initial focus on delivering a functional and efficient baseline.

Therefore, we are reaching out to the community:

Would any developer be willing to contribute by creating a pull request to implement parallel processing in the Newt/Gerbil/Pangolin codebase?

Your contribution would be invaluable in overcoming this current software limitation and could dramatically improve performance and user experience. For those interested, we can provide guidance on the project structure and current bottlenecks to help get started.

In the meantime, some workarounds such as avoiding Docker for Newt installation, choosing high-performance VPS providers like Hetzner CAX11, and minimizing proxy layers (e.g., Traefik) can help improve performance temporarily. However, the real solution lies in parallelizing packet handling.

If you are interested or have any questions about how to proceed with this contribution, please do not hesitate to reach out. Your participation would make a significant impact!

Thank you for considering this request.

[oschwartz10612]

Change would be in https://github.com/fosrl/newt/blob/main/proxy/manager.go

If someone wants this would be cool to add.

If not I DO want to do this soon and I am sorry for the slow speed. SO much we are doing these days lol.

I also think there is an issue with too many UDP sockets being created that needs looking at...

[steuerlexi]

Thank you so much @oschwartz10612 I can see there’s so much happening, and we all truly appreciate your efforts. This is an amazing product that really has no real competitors! :-)

Interesting fact around the UDP sockets!

[steuerlexi]

btw. i would like to share my workaround until we have this. It´s simple. Basically let every service/ressource have it´s own newt tunnel/sites.

[MisterGoodcat]

I've been pulling my hair out over this until I found this discussion. I had no problems at all for several months, but only with low-bandwidth services. Recently I added my photo library to the setup, and then realized I'm not able to stream any videos reliably.

The behavior I see is consistently the following: with larger files, the initial transfer speed is high for a few seconds (meaning a few megabytes), and then it rapidly drops to unacceptable levels (around 10 kb/s!). After a quite long time (1-2 minutes) it equally rapidly starts to pick up speed again and then keeps saturating my line until the transfer is complete (several mb/s). If I interrupt the transfer and restart, it starts all over again with the same behavior.

During the time it's only dripping, the load on the VPS is non-existent. Once it picks up speed again I see Traefik at ~25% CPU. I did extensive testing to check for bandwidth and speeds all along the setup, and I'm able to saturate my public line to and from the VPS (Strato Germany) no matter what (raw transfer through SCP, with or without Docker etc.). I'm also able to max out the local GBit network when I access the services only locally.

It doesn't sound like the exact symptoms you describe, but similar in that I cannot find any issues at any stage of the setup, but when I add Newt to the equation it reproducible shows this behavior.

[aszurnasirpal]

Just want to upvote to raise attention to @oschwartz10612. I face the same. My Pangolin is located on an Oracle Free Tier instance (so 480 Mbps network cap), but in fact, while traffic is routed via Pangolin, the real speed seems to be limited to 2-3MB/s, instead of 60 that I should have)

[Karsto-sys]

+1

[AstralDestiny]

@MisterGoodcat Mind bugging me in the pangolin discord by chance?, When running did you use the iperf within the namespace of the gerbil container downstream to newt then off to a service without hairpinning locally? and kept it running then started to shunt data through and see if it remains stable or not?

[lamixer]

A google search brought me here. I am running Nextcloud behind Pangolin on a VPS for local resources only (no tunnels / Gerbil / Newt at all). Max out around 3 megabit/s when should be at least 200. I guess not a Newt problem, or not exclusively Newt anyway.

[MisterGoodcat]

@MisterGoodcat Mind bugging me in the pangolin discord by chance?, When running did you use the iperf within the namespace of the gerbil container downstream to newt then off to a service without hairpinning locally? and kept it running then started to shunt data through and see if it remains stable or not?

Shortly after that weekend of analysis I moved away from Pangolin to a different setup for my requirements. Apologies that I don't remember the specifics because I tried so many combinations back then (and it's been a few months), and that at this time I don't have a repro environment anymore.

I really like the project and still watch its progress and this discussion, to maybe reconsider it in the future. All the best!

[alexandrescieux]

I migrated my Pangolin instance from the default SQLite database (pangolin:1.12.2) to a PostgreSQL backend (pangolin:postgresql-1.12.2).
I’m seeing a noticeable improvement in UI responsiveness and in the loading speed of static web pages (which was already reasonable). However, I’m still experiencing the same issue when streaming or downloading large files, even with PostgreSQL.

It was definitely worth trying, and the performance boost for the UI and static pages is appreciated.
My guess is that SQLite is simply too slow on VPS setups that rely on shared spinning-rust HDD backends.

[aszurnasirpal]

Just out of curiosity. How did you migrate to PostgreSQL? using this script or differently?

[alexandrescieux]

I use Pangolin as a hobby/self-hosting setup, so I was able to afford some downtime. I deployed the new image on a fresh database, redeployed Newt with the new secrets, and reapplied my blueprints that contain all my configuration as code.
So unfortunately, I didn’t use any migration script and can’t provide any feedback on that part.

[thutex]

using this script

i've just looked at it, and this script is extremely outdated and does not have all the correct tables/table structures,
so it cannot be used in its present form.
it's probably not an enormous amount of work to get it fixed up though.

[aszurnasirpal]

Ohh, thank you very much! You saved me a lot of time that I would have spent trying to figure out why it isn't working.

[steuerlexi]

This discussion seems to be one of the most active ones, yet I haven’t seen this issue being addressed. I used to really like Pangolin, but it’s time for me to move on from the project.

Thank you very much for everything so far. I wish the project and everyone involved all the best — and as they say, paths always cross again.

[Karsto-sys]

Look at https://github.com/wiredoor/wiredoor
Its similar but whitout the speed issue I get nearly 100% bandwith on the same vps I used Pangolin, really happy with it

[gaizkaeu]

Same here. I couldn't get more than 6 MBps, used Wiredoor and now +20 MBps...

[keonramses]

Unfortunately issue still persists in the latest newt version. (IPV6 only)

local site:


newt site:


wireguard site:

[AstralDestiny]

What's your upload speed? at the newt location ? and can you do an iperf3 test?

[keonramses]

Newt upload speed is above. Here is the iperf3 test result.

[joshuadow]

After struggling with inconsistent throughput over Newt, I switched to a Basic WireGuard Site. Setup was pretty painless and immediately fixed the “bursty stall/buffering” behavior for long-lived streams.

Problem

Persistent connections would stall in bursts over Newt tunnels (CPU wasn’t pegged). Shorter/lower-bitrate flows were ok which strongly pointed to PMTU/fragmentation (TCP bursts + long stalls).

Fix

Use a Basic WireGuard Site and route traffic over kernel WireGuard. Key points:

• Point resources to the WG peer IP, not your LAN IP.
• Pin MTU (1280) + add TCP MSS clamping (otherwise wg-quick can revert MTU to 1420 after restart and the buffering comes back).

---

Solution

1) Create a Basic WireGuard Site in Pangolin

• Create the site and bring up WireGuard on a LAN gateway LXC/VM: wg-quick@wg0
• Confirm handshake: wg show

Note: the server-side WG interface may live inside the gerbil container (not on the pangolin host):

docker exec -it gerbil ip -br addr   # should show wg0 with e.g. 100.89.x.x/24

2) Point the resource upstream to the WG peer IP (NOT the LAN IP)

In Pangolin, set the resource Target/Upstream to:
https://{WG_PEER_IP}:{PORT}
Do not target 192.168.x.x directly for Basic WG sites unless you’re forwarding/NATing on the peer.

3) On the WG peer gateway: DNAT only what you need to {TARGET_IP}:{PORT}

Put the following in /etc/wireguard/wg0.conf on the WG peer (replace {TARGET_IP}, {PORT}, and your LAN interface name if it’s not eth0):

[Interface]
Address = <WG_PEER_IP_CIDR>
PrivateKey = ...
MTU = 1280

# Forward <PORT> from WG -> target
PostUp   = sysctl -w net.ipv4.ip_forward=1
PostUp   = iptables -t nat -A PREROUTING -i %i -p tcp --dport <PORT> -j DNAT --to-destination <TARGET_IP>:<PORT>
PostUp   = iptables -t nat -A POSTROUTING -o eth0 -p tcp -d <TARGET_IP> --dport <PORT> -j MASQUERADE
PostUp   = iptables -A FORWARD -i %i -o eth0 -p tcp -d <TARGET_IP> --dport <PORT> -j ACCEPT
PostUp   = iptables -A FORWARD -i eth0 -o %i -p tcp -s <TARGET_IP> --sport <PORT> -j ACCEPT

# MSS clamp (prevents PMTU blackholes / burst-stall behavior)
PostUp   = iptables -t mangle -A FORWARD -i %i -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu
PostUp   = iptables -t mangle -A FORWARD -o %i -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu

PostDown = iptables -t nat -D PREROUTING -i %i -p tcp --dport <PORT> -j DNAT --to-destination <TARGET_IP>:<PORT>
PostDown = iptables -t nat -D POSTROUTING -o eth0 -p tcp -d <TARGET_IP> --dport <PORT> -j MASQUERADE
PostDown = iptables -D FORWARD -i %i -o eth0 -p tcp -d <TARGET_IP> --dport <PORT> -j ACCEPT
PostDown = iptables -D FORWARD -i eth0 -o %i -p tcp -s <TARGET_IP> --sport <PORT> -j ACCEPT
PostDown = iptables -t mangle -D FORWARD -i %i -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu
PostDown = iptables -t mangle -D FORWARD -o %i -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu

4) Persist across reboots

systemctl enable --now wg-quick@wg0

Quick sanity checks
ip -d link show wg0 | grep mtu # note that 1420 caused buffering to return
iptables -t nat -S | grep
iptables -t mangle -S | grep TCPMSS

[van-geaux]

After struggling with inconsistent throughput over Newt, I switched to a Basic WireGuard Site. Setup was pretty painless and immediately fixed the “bursty stall/buffering” behavior for long-lived streams.

I've confirmed that this workaround works for me. In mine I'm using docker for the wireguard and I had to bound the apps to the wireguard network, example:

services:
  wireguard:
    image: lscr.io/linuxserver/wireguard:latest
    container_name: wireguard
    cap_add:
      - NET_ADMIN
      - SYS_MODULE
    environment:
      - PUID=1027
      - PGID=100
      - TZ=Asia/Jakarta
    volumes:
      - /mnt/user/docker/00000-wireguard:/config
      - /lib/modules:/lib/modules:ro
    ports:
      - 51820:51820/udp
      - 3000:3000                                           # the app's port
    sysctls:
      - net.ipv4.conf.all.src_valid_mark=1
    restart: unless-stopped

  whoami:
    image: ghcr.io/traefik/whoami:latest
    container_name: whoami
    # network_mode: host
    network_mode: "service:wireguard"       # in network mode bound it to wireguard service
    # ports:
    #   - 3000:3000
    depends_on:
      - wireguard
    restart: unless-stopped

edit:
Forgot to mention that I need to put the wireguard config from pangolin in a wg0.conf inside wireguard's /config dir

[essys]

I wonder if there are any clues as to what might be causing this.
Has there been any official feedback from the maintainers/developers on this issue?

[memesalot]

Same issue here.

[BSZ7]

Yeah, I’ve been running into the same issue with large file streams. I also tried replacing Newt with a plain WireGuard connection, but unfortunately the speeds still dropped quite a bit. For now, I’ve had to switch to a different tool for the affected applications.

Really hoping this gets improved at some point. Everything else works great, so it would be amazing to see this resolved.

[kylepyke]

Same issue here. I understand there's likely no easy fix, bu it seems like a pretty critical issue.

[garysan-uk]

I've encountered same issue. Using a 2vCPU VPS with Pangolin installed and running on it. Newt, as a service on my Win11P box. Have nice 'elegant' solution that I can point to (plex.mydomain.com, overseer.mydomain.com, etc.) but Plex streaming is limited to 720p/4mbps through the Newt tunnel... I've had to switch out to Tailscale which isn't quite as elegant, as it requires something installed on my Plex users' machines.

I too would really like to see this resolved as Pangolin is a super-neat solution otherwise.

[Boscovitz]

Same here. Streaming 4K over newt is impossible. Working ok on standard Wireguard.

[balgerion]

Racknerd 2vcpu streaming 4k was working but it took good minute to start switched to std WireGuard and its almost instant

[vadim0872]

In your setup did you happen to have CrowdSec protection? There's an adacent issue where CrowdSec can't handle large file uploads, but I'm curious if they're related somehow

NO, NO NO , it not problem in crowdsec, I turn off it, and it is no changes. When big is uploading in htop I see CPU load on server.js process, that seems that is pangolin core is bottlenecking.

[essys]

I agree, this has nothing to do with Crowdsec, as I did not use Crowdsec in my tests, but I did experience serious performance issues with Newt.
I have not yet tested the direct Wireguard connection yet.

[Razdnut]

It should be added that Traefik is slower than other reverse proxies in circulation.
A test with Traefik and a tunnel wireguard should be done to test performance.

[vadim0872]

Upload/Download - no problem and max speed with traefik, without pangolin

[kylepyke]

I was referring to this issue, which I’m also having:

crowdsecurity/crowdsec#3837

Apologies, was just polling to see if the issues were linked by having Crowdsec installed, but clearly not.

[IvanSivric]

Same issue for me with newt ->120-150 Mbps (6-30 MB/s). iperf between the local vm and my vps (without tunnel) -> 6 Gbps.
I will try to replace it with wireguard.

[EDIT] With wireguard instead of newt -> 600+ Mbps . It's 5-6 times faster.

[tzhouML]

Hi there, for wireguard setup, does it still involve a VPS? If so, would you be able to give me a brief tutorial on how can I make the switch?

[kylepyke]

Switching to wireguard did not solve the issue for me, but here is the guide I used:
https://forum.hhf.technology/t/fix-slow-or-bursty-speed-in-pangolin-when-using-newt-tunnels/4082

[tzhouML]

This issue is painful and it really has be needing an alternate system despite Pangolin serving me so well. What I find puzzling is why did this happen some time in recent months? I recall my network transfer speed being almost as fast as local in the past.

[TheD3vilsAdvocate]

I get usable 15MBsec speeds from my Windows Host using Newt via Docker. I'd like it to be more but compared to the others here I can't complain. My VPS offers 500Mbit speeds and iperf3 to the VPS really provides that speed.

When downloading large files from Nextcloud, all involved containers (Nextcloud, Authentik, Nginx and ultimately Newt) consume a lot of CPU power on the host with almost all 16 cores used. If I reduce the CPU max speed to 90% (via Power Settings) the DL speeds will drop to 13MBsec, at max speeds I get 15MBsec

Upgrading the VPS from 1 to 2 cores did absolutely nothing, however.

[Fuschl-dev]

The Newt tunnel significantly limits my available bandwidth. While my raw connection achieves ~180 Mbit/s, the throughput drops to ~20 Mbit/s once the tunnel is active, making high-bitrate 4K streaming impossible.

I set everything up fresh on a Hetzner CCX13 instance and tested it with my PC to rule out the possibility that anything was wrong with my local home server.

Environment
• Server: Hetzner CCX13, fresh Ubuntu installation. Pangolin Installed Without CrowdSec
• Client: Windows 11 PC
• ISP Speed: 200 Mbit/s Up/Down (Symmetrical)

Chronological Steps & Observation

1. Raw Connection: Tested the connection between my Windows Client and Hetzner CCX13 using iperf3 without any tunnel.
   o Result: ~180 Mbit/s.
2. Setup: Installed Pangolin on a fresh Ubuntu instance (CCX13) and established a Newt tunnel to the Windows client to route Jellyfin traffic.
3. Tunnel Performance Measurement: Paling a Jellyfin file with a bitrate of 70 Mbit/s through the active Newt tunnel.
   o Result: ~20 Mbit/s.

The bandwidth is insufficient for 4K video playback. I can't stream Jellyfin without constant buffering as the bitrate of the media exceeds the tunnel's capacity.

[tzhouML]

Hey all just an update that on newt i was getting 32mbps, switched to wireguard and am slightly below 1000mbps now.

[lazidoca]

it is almost a year and we still have this issue :))

[aszurnasirpal]

Just one note, we're focusing on Newt as a possible "bottleneck" here, but I'm no longer sure if this is the case.
I briefly installed Netbird; the latest version includes a reverse proxy feature based on Traefik (it uses WireGuard to connect endpoints).
Result: When I tried to stream a 4K movie from Jellyfin, I couldn't achieve smooth playback. It was very slow (comparable to what I get via Pangolin).
When I connected to Jellyfin directly via Netbird tunnel (without proxy), the playback was smooth.
So, maybe the problem is with Traefik, which isn't designed to handle high traffic volumes, rather than Newt?

[gaizkaeu]

Just one note, we're focusing on Newt as a possible "bottleneck" here, but I'm no longer sure if this is the case. I briefly installed Netbird; the latest version includes a reverse proxy feature based on Traefik (it uses WireGuard to connect endpoints). Result: When I tried to stream a 4K movie from Jellyfin, I couldn't achieve smooth playback. It was very slow (comparable to what I get via Pangolin). When I connected to Jellyfin directly via Netbird tunnel (without proxy), the playback was smooth. So, maybe the problem is with Traefik, which isn't designed to handle high traffic volumes, rather than Newt?

Same here. I was using Netbird with a Caddy server as a reverse proxy for jellyfin. Very poor performance. If I connected to the Jellyfin instance directly I had no problems.

Switched to Pangolin, same problem.

Tried wiredoor and I got full speed with wireguard tunnel + traefik reverse proxy.

Finally I’m using caddy + direct tunnel. No performance issues.

[vadim0872]

So, maybe the problem is with Traefik, which isn't designed to handle high traffic volumes, rather than Newt?

No, Traefik is not bottlenecking. I test upload big files on Nextcloud via Traefik, without pangolin and it`s pass on full speed without any drawdowns in speed. Bottlenecking in pangolin backend.

[legionGer]

My speed doubled by switching from newt to WireGuard … WG uses basically my full upload…

[K-MeLeOn]

After struggling with this issue for several weeks, I found the solution for my configuration.

The main problem comes mainly from Docker, so don't expect any more speed with it. You MUST install Newt AND Gerbil in pass-through mode (host mode, not bridge mode) to allow Gerbil and Newt to create a true WG virtual card on your servers. Without this, you can't exceed a speed of ~300-400 MB/s.

In addition, for 4k streamers, in my opinion, you MUST have a dedicated WireGuard site on your router/server (and only for this use). It's a painfull setup for some, but it's worth it. For example, I now reach 750-850 MB/s for a 950MB/s real capable upload speed.

[garysan-uk]

I'd love to see a straightforward set of instructions for having Pangolin on a VPS, and setting up a WG client at the Windows 11 end... I can't be the only one with this setup (currently with Newt, and struggling with the speed for streaming media). I'm using Tailscale now which does work but I'd like to make use of the VPS I'm paying for.

Everyone says setting a WG tunnel up is the answer but I can't find anywhere that instructs exactly how to do it - I mean, like I'm an idiot and have no experience of WG. They either have Windows at both ends or a Linux distro where my Win 11 box is... Please?

[GlenNicholls]

This isn’t really relevant to this discussion so you’d get the best help in a new discussion or issue with specific details regarding your setup, what you’ve already done, and a clear outline of what is not working.

With that said, the quick install guide in the docs should get you up and running on your VPS. Then the only steps once you have pangolin installed is to open the required ports on your VPS, install Newt on your windows system, point newt to your VPS static IP, and create a newt site in pangolin. You could use wireguard instead of newt but the config is slightly more complicated.

The easiest way to set up newt would be via docker or docker compose on your windows system. While I haven’t deployed on Windows, docker should make the OS mostly irrelevant.

EDIT: sorry, I misread your question. It sounds like you got things working with newt but experienced speed/bandwidth issues and would like to get this working with WG. It’d be helpful if you posted your WG config and pangolin WG site config with sensitive info obfuscated.

[garysan-uk]

Not sure if anyone else has noticed a difference but since I updated Newt to v1.10.0 and Pangolin (self hosted on a VPS) to v.15.4 - things have markedly improved and I'm able to now stream directly from my media server at 1080p/12mbps (previously I couldn't achieve anything above 720p/4mbps, as many above have also mentioned).

Perhaps someone else can join these dots as to which is having the desired effect, but I installed Newt using the newt_windows_installer.exe as opposed to the previous version I had, which was just the Newt.exe - (always been installed/set-up as a service on Win 11 Pro). This time around, I noticed the installer also installed wintun.dll, along with Newt.exe. Now, I do remember some mention of perhaps needing wintun.dll in the Pangolin/Newt docs but as Newt 'appeared' to be working ok, I didn't think any more about it at the time.

Maybe it's the new version of Newt or maybe it's the inclusion of the wintun.dll into the Newt directory this time around but so far, it's working really well for me now and just what I wanted it to be. Thanks 🙏🏻

[LaurenceJJones]

Hey all, I wanted to jump in as one of the newest members of the Pangolin team to keep communication clear on this issue.

I know this has been open for a while, but a lack of active replies does not mean this discussion has been deprioritized. @oschwartz10612 has an active task to investigate and improve performance, and I've also opened follow-up tasks to improve Go-side performance over time.

The performance work merged into dev so far includes:
fosrl/newt#204
fosrl/newt#205

In Newt, we currently use two WireGuard netstack implementations: netstack and netstack2. The main tunnel to Gerbil is still using netstack, while child/downstream tunnels use netstack2, which contains the newer performance improvements.

A full switch is not just a simple toggle. Even though the implementations are compatible, we still need proper test coverage before promoting this broadly, to avoid inconsistencies or regressions.

[memesalot]

Awesome thanks!

[v1rusnl]

@LaurenceJJones Just to clarify, this did not make it into 1.10.2 or 1.10.3, right?

[LaurenceJJones]

Nope, as I just put in comment in my testing even with netstack2, the perf improvement was very minimal, I found more optimizing the kernal options gave better results (10mb/s on larger packets -> 140mb/s). However, be wary about changing these, maybe we can write a guide on what to tinker with.

[seanjones80]

Aye would be good to see a cheeky guide on this :)

[v1rusnl]

Nope, as I just put in comment in my testing even with netstack2, the perf improvement was very minimal, I found more optimizing the kernal options gave better results (10mb/s on larger packets -> 140mb/s). However, be wary about changing these, maybe we can write a guide on what to tinker with.

You mean stuff like net.core.rmem_max, net.core.wmem_max, net.core.rmem_default, net.core.wmem_default etc.?

I'm sitting now with the following changes to the default configuration via sysctl.d of the two clients. Changing the default way to small values e.g. for rmem and wmem made a big change for e.g. loading speed of e.g. images etc. Not just via Newt, but also my local Pangolin Instance running on the Newt Client.

VPS (Hetzner):

net.core.rmem_max           = 16777216 
net.core.wmem_max           = 16777216 
net.core.rmem_default       = 1048576 
net.core.wmem_default       = 1048576
net.ipv4.tcp_rmem           = 4096 1048576 16777216
net.ipv4.tcp_wmem           = 4096 1048576 16777216
net.ipv4.tcp_congestion_control = cubic
net.core.default_qdisc      = fq_codel
net.ipv4.tcp_slow_start_after_idle = 0
net.core.netdev_max_backlog = 5000

Newt Client (Binary on OpenMediavault with CPU N3540):

net.core.rmem_max           = 16777216
net.core.wmem_max           = 16777216
net.core.rmem_default       = 1048576
net.core.wmem_default       = 1048576
net.ipv4.tcp_rmem           = 4096 1048576 16777216
net.ipv4.tcp_wmem           = 4096 1048576 16777216
net.ipv4.tcp_slow_start_after_idle = 0
net.core.netdev_max_backlog = 5000

Disclaimer for anyone trying it out: Beware, these values work really well in my setup. Other setups may need other values or some values are not needed since the defaults are well by default.

[TheD3vilsAdvocate]

I tried a lot of things including Newt inside Docker, outside docker, a direct connection (ie DNS to open port without pangolin) and ultimately Pangolin + Wireguard Basic connecting to a Unifi Router.

What's pretty clear is that VPS performance and also Host performance play a minor role.

When using immich, I was getting top speeds with a direct connection and with wireguard basic. It goes up to 25MBsec and beyond, but it still starts pretty slow, i.e 600KBsec-3MBsec. That's apparently a TCP limitation and after about 20 seconds or so it should go through the roof.

Wireguard basic gives me much better speeds than newt overall. So I'm happy it works and that I can keep using Pangolin that way.

Interestingly, downloading a large file via Nextcloud will work with 15MBsec using both Newt and WG-Basic, but WG has less fluctuations. Here the difference is somehow minor compared to immich. I suspect Nextcloud Overhead but who knows...

[twocoolbeans]

Food for thought...speed while using Pangolin in some situations may have little to do with Pangolin or newt. Some CGNAT's don't play well with UDP—especially when it comes to video streaming. WireGuard is always over UDP. In order to test if UDP behind a CGNAT is impacting speed, try accessing a resource behind an SSH reverse tunnel (which is always over TCP) and compare the speed to accessing the resource behind Pangolin.

For example, Jellyfin over a reverse SSH tunnel:
ssh -R 8096:localhost:8096 user@domain

[LaurenceJJones]

Yes as well as there a few kernal optimizations you can do with UDP packets that can improve speed but from my testing so far, even with netstack2 it didnt really improve speeds. However, changing the kernal changed it from 10 mb/s to 140 mb/s. So in short, it all depends on the infrastructure and tcp/udp.