Panic

Panic Blog

March 5th, 2018

A few months ago, a complaint started popping up from users downloading or updating our apps: “Geez, your downloads are really slow!”

If you work in support, you probably have a reflexive reaction to a complaint like this. It’s vague. There’s a million possible factors. It’ll probably resolve itself by tomorrow. You hope. Boy do you hope.

Except… we also started noticing it ourselves when we were working from home. When we’d come in to the office, transfers were lightning fast. But at home, it was really, seriously getting hard to get any work done remotely at all.

So, maybe there was something screwy here?

The Video

Before digging in, here’s this story in convenient summarized video form, if you’d prefer!

Now on to the details.

The Test

The Panic “network topology” is actually very simple. The Panic web servers have a single connection to the internet via Cogent. We colocate our own servers, rather than using AWS or any other PaaS, and we also don’t currently use a CDN or any other cloud distribution platform.

So, if something is making our downloads slow, it ought to be pretty easy to do some analysis and figure out why, or at least where.

We wanted to know three things:

  • How fast can people download from our website?
  • How fast can people download from a “control” website that’s not on our network?
  • What are people using for their internet provider?

We made an extremely simple test page that transfers 20MB of data from our server to the browser, then sends the user to run the same script on the control server, which we chose to host with Linode. (The Linode server is located in Fremont, CA, the closest we could find to us here in Portland.)

We tweeted the link out, and data started pouring in…

The Results

Here’s what we got back, comparing how fast our users could download from our control server through Linode, and from our own servers through Cogent:

Graph comparing transfer speeds between servers

(There are 1,645 samples in our target range, after filtering out TLDs with fewer than 10 occurrences, and we’ve done a box plot, which shows a spread of all the data points.)

Well, well, well. It doesn’t take statistical genius to see one glaring outlier — and that was Comcast, with download speeds often being as low as 300 kilobytes/second. And you’ll never guess what provider is used by virtually every Panic employee when they work from home? Yeah, Comcast. There is, in fact, no other cable ISP available to Portland residents.

But, before jumping to conclusions, there was something else that was weird with the Comcast data: a huge number of outliers, way more outliers than any other provider. See all those red dots on the graph, ranging from very slow to very fast?

The answer to that mystery was solved when we plotted out Comcast data across different times of the day…

Graph of Comcast transfer speeds in the morning versus the evening

Nuts. The problem reports we’d been hearing were indeed a real thing.

Our downloads really were slow — but seemingly only to Comcast users, and only during peak internet usage times. Something was up.

At first we thought, maybe Comcast bandwidth is just naturally more congested in the evening as people come home from work and begin streaming Netflix, etc. But that didn’t explain why the connections to our Linode control server from Comcast, during the exact same time windows for each tester, were downloading with good speeds.

We wondered, is Comcast intentionally “throttling” Cogent customers? And if so, why?

The Why

Peering.

Major internet pipes, like Cogent, have peering agreements with network providers, like Comcast. These companies need each other — Cogent can’t exist if their network doesn’t go all the way to the end user, and Comcast can’t exist if they can’t send their customer’s data all over the world. One core tenet of peering is that it is “settlement-free” — neither party pays the other party to exchange their traffic. Instead, each party generates revenue from their customers. Cogent generates revenue from us. Comcast generates revenue from us at home. Everyone wins, right?

After a quick Google session, I learned that Cogent and Comcast have quite a storied history. This history started when Cogent started delivering a great deal of video content to Comcast customers… content from Netflix. and suddenly, the “peering pipe” that connects Cogent and Comcast filled up and slowed dramatically down.

Normally when these peering pipes “fill up”, more capacity is added between the two companies. But, if you believe Cogent’s side of the story, Comcast simply decided not to play ball — and refused to add any additional bandwidth unless Cogent paid them. In other words, Comcast didn’t like being paid nothing to deliver Netflix traffic, which competes with its own TV and streaming offerings. This Ars Technica article covers it well. (How did Netflix solve this problem in 2014? Netflix entered into a business agreement to pay Comcast directly. And suddenly, more peering bandwidth opened up between Comcast and Cogent, like magic.)

We felt certain history was repeating itself: the peering connection between Comcast and Cogent was once again saturated. Cogent said their hands were tied. What now?

The Fix

There was only one last hope: get Comcast to fix it. I know, like we were somehow going to convince this 200 billion dollar corporation to add more capacity to their interconnection with Cogent. If I asked you to rate the possibility of that actually happening on a scale of “no” to “never”, you’d probably pick “come on man are you serious”, right?

But after a lifetime of being a “hey, it’s worth a shot” guy, I had to try. I did a real quick Google search for Comcast corporate contacts and found a person who seemed like they were involved in network operations PR, and I fired off a quick e-mail explaining the situation to Comcast.

And then, the craziest thing happened…

They wrote back quickly. Not only that, but they were on it. We set up a phone call. They took us seriously, they wanted to know the backstory, they wanted to know what our customers were seeing, and they were going to talk to the right people — they even e-mailed Cogent to connect with the right person in peering over there.

And pretty soon a call came back with a definitive-sounding statement: “Give us 1 to 2 weeks, and if you re-run your test I think you’ll be happy with the results.”

Sure enough, we waited two weeks, had our users re-run the speed test, and wouldn’t you know it…

Graph comparing transfer speeds of different ISPs

Graph comparing Comcast transfer speeds at different times of day

…the problem was essentially gone. Comcast really did fix it. We were now able to measure our Comcast download speeds in megabytes/second instead of kilobytes.

According to Comcast, two primary changes were made:

  1. Comcast added more capacity for Cogent traffic. (Exactly as we suspected, the pipe was full.)
  2. Cogent made some unspecified changes to their traffic engineering.

Here’s where I have to give Comcast credit where credit is due: they really did care about this problem, and they really did work quickly to make it go away.

(One weird thing, though: I was so prepared for a total Comcast dead-end, so sure that Comcast would never even reply, let alone help, that this incredibly positive outcome made me feel suspicious: why me? Why was I able to get this corrected with an e-mail when Cogent couldn’t?

It felt like there was no way this should have worked. If I had to guess, I’d say it’s simple: in the middle of a serious ongoing debate over net neutrality, the last thing Comcast wanted to look like was a network-throttling bad guy in this blog post. But then again, maybe I’m still being too cynical — maybe they just saw a problem they hadn’t noticed and fixed it. (But really, did they really not notice that pipe was full until I asked? Surely there are network monitoring tools?) Frankly, I have to stop thinking about it, because I’ll never know. But no matter the reason, I’m very grateful: thanks for listening to us, Comcast.)

What Does This All Mean

I’d summarize it as follows:

  • The internet is fragile — and that’s pretty scary.

And while this story amazingly had a happy ending, I’m not looking forward to the next time we’re stuck in the middle of a peering dispute between two companies. It feels absolutely inevitable, all the more so now that net neutrality is gone. Here’s hoping the next time it happens, the responsible party is as responsive as Comcast was this time.

Check Our Work

All of our data, our data analysis scripts, and more, is available at this GitHub repository. You can even click the button in the readme and it will take you to a running JupyterHub notebook where you can play with the data yourself, live in your browser. If you find any insights, or mistakes, please let us know.

Posted at 2:45 pm 29 Comments
January 5th, 2018
Transmit iOS on iPad Pro

Hello. Here’s an update on Transmit iOS that I promise will not use the words “sunset” or “journey”.

Quick summary:

  • We are suspending the sale of Transmit iOS very soon
  • Revenue was not enough to cover development — we won’t sell something we can’t actively develop
  • This does not affect Transmit 5 for Mac. It’s doing extremely well
  • This also does not affect Coda iOS and Prompt iOS, both of which are still going strong
  • We really hope to bring it back someday in some form

Why?

Transmit for iOS always felt like an obvious addition to our lineup, but we never thought it made a ton of sense in the tightly-restricted world of iOS until Apple announced the “Share Sheet” for iOS 8. Finally, we thought, in addition to using Transmit iOS to upload/download whatever you need, you could easily (?) get data out of apps and send it to your favorite servers. So, in 2014, we built it!

I’m extremely proud of the finished product. I think we made an app that is beautiful, elegant, and extremely powerful — a really great way to manage files on iOS with a wide variety of server types.

For people who needed it,  Transmit iOS was truly much-loved…

★★★★★

Brilliant for sending photos to clients

As a photographer who has been using Transmit on a Mac for a very long time I find this to be the best FTP client on iOS devices for my needs. Not only does it work really with the iOS11 Files system but it can output straight from Lightroom CC too. Possibly the best bit is the sync between the Mac OS version which transfers all of you favourite server settings to this version and vice-versa. If all of that isn’t enough, Transmit has proved to be ultra-reliable on both the iPad and iPhone.
By dg28com · v1.3.9 · United Kingdom · 14 days ago

★★★★★

Way more than I expected for iOS

I’m a longtime user of Transmit on Mac OS. When I upgraded to v4 I decided to look into the iOS version (on iPhone 7 and iPad Air 2). Glad I did. Every feature that makes sense in the iOS environment is implemented here, and the interface is functionally identical to the Mac version. Even Panic Sync worked well. Impressed.
By Bikerbudmatt · v1.3.9 · United States · 10 days ago

★★★★★

Another winner from Panic

This app is easier to use, faster, better written and more effective at managing files on my NAS and servers than the vendor-supplied apps.

Btw it’s also beautiful.
By *** Diabl0 *** · v1.3.9 · United States · 2 months ago

…but, quite sadly, there just weren’t enough of these lovely people. 

Transmit iOS made about $35k in revenue in the last year, representing a minuscule fraction of our overall 2017 app revenue. That’s not enough to cover even a half-time developer working on the app. And the app needs full-time work — we’d love to be adding all of the new protocols we added in Transmit 5, as well as some dream features, but the low revenue would render that effort a guaranteed money-loser. Also, paid upgrades are still a matter of great debate and discomfort in the iOS universe, so the normally logical idea of a paid “Transmit 2 for iOS” would be unlikely to help. Finally, the new Files app in iOS 10 overlaps a lot of file-management functionality Transmit provides, and feels like a more natural place for that functionality. It all leads to one hecka murky situation.

Was the use case for this app too edge-casey or advanced? Did we overestimate the amount of file management people want to do on a portable device? Should we have focused more on document viewing capabilities? Maybe all of the above?

My optimistic take: we hope that as iOS matures, and more and more pro users begin to seriously consider the iPad as a legitimate part of their daily work routines, Transmit iOS can one day return and triumph like it does on the Mac.

In the meantime, we can now better focus on our other great apps, including Coda iOS (which, by the way, has full file management too) and Prompt.

What Next?

  1. We will soon remove Transmit iOS from sale. This is your last chance to purchase it — if you think it’ll be useful for you. It does a lot of useful things, even if it won’t be updated in the foreseeable future.
  2. Everyone that has Transmit iOS installed can use it on their devices in the foreseeable future. Plus, it should be easily re-downloadable from your App Store “Purchased” zone — at least until a hopefully-far-future iOS update breaks compatibility with it.
  3. We’ve posted Transmit iOS 1.3.9 which adds full iPhone X support! If you use Transmit iOS, make sure to grab that final update as quickly as possible, since auto-updates will stop when the app is removed from sale.
  4. We’ll keep Panic Sync working for Transmit iOS. There’s no reason for us to turn it off as long as the app continues to run!
  5. Finally, any customers who purchased Transmit iOS in the last 60 days or so should contact us — because that sucks. While Apple doesn’t provide us with the ability to provide you with a refund, we’ll do everything we can to help.

Thank You

For everyone who purchased, used, and enjoyed Transmit iOS during its existence, we thank you so sincerely! Here’s hoping the computing stars will align for its return in the future.

Posted at 12:48 pm 50 Comments
November 7th, 2017

Hello, Panic friends! Just a quick announcement:

All three of our iOS apps — Prompt, Coda and Transmit — have all been updated for iPhone X.

That means that you can now SSH into your servers, make quick web fixes, and transfer files to and fro, all while taking advantage of every single pixel of that crazy-nice 2436 × 1125 OLED display.

We hope you enjoy these free updates.

Posted at 2:21 pm 2 Comments
July 18th, 2017

 

It’s here.

Seven years after the first release of Transmit 4, our well-loved and widely-used macOS file transfer app, we sat down with an incredibly exhaustive list of ideas, and — this’ll sound like I’m exaggerating but I’m mostly sure I’m not — we did it all.

With one massive update we’ve brought everyone’s favorite file-transferring truck into the future with more speed, more servers, more features, more fixes, a better UI, and even Panic Sync. Everything from the core file transfer engine to the “Get Info” experience was rethought, overhauled, and improved.

It definitely wasn’t an easy task. It definitely took a while, especially with other apps to tend to as well. But this is what we do — we wouldn’t be here if it wasn’t for Transmit, and Transmit users deserve the best.

Eager to learn more? Head over to our Transmit 5 internet web-site to get started! (We’ll also be updating the Transmit 5 Library with information over time!)

After so many massive improvements, we think Transmit 5 will meet your file transfer needs and then some. But there’s always more to do, and we’ll be listening closely to your ideas and improving Transmit 5 often. (In fact, work on 5.0.1 is already underway!) Let us know how Transmit can help you do your job. This truck is your truck.

Of course, my deepest thanks and appreciation to all of the amazing people at Panic who made Transmit 5 happen. Starting with Wade and Will, of course, who have lived and breathed this app for a long time, but almost everybody at Panic was involved: Aaron, Heather, Ashur, Logan, Tim, Jesus, Steve, Patrick, Helen, Thomas, Greg, James, Neven, June, Noby and Kenichi all do critical work that made this app as good as it is. (Dan and Dave and Shaun were off working on something else but they still deserve thanks too!)

It’s pretty amazing that we’ve been working on the same app for 20 years, and it’s pretty amazing that it’s still exciting to release an update for it. Thank you so much for buying Transmit, supporting Panic, and allowing us to bring even more software into this world.

Enjoy it!

Get Transmit 5!!

Where to Download

Head over to the Transmit 5 web site and hit that big “Try it” button. Our trial version is also the full version; just purchase to unlock the full app.

Where to Buy

Just head over to the same Transmit 5 web site and hit “Buy it”. Your serial number will be delivered immediately and unlock all the features of Transmit 5.

(If you have Apple Pay, we highly recommend the “Buy With Apple Pay button” — it’s like magic!)

One thing you should know: Transmit 5 is on sale for $35 for one week only. If you want the cheapest price for Transmit 5 you should grab it now before it goes up to its new regular price of $45. (Yes, we’re finally accounting for inflation.)

FAQ

Q: How much is it?
A: For a limited time — one week — only $35. After that, $45.

Q: Is there an upgrade discount?
A: No, it’s one price for all customers. (Fun fact: it’s been seven years since we last charged for an update to Transmit!)

Q: Is it in the Mac App Store?
A: No, just from us. This allows us to distribute a demo which we think is extremely helpful for people considering Transmit. (We’ll constantly re-evaluate the Mac App Store, though, and hope to return.)

Q: How many computers can I use it on?
A: Up to five computers, as long as you are the only user. For multi-user licenses, contact us!

Q: What if I just bought Transmit 4?
A: We’ve got you covered! If you bought from us after June 1st (or maybe even a little earlier), grab your Transmit 4 serial number and go here. If you bought from the Mac App Store in that timeframe, please send us an e-mail and we’ll help you out.

Q: How can I sync my favorite servers to Transmit on another Mac?
A: Using Panic Sync! Just make an account in Preferences and you’re off and running. It’s free, secure, fast, simple, lets us sync your data with other Panic apps, and makes it easy for us to troubleshoot. You can read more about it here.

Q: How can I have transfer status always shown?
A: The new Activity View in Transmit 5 is designed to focus on what Transmit is doing at the moment. But if you want something you can always look at while you work, choose View > Show Activity Bar to show Transmit 4-like status bar at the bottom of the window.

Q: Anything new in Transmit Disk?
A: Actually, not much — Transmit 5 essentially ships with Transmit Disk 4 with a few updates. We’re currently figuring out what to do, given the inevitability that Apple will remove support for kernel extensions, which Transmit Disk relies on. But we have some ideas — we’ll keep you posted!

Q: Will it be seven years before the next major Transmit update?
A: I sure hope not!

Posted at 11:34 am 123 Comments
May 17th, 2017

Last week, for about three days, the macOS video transcoding app HandBrake was compromised. One of the two download servers for HandBrake was serving up a special malware-infested version of the app, that, when launched, would essentially give hackers remote control of your computer.

In a case of extraordinarily bad luck, even for a guy that has a lot of bad computer luck, I happened to download HandBrake in that three day window, and my work Mac got pwned.

Long story short, somebody, somewhere, now has quite a bit of source code to several of our apps.

Before I continue, three important points:

  • There’s no indication any customer information was obtained by the attacker.
  • Furthermore, there’s no indication Panic Sync data was accessed.
  • Finally, our web server was not compromised.

(As a reminder, we never store credit card numbers since we process them with Stripe, and all Panic Sync data is encrypted in such a way that even we can’t see it. Read more.)

The other important fact is that I feel like a monumental idiot for having fallen for this.

How did this happen?

Story

HandBrake had been nagging me for some time to install an update. I finally decided, for whatever reason, to do the update. There was a note in HandBrake’s update dialog that the incremental update was not available, and that I’d have to download an entirely fresh copy from their server. I didn’t think too much of this, as we’ve been in a similar situation with a broken Sparkle update channel once before (the worst).

So, I managed to download within the three day window during which the infection was unknown, managed to hit the one download mirror that was compromised, managed to run it and breeze right through an in-retrospect-sketchy authentication dialog, without stopping to wonder why HandBrake would need admin privileges, or why it would suddenly need them when it hadn’t before. I also likely bypassed the Gatekeeper warning without even thinking about it, because I run a handful of apps that are still not signed by their developers. And that was that, my Mac was completely, entirely compromised in 3 seconds or less.

By the time news broke of the HandBrake infection, git credentials had already been stolen from my Mac and used to clone several of our source code repositories, according to our logs.

As soon as I discovered the infection on my Mac, I disabled it, took the Mac out of commission, and we began the incredibly lengthy process of changing all of my passwords, rotating the relevant secret keys throughout our infrastructure, and so on, to re-lock our doors and hopefully prevent anything else from being stolen. The vast majority of these things were changed or rolled simply out of an abundance of caution — again, there’s no indication our web servers were compromised — but in this kind of a situation, you change all the locks.

Then, the forensics: we began combing through our logs to try to determine the extent of what was accessed which, to reiterate, we believe is limited to source code and personal data on my Mac. Thanks to good logging (thank you, James) we got a very complete picture. The method the attacker used prevented them from cloning all of our source code — they were making educated guesses at our repo names, one-by-one, which did not expose everything.

The source code theft was confirmed when we received an email from the attacker (with a few source code files attached as proof of the theft) demanding a large bitcoin ransom to prevent the release of the source code, which would “suffocate” our company, in their words. We’re working on the assumption that there’s no point in paying — the attacker has no reason to keep their end of the bargain.

And that brings us to today.

So…

When the dust settled, we sat down for a company all-hands meeting, and the conclusion was a little different than I originally expected.

Someone has a bunch of our source code. But does it really matter?

There are essentially three “worst case” scenarios we considered with our source being out there in somebody’s hands:

  • They build free, cracked version of our apps.
    Guess what — those already exist. You can already pirate our software if you want to pirate our software — but please don’t — so this doesn’t really change anything in that regard. Also, whatever “free” version of our apps that would come from this person are virtually guaranteed to be infected with malware.
  • They create malware-infected builds of our apps.
    This seems likely. Given the person’s entire MO was to infect a well-used Mac app with malware, it seems inevitable. But we will find them, and working directly with Apple, shut them down. To minimize your risk, never download a copy of one our apps from a source that is not us or the Mac App Store. We are going to be hyper-vigilant about the authenticity of downloads on our servers.
  • A competitor obtains this source to attempt to use it to their advantage in some way.
    The many Mac developers we’ve met over the years are fine, upstanding people. I can’t imagine any of them being this unethical, or even being willing to take the risk of us finding fingerprints of our code in theirs. And let’s not forget that — you guessed it — there’s a good chance any stolen source could have malware slipped into it.

Also, one important thought gave us some comfort:

With every day that passes, that stolen source code is more and more out-of-date.

This hack hasn’t slowed us down. That source is already missing a ton of fixes and improvements we committed over the last week alone, and six months from now it will be missing major critical new features. In short: it’s old and getting older.

At this point in our discussion, we even half-seriously considered releasing the source code ourselves — and when that idea was floated, and we realized there wouldn’t be any fallout (other than a lot of code questions!), that’s when we truly felt free.

Assistance

Within 24 hours of the hack, we were on the phone with two important teams: Apple and the FBI.

Apple rallied the right security people quickly to learn all they could about our situation. (They had, of course, already blocked the HandBrake-attached malware for the broader Mac population once it was discovered widely.) They walked us through the best way to roll our Developer ID and invalidate the old one, which we don’t think was leaked, but we’re being overly cautious. And more importantly, the right people at Apple are now standing by to quickly shut down any stolen/malware-infested versions of our apps that we may discover.

The FBI is actively investigating, so I can’t say anything more about that.

Together

We’ll be working overtime for the foreseeable future to keep an eye on this situation.

But we could also use your help.

If you see any cracked or otherwise unofficial versions of our apps in the wild, it’s safest to assume they are infected, and we ask that you please let us know. If you see our source show up somewhere, also let us know. And if you have information that could help with the investigation into this incident, definitely let us know.

The more we know, the more we can use every method available to us — legal, technical, you name it — to fix it.

Feel free to e-mail us or DM us on Twitter anytime — even if you just have questions. We’re here.

And as a reminder, never download one of our apps from a source that is not our website or the Mac App Store.

This has been a hard post to write. I hate that this happened. I kick myself every day for not paying attention to what I was doing; the tells were obvious in hindsight. It’s a good reminder though — no matter how experienced you might be with computers, you’re human, and mistakes are easily made. And even though this doesn’t affect our customers directly, we want to apologize that we’re even having to have this discussion with you.

We’ve been doing this 20 years because you keep us going every day — by buying our software, by giving us your good ideas, by telling your friends about us. You are the good in the world. So we’re going to do everything we can to rise above this and keep going even further — together.

Posted at 10:50 am 87 Comments