Super-Weird DSL Connectivity Issue

From A Certain Point of View
Donator V3.0
Parallax Abstraction's picture
Location: Ottawa, Ontario, Canada

So I've got this really annoying problem. I'm doing some work for a local guy who sells and services a popular PC based restaurant point of sale system. This system has the capability to process credit and debit cards through the software. At certain restaurants, a problem has recently developed with debit transactions. A very few of them don't process correctly. The charge will authorize but the software will process the sale as a cash transaction and this will screw up how it calculates tips for the servers. The debit card authorization has to be processed in a very precise manner in order to meet security compliance. If even a slight part of this isn't carried out properly, it can cause problems.

The software vendor's theory is that during certain transactions, small parts of the data stream are being lost and confusing the database. Our customer's connections never go down so this could be as small as a few lost packets. We've spoken to some other Ontario dealers of this product and they all claim that when they've had this happen, all of the affected customers were on Bell Sympatico DSL and that switching the customers off Bell solved the issue. I won't go into another rant about Bell here but I've seen this kind of thing before with their horrendous DSL service. The problem is that most of our customers who are using them are doing so because they have the service bundled with their phone lines. They will not give up this bundle without a very good reason but they are also not prepared to have this problem continue.

The only way we're going to be able to convince them of this is to show them evidence that Bell is at fault. Getting Bell to admit any fault is a pointless effort so that's not our goal here. I'm wondering if anyone knows of anything we can do to test the connection while processing debit transactions to see if there is evidence of our theory. I have though of the dslreports.com line monitoring tool and using the Wireshark software but the former isn't really made for tests this specific and the latter will require hours and hours of setup and detailed analysis and I don't know if the guy I'm working for will be willing to pay me for the time necessary, just to prove that the customer needs to switch ISPs. I've been thinking of more efficient ways to do this but can't come up with anything.

Any ideas on other ways I can try this? Thanks all.

"Just because something's popular, that sure doesn't make it right." -Penn Gilette
"You can't fix stupid." -Ron White
blog.digital-lifeline.ca

Unprncbl
Donator
Duoae's picture

Hmm. Do you think this could be part of the network throttling Bell is currently doing? Could their software be reducing traffic by eliminating 'non-vital' packets?

I don't have any advice on fixing it but perhaps you could get in touch with the NUPGE or one of the campaigns currently underway (links from the ars article below) and get the customers to do the same? The only way Bell would admit to the problem is by overpowering customer revolt and outcry.

Good luck!

Ars-technica article

NUPGE page

Campaign to stop the throttling

Of - power - insessantly
Plagued - by - malefisense
Doomed - to - insidious -
Death - is - he - who - breaks
this - monument - i - prophesy

From A Certain Point of View
Donator V3.0
Parallax Abstraction's picture
Location: Ottawa, Ontario, Canada

This has been happening since well before the traffic shaping according to the guy I work for. He has the software vendor and the customer's payment service working on this and it looks like he wants to hold off pursuing the issue further until he gets an answer from them. We'll see what happens.

"Just because something's popular, that sure doesn't make it right." -Penn Gilette
"You can't fix stupid." -Ron White
blog.digital-lifeline.ca

they charge per letter
pol's picture
Location: Charlottesville, VA

Quote:
They will not give up this bundle without a very good reason but they are also not prepared to have this problem continue.

Welcome to small business consulting!

In all honesty though I think what you have done here is to take responsibility for something that is beyond your scope. You can't be expected to know everything, and you certainly can't be expected to support anything a client might happen to have issue with. This should be sorted out by the software vendor. If your client would like to pay to have you to deal with the software vendor, then thats great, and you can probably expedite the process greatly because you can speak the language. But by no means can you take responsibility for the functionality of the system.

If you approach the problem from that point of view, then your client already has their answer. They have been told by the vendor that they should switch ISP's if they want the issue resolved. The burden of proof here should not be yours

*censored*
Donator
doihaveto's picture
Location: SF, CA

To test the "lost packet" theory, you'd really have to set up Wireshark at both ends - the POS client and the processing server- and then compare the logs. I somehow doubt the payment processor will let you do that.

In the long term, you can try leaning on the software vendor to start using standard error-correcting protocols (eg. TCP) if their communication is going to go through unreliable networks. But that's of little help in the short term...

Office Linebacker
KingGorilla's picture

Can we back up a moment. Did I understand the situation correctly? Are there computers that handle financial transactions with credit and debit cards, connected to the internet via DSL?

From A Certain Point of View
Donator V3.0
Parallax Abstraction's picture
Location: Ottawa, Ontario, Canada

It's very commonplace. Most of the pin-pads you run your debit through at stores are connected to an Ethernet network which does the authorizations online. Everything's done over SSL of course and it is very secure but yes, most small businesses process debit and credit through the Internet.

I get that this shouldn't be our problem but as my employer is a small business and is the first line of support (our customers don't talk to the software vendor), we have to deal with it. Those who aren't on an SLA with us are paying us to deal with it. The problem is that the software vendor isn't the one saying to drop Bell, other dealers are and the evidence is all anecdotal. There's no documentation saying this works, just these other guys say it did. We have roped both the software vendor and Moneries (the payment provider) into the mix, basically saying "You two figure this out and let us know." If they confirm the problem, we are going to tell these customers to either switch ISPs or they lose support for the issue.

"Just because something's popular, that sure doesn't make it right." -Penn Gilette
"You can't fix stupid." -Ron White
blog.digital-lifeline.ca

*censored*
Donator
doihaveto's picture
Location: SF, CA

Oh wait, if your traffic goes through SSL, then it's going through TCP already. And TCP will detect and recover from packet loss.

The software vendor's theory is starting to sound a little fishy...

Unprncbl
Donator
Duoae's picture

doihaveto wrote:
Oh wait, if your traffic goes through SSL, then it's going through TCP already. And TCP will detect and recover from packet loss.

The software vendor's theory is starting to sound a little fishy...

I just read an article on Ars saying that there was evidence that SSH has been affected. I know that SSL and SSH are different (i read up about them on wikipedia before posting this) but their principles don't seem too far removed from one another. Could evidence of SSH problems arising from throttling also implicate a link to throttling for problems with SSL?

Of - power - insessantly
Plagued - by - malefisense
Doomed - to - insidious -
Death - is - he - who - breaks
this - monument - i - prophesy

Office Linebacker
Donator V4.0
Location: Minneapolis

I'm extremely skeptical that it would be dropping enough random packets (and not requesting resends) to cause occasional problems and not enough that it flat-out doesn't work. If they were restricting SSL in a way that caused it to just randomly fail, then all of their customers who were buying things on the internet, doing online banking, etc. would be having problems.

I think you're likely barking up the wrong tree here.

1 Perk Every 1000th Post
Donator V4.0

Yeah, TCP/IP eliminates the issue of packet loss. Another vote for wrong tree barking.

From A Certain Point of View
Donator V3.0
Parallax Abstraction's picture
Location: Ottawa, Ontario, Canada

Well, it turns out we won't have to figure this out. The customer got fed up with waiting for the software vendor to come up with a solution and we pulled out their entire setup this past week. Sucks a lot but my employer found someone to buy the equipment so it isn't all bad. I hate Bell so much.

"Just because something's popular, that sure doesn't make it right." -Penn Gilette
"You can't fix stupid." -Ron White
blog.digital-lifeline.ca