The pMD Blog

Welcome to the
pMD Blog...

where we cover interesting and relevant news, insights, events, and more related to the health care industry and pMD. Most importantly, this blog is a fun, engaging way to learn about developments in an ever-changing field that is heavily influenced by technology.

Weekly Byte: Vigilance and the Difference Four Lines Can Make
One of the great promises of modern medical technology is the ease of portability of your own medical data. Regardless of which physician you see as a patient, they should have your up-to-date medical history. This is achieved by either sharing a single system among all your providers or allowing many different systems to exchange information, hence creating an illusion of a single system. Yet when there are copies of your data across different systems, this illusion of a single system falters when the messages are not exchanged in real time. If it takes five minutes for an update to your medical record to percolate through the web of systems in your community, it may or may not be ‘OK’ depending on the urgency of your needs. As a software developer, five minutes would be an eternity and points to something that can be done better.

A core principle at pMD is that interoperability with other systems is a crucial ingredient to improving health care. As a result, we never charge for interfaces. These interfaces can be between pMD and EHRs, practice management systems, call centers, and other Health Information Exchanges (HIE). They exchange anything from demographics, scheduling, medical results, to billing information. The list and sophistication is constantly expanding. With our growing user base, the sheer volume of data we process has grown at an accelerated rate. As of today, we’re processing over a million messages every day.

Each message we send or receive can create numerous events that need to be calculated, and consequences to be recorded. Over the last year, we’ve seen this rapid growth of communication use up a growing portion of our computing power. In the last few months in particular, we noticed a significant rise in the average load on our application servers and increased latency on how long it takes to process some messages. Adding some sophisticated monitoring to our servers, we quickly identified the interfaces as the main culprit. The latency for processing server messages had become very volatile as seen by the chart below.

The concern was that if some messages took more than a second to process, the real-time nature of our data would suffer. Last week, we spent a significant amount of effort tracking down the issue. Setting up a large test bench, we set out to try to reproduce the peaks in latency we were seeing in our production system. Yet there were no individual messages that would result in significant delays; everything processed in our test system in a snappy fashion, usually in sub-second time. The problem then was systemic, something that grew with volume in a non-linear fashion. Writing some quick and dirty scripts to send the same handful of messages in an infinite loop from multiple threads, we finally reproduced the CPU performance we were seeing in our real systems.

From there, it was just a matter of determining which lines of codes cumulatively took the most amount of time and impacted our memory management system. After some sleuthing, we determined that the slowdown was due to the repeated re-computation of some seemingly benign variable. Caching it would solve it -- and it did. The chart below shows the average of 300 percent improvement of our interface processing, and in fact, entire application service load after the fix compared to some historical averages

The ratio of lines of code software developers write to their impact is extremely rewarding. The caching of this variable took a mere four additional lines of code but it helped preserve the promise of data portability for our customers and their patients. It was very satisfying. The experience reminded us of the importance to hold firm our standards for latency because beyond some dry technical metric, it is a critical component to realizing the vision of a modern medical system and better patient care.
5G ACA Account Management ACI adoption Advancing Care Information AHCA AHRQ AI Alternative Payment Model Amazon S3 Android API Apple appointment reminders Artificial Intelligence arts Audit Bay Area BCRA Best Place to Work Best Practices Big Data bootcamp BPCI BPTW Bundled Payments Business relationships Care Communities care coordination Care Navigation Care Team Career Fair CDC Charge Capture Charge Capture App Charge Capture Software Charge Capture Solution Charge Capture Statistics Charge Lag CHIP CIO Clinical Communication clinical data Clinical Data Registry Clinically integrated network Cloud CMS cms regulations Collaboration Communication Company Culture Conferences Connected Health Record contact practice Coronavirus COVID-19 COVID19 Cross-functional culture custom reports Customer Interaction customer relations customer service customer success customer support Customers Cyber Security cybersecurity Developer DHHS Dialysis discharge instructions EHR elderly Electronic Charge Capture Encryption epidemic Expansion FDA FHIR Flu fundamentals GI GI Outlook Goals group messaging Health Care Health Care Technology Health Care Web health data Health Information Exchange Health Record Healthcare healthcare interface healthcare interface integration healthcare software HIE HIPAA hipaa compliant communication hipaa compliant communication platform HIPAA-Compliant HL7 Home dialysis Hospital Census Hospital Communication humanity IA ICD-10 ICD-11 Implementation Improvement Activities in-app calling Instant capture Integration interface interoperability iOS iOS 8 iOS7 iPad iPhone iPhone 6 Lead Generation length of stay Long Term Care LTC Machine learning MACRA Medaxiom Medical Billing Medical Errors Medical Software Medicare Mental Health Mentorship Messaging Messaging with Patients MGMA MIPS MIPS Registry mobile Mobile App Mobile Charge Capture Mobile EHR Mobile Health Mobile Messaging Mobile security Mobile Technology Modern Healthcare Native App Network new feature new features news NIH OCM onboarding Oncology Care Model operations opioid crisis Pandemic Parenting partnerships Patient Patient access Patient App Patient Care Patient Chat Patient chat routers Patient Communication patient data Patient Engagement patient experience Patient Generated Health Data Patient Handoff Software Patient Information Patient Messaging Patient Safety patient support Patient Visits Patient-Centric PCP Performance bonus pmd pMD Team population health Population Health Management PQRS Product Development Productivity products Promoting Interoperability QCDR QPP quality reporting Quality Scoring Recruiting Recruitment Reimbursement remote jobs Remote Work reports Residents Responsibility ROI Rural communities Sales Secure Communication Secure Data Secure Messaging Secure Messaging Video Secure Text Messaging Secure Video Secure Video Chat Security Audit self care seniors SF Biz Times SNF SOC 2 software vendor Specialty Care success support TCM Team culture Teamwork technology telehealth Telemedicine telemedicine in long term care Telemedicine skilled nursing facilities Teletherapy Text Messaging texting training Travel Upgrades UX Design VA Value-based care Video Calling Video chat video communication video conferencing Virtual Care Virtual Visit voice calling voice memos Wearable Device wearing many hats Web App wellness wfh Wireless Work From Home Work Life Balance