Jump to content
Sign in to follow this  
Andrew at MEA

What does it take to be an IoT engineer?

Recommended Posts

End Game in the IoT

It’s been a long autumn and winter effort completing development of a CAT-M1-based on-farm data logger for release in the southern Spring, now only two days away.

These product development sprints are a long litany of small crises that are surmounted and left behind in the rear-view mirror.

When you’re stuck in the middle of it, it feels like crawling over broken glass.

But we have early orders, some stock on the shelf with more next week, and rudimentary field trails conducted that have already thrown up a weird software bug in the ultra-low power state that this logger depends upon to keep energy consumption within budget. A software work-around nipped that in the bud before it got released into the wild.

Our marketing department have been on the road, talking up the benefits of a completely new Internet-connected data logger that’s attractively priced and robustly packaged. This new product offering has been enthusiastically embraced by our key agents.

Great technology is indistinguishable from magic.

In this case, much of the excitement would appear to be generated by the simplest part of this technological wizardry: the coloured light behind our Green Brain symbol. This indicates CAT-M1 network connectivity, data transfers and GPS fixes to the guys with the muddy boots working under primitive conditions out in the field.

834872259_20190816_165456(1).thumb.jpg.47bca34e2a7b15c2dc3886a0840671c5.jpg

This is MEA’s third IoT development in eighteen months, with two more key projects beginning as soon as this new technology is properly bedded down.

I’ll lead the product development team through the coming Spring and Summer, until my 67th birthday next March.

So, the real end-game in this IoT race is actually to invest the next generation of MEA engineers with the spirit of the sprint, a ‘can-do’ ethos and the sense that with hard work, anything is possible.

Share this post


Link to post
Share on other sites

Cash-flow in the IoT.

Even for a mature company like MEA, the expensive business of IoT product development can be a scary process.

It’s always a race against the clock, as one burns cash reserves against the promise of returns from early sales of new-to-market products.

More companies go broke from cash-flow crises than anything else.

Just a few weeks’ delay in product launch can invoke such a cash-flow crisis (unless you have extraordinarily-deep pockets).

This is especially true in a seasonal business such as irrigated agriculture – late to market can be disastrous. What might well have sold a month ago must now wait a further eleven months to be of interest to customers. If – that is – some competitor hasn’t gained the upper-hand by then…

However, when all that risk pays off, it’s a true delight.

Cash flow turns positive and management breathes a sigh of relief.

These last few weeks before product launch of MEA’s CAT-M1 on-farm IoT data logger have had all those elements of tension, suspense, crisis and fear. Last-minute software bugs, production issues, field trial feedback, creation of extra test jigs and finally, ramming product through the new production line – all these things added to the pressure.

As Engineering Director, it’s my job at these times is to stay outwardly calm and cheerful, buffering an engineering staff beset by technical problems from marketing staff beset by an agent network clamouring for product delivery.

Today – just a few weeks after product launch – we’re well on the way to selling out of our entire first production run of 100 units. Funds expended in their manufacture will now flow back into the company coffers.

Now we face new challenges to our cash reserves; we have to swing into full production, investing in greater numbers of units while seasonal demand lasts and before the rapidly dwindling existing stock of these new loggers runs dry.

Such is business.

1497510515_TablegrapeGBL-C.thumb.jpg.0e11ae53ef657c882b1258ac597c6f8f.jpg

A brand-new MEA solar-powered CAT-M1 data logger for soil moisture monitoring, deployed in table grapes in the Victorian ‘Riverina’ region. Neatly installed at the foot of a wooden trellis post (in the foreground), the GBL-C logger is safe from farm machinery and foxes and hares that chew exposed cables.

 

Share this post


Link to post
Share on other sites

Primitive upgrades in the IoT.

If the first casualty of war is truth, then the first casualty of product launch is confidence.

The harsh reality is that following product launch burgeoning deployments, the passage of time, the machinations of Mother Nature and the compounding of human errors will eventually throw up a bug; it’s a numbers game.

Such bugs end the ‘jubilation phase’ of a successful product launch and call in the ‘humiliation phase’.

In an ideal world there would be no in-field bugs causing customers and agents distress. Lengthy and private field trials over hundreds of farms in multiple crops right across the country would have shaken out all such gremlins.

In the real world of practical IoT engineering, such a leisurely approach to perfection is denied the product development team. Commercial, budgetary and market imperatives intrude.

The first customers become the beta testers.

MEA’s CAT-M1 IoT data logger launched just in time ­ – in early September 2019 ­­– and our prototype stock of 100 units was sold out within three weeks.

Now, that’s a great feeling!

A long and sustained period of intensive product development effort was rewarded by first-to-market status and demand outstripping supply.

As a consequence, the emphasis shifted immediately to boosting production to meet back-orders.

In the meantime, data flowed to Green Brain from 80 sites, and down in the MEA basement those of us in the product development team waited anxiously to learn our fate.

Had we out-witted Murphy’s Laws?

Nope!

Finally, after a lengthy silence from the modem manufacturer and increasingly strident demands in the engineering forums, the chip maker admitted to a firmware bug that’s been the chief cause of our lack of confidence; they acknowledged a combination of events that could lock out connection to the Internet through a failure to safely enter ultra-low power mode.

Weeks after our product launch, they released a new firmware version purporting to have fixed the bug.

Simultaneously, reports began to arrive back from the field showing product lock-up after weeks of perfect operation. Data just stopped flowing from a modest percentage of stations.

AAaaHHhRRrr!

More midnight oil, and we upgraded to the modem manufacturer’s latest firmware – incorporating the critical bug-fix – and we tightened up a few possible edge-cases while we were in there; anything to improve safety and fault recovery.

Here’s where over-air programming (OAP) would have allowed us to seamlessly upgrade all deployed systems without leaving the office.

That developmental luxury had been set aside under time-to-market pressures and slated for Version 2 release.

Without that remote upgrade facility, our techs had to load up the company truck and head out into the vineyards and orchards of our irrigated agricultural districts to begin the laborious task of individually and directly upgrading those first deployed units (photo, below)

The silver-lining in this debacle is that the engineering team now has a stronger case for developing an over-air programming facility in these new loggers, expensive though that development will be.

A week has passed, the truck and its weary tech are back in Adelaide and every deployed CAT-M1 logger is back on the air and bullet-proofed as best we know how.

New stock is due in next week and life will return to normal, whatever that is around here.

image.png.5722f3ada990c913adc9992c5a2fdc67.png

MEA's open-air Green Brain Logger upgrade station somewhere in rural Victoria.
Our reserve stock of 20 units – loaded with the latest firmware – is swapped into field sites and the older versions get to enjoy the sunshine on some local park bench while their heads are upgraded. Then another round begins, with the attraction of over-air upgrades rising by the kilometre.

Share this post


Link to post
Share on other sites

Frustrations in the IoT.

It’s a day marked by high winds and even higher temperatures, with the ‘Fire Danger Index’ ratcheted over to ‘Catastrophic’ and 14 bushfires burning throughout South Australia.

All this is perfect for testing worst-case conditions for an ugly problem that’s taken nearly two months to resolve; ‘noise’ in our CAT-M1 IoT loggers when making ac resistance measurements in soil moisture tension sensors called ‘gypsum blocks’, invented nearly eighty years ago.

Frustratingly, the CAT-M1 network is down again; we later find that Telstra are installing a 5G network in the area and hence the intermittent LTE-M service. I’m having to bite my nails and trust in the data logging function to record the performance of this upgrade versus the faulty ‘control loggers’ mounted alongside it in the MEA Test Yard.

‘Noise’ to an irrigator looking at his soil moisture data  means data bouncing about in some unseemly fashion that frustrates easy interpretation and that destroys confidence in the equipment.

‘Noise’ to an electronics engineer means electrical noise, and I’ve ploughed through endless measurements chasing elusive sources of spikes and other artefacts that can fool modern analog-to-digital converters. Fixes, patches, filters, firmware changes, more careful grounding – nothing makes any difference to the lousy data spewing forth.

I’ve worked on this type of ac measurement through four-generations of gypsum block loggers. The problem is to generate a pure sinewave at the lowest possible cost. In this latest evolution, I’ve managed the sinewave generation, gypsum block excitation, gain block and full-wave rectification in a single quad op-amp. But it’s not working under field conditions…

Finally, I push my test regime up beyond the 53° limit of our ancient environmental oven and the problem shows itself at 64°; it’s temperature-related, and being exacerbated in this new logger operating up to 70° thanks to its built-in solar panel. The digital circuitry doesn’t care, but the analog circuitry does. The sinewave collapses with temperature, and what looks like noise is actually a quantization error brought on by collapsing range, despite being held in check by proper ratiometric measurements against internal reference resistors.

My fix of four extra resistors works in this worst of hot weather, and as a bonus I find that I can now run these sensors over tens of meters of cable; handy for odd deployments of gypsum blocks on particular farms. Sensor output is rock solid.

Now MEA need only endure the product recall of the dozen units released to beta customers.

Sure, I could have generated that sinewave with a digital-to-analog converter for $20, and I’ve done that in the past. But my 20-cent solution will serve the company better as we strive for a mass-market, riding on the coat-tails of the IoT wave.

In the immortal words of Arthur M. Wellington “An engineer can do for a dollar what any fool can do for two.”

Share this post


Link to post
Share on other sites

The IoT under the Microscope.

Terror stalks the floor at MEA: Christmas is only a fortnight away and sales orders for our new CAT-M1 data loggers are raining down from above.

But production has jammed: SD memory cards are failing to pass production testing and the engineer who designed this part of the circuitry is incommunicado somewhere in Europe.

Our production engineer is going quietly mad with fear and frustration.

I need to step in.

It’s now 45 years since I graduated in electronic engineering from the South Australian Institute of Technology so these moments of terror are nothing new, though their impact never seems to diminish.

At these times I follow a standard routine to bring the young engineers through the crisis, while knowing full-well that I can no longer solve many of these problems myself. Designing IoT technology is a team effort. No single one of us on the product development team has all the skills to function alone.

So, I stay outwardly calm and clear some mental and physical space to sit down in a quiet place with our production engineer. I ask to be walked through the history of the problem. Then we take a look at all the circuit schematics and relevant data sheets.

It’s not that I am here to fix the problem myself, but merely to act as a mirror and a sounding board, asking penetrating questions if I find a weakness in the fabric of the case and quietly letting these talented youngsters solve the problem themselves.

The problem itself is simple enough. MicroSD memory cards that worked in previous batches and previous products don’t work in this new product though circuitry remains the same. Even part numbers remain unchanged. Worse, devices from the same manufacturer work from one source but not from another. We scour the computer stores around Adelaide, buying up small handfuls of different memory cards for testing, then ordering up many hundreds of the apparent successes from warehouses interstate. These then fail on arrival to work at all.

Nothing makes any sense.

Just for something to say, I ask to look at the schematic for the jellybean ESD protection part that protects the memory card from damage from static discharge during installation.

Whoa!

I’m no digital genius, but us old analog engineers recognise a low-pass filter when confronted by one; this thing is sitting on the memory card data bus, as it has done in previous products from this modem manufacturer, and has never before caused problems. I ask for it to be removed and tracks bridged over.

Suddenly, all the lights come on and even our worst-case memory cards start passing test.

Once again hardware re-work is needed. This is only possible thanks to a beautiful new microscope, beloved of all of us aging techos squinting at parts having twelve legs in the space of tiny resistors that have only two.

20191211_103117.thumb.jpg.41f0d0509af36eb8cbe64ba460a5cf5a.jpg

The production line grinds back into life and filling back-orders before Christmas is once again a possibility.

Off to the side, we rotate our techs through the microscope desk, laboriously upgrading valuable PCB assemblies with this new fix then feeding them into production.

Should I be feeling professionally remiss that this happened at all?

Nah! Software engineers roll out fixes seemingly forever.  Scaling up production in the IoT will inevitably produce more of these moments of terror, allowing me to invoke yet one more old adage:

“When the going gets tough, the tough get going!”

  • Like 1

Share this post


Link to post
Share on other sites

Tripwires in the IoT.

Once more to the barricades…

MEA’s new CAT-M1 IoT data loggers – like the majority of Australians – have their own SIM cards and (a meagre 3Mb) data plan.

But with the production line rolling and the number of deployed loggers rising inexorably some way of keeping these bulk data plans in check proved critical. Fortunately, specialist companies provide boxes of 100 SIMs and platforms for generating alerts when things go awry.

Just weeks into the new decade we hit a tripwire: a significant number of loggers are exceeding their data allowance while draining batteries.

At the same time, data flow inexplicably becomes bumpy from all these new loggers. Customer complaints hit our agents who hit our marketing folk who hit up product development for answers and action.

What’s going on?

Once again, we huddle in corners and pore over screens and flowcharts and circuit schematics.

Within the week the source of all these woes comes to light; two hundred loggers locked to UTC time hit up Green Brain at exactly the same moment and jam the Green Brain server CPU to 100%. Late comers who can’t get through and deliver their data get shrugged off and must try again, at the expense of a finite energy budget or no luck at all before communication attempts time out four minutes later.

This feels like a denial of service attack!

But we’re on Google’s IoT platform and the Internet is supposed to be infinitely elastic, surely? There’s no scalability if we are saturating with deployments still down in the hundreds…

It turns out that this problem is something that never occurred under our older Plexus ftp data transfer systems; those were solidly buffered. Green Brain Loggers use newer https secure data transfers. It transpires that Green Brain runs each logger’s database interaction to the bitter end before moving to the next caller. Buried in there is a ‘backup-to-server’ piece of code that’s taking forever. Things only get worse as the day rolls on and the files to be retrieved and appended just get bigger.

The Green Brain boys excise that redundant piece of backup code and once again we’re back on track. CPU activity drops back into the normal range and deployed systems get through to the mothership with metronomic certainty.

Phew!

But something good has come from all this; we’ve discovered a source of universal ‘roaming’ international SIM cards for our CAT-M1 loggers. We get samples in from the USA and they work exactly as described.

Green Brain loggers can now be deployed anywhere in the world after set-up and test in Australia. They simply log onto whatever telco network is providing the strongest CAT-M1 signal when they arrive on site.

Prices are good, service is great and – best of all – we can set up our own trip wires to generate alerts and track deployments and activity on-line.

 

20200204_164353.thumb.jpg.ae93ed9d63ba8033b03c432b61159040.jpges t

Waiting patiently for the trip wires to trip...

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
You are posting as a guest. If you have an account, please sign in.
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Sign in to follow this  

×
×
  • Create New...