Jump to content
Sign in to follow this  
Andrew at MEA

What does it take to be an IoT engineer?

Recommended Posts

End Game in the IoT

It’s been a long autumn and winter effort completing development of a CAT-M1-based on-farm data logger for release in the southern Spring, now only two days away.

These product development sprints are a long litany of small crises that are surmounted and left behind in the rear-view mirror.

When you’re stuck in the middle of it, it feels like crawling over broken glass.

But we have early orders, some stock on the shelf with more next week, and rudimentary field trails conducted that have already thrown up a weird software bug in the ultra-low power state that this logger depends upon to keep energy consumption within budget. A software work-around nipped that in the bud before it got released into the wild.

Our marketing department have been on the road, talking up the benefits of a completely new Internet-connected data logger that’s attractively priced and robustly packaged. This new product offering has been enthusiastically embraced by our key agents.

Great technology is indistinguishable from magic.

In this case, much of the excitement would appear to be generated by the simplest part of this technological wizardry: the coloured light behind our Green Brain symbol. This indicates CAT-M1 network connectivity, data transfers and GPS fixes to the guys with the muddy boots working under primitive conditions out in the field.

834872259_20190816_165456(1).thumb.jpg.47bca34e2a7b15c2dc3886a0840671c5.jpg

This is MEA’s third IoT development in eighteen months, with two more key projects beginning as soon as this new technology is properly bedded down.

I’ll lead the product development team through the coming Spring and Summer, until my 67th birthday next March.

So, the real end-game in this IoT race is actually to invest the next generation of MEA engineers with the spirit of the sprint, a ‘can-do’ ethos and the sense that with hard work, anything is possible.

Share this post


Link to post
Share on other sites

Cash-flow in the IoT.

Even for a mature company like MEA, the expensive business of IoT product development can be a scary process.

It’s always a race against the clock, as one burns cash reserves against the promise of returns from early sales of new-to-market products.

More companies go broke from cash-flow crises than anything else.

Just a few weeks’ delay in product launch can invoke such a cash-flow crisis (unless you have extraordinarily-deep pockets).

This is especially true in a seasonal business such as irrigated agriculture – late to market can be disastrous. What might well have sold a month ago must now wait a further eleven months to be of interest to customers. If – that is – some competitor hasn’t gained the upper-hand by then…

However, when all that risk pays off, it’s a true delight.

Cash flow turns positive and management breathes a sigh of relief.

These last few weeks before product launch of MEA’s CAT-M1 on-farm IoT data logger have had all those elements of tension, suspense, crisis and fear. Last-minute software bugs, production issues, field trial feedback, creation of extra test jigs and finally, ramming product through the new production line – all these things added to the pressure.

As Engineering Director, it’s my job at these times is to stay outwardly calm and cheerful, buffering an engineering staff beset by technical problems from marketing staff beset by an agent network clamouring for product delivery.

Today – just a few weeks after product launch – we’re well on the way to selling out of our entire first production run of 100 units. Funds expended in their manufacture will now flow back into the company coffers.

Now we face new challenges to our cash reserves; we have to swing into full production, investing in greater numbers of units while seasonal demand lasts and before the rapidly dwindling existing stock of these new loggers runs dry.

Such is business.

1497510515_TablegrapeGBL-C.thumb.jpg.0e11ae53ef657c882b1258ac597c6f8f.jpg

A brand-new MEA solar-powered CAT-M1 data logger for soil moisture monitoring, deployed in table grapes in the Victorian ‘Riverina’ region. Neatly installed at the foot of a wooden trellis post (in the foreground), the GBL-C logger is safe from farm machinery and foxes and hares that chew exposed cables.

 

Share this post


Link to post
Share on other sites

Primitive upgrades in the IoT.

If the first casualty of war is truth, then the first casualty of product launch is confidence.

The harsh reality is that following product launch burgeoning deployments, the passage of time, the machinations of Mother Nature and the compounding of human errors will eventually throw up a bug; it’s a numbers game.

Such bugs end the ‘jubilation phase’ of a successful product launch and call in the ‘humiliation phase’.

In an ideal world there would be no in-field bugs causing customers and agents distress. Lengthy and private field trials over hundreds of farms in multiple crops right across the country would have shaken out all such gremlins.

In the real world of practical IoT engineering, such a leisurely approach to perfection is denied the product development team. Commercial, budgetary and market imperatives intrude.

The first customers become the beta testers.

MEA’s CAT-M1 IoT data logger launched just in time ­ – in early September 2019 ­­– and our prototype stock of 100 units was sold out within three weeks.

Now, that’s a great feeling!

A long and sustained period of intensive product development effort was rewarded by first-to-market status and demand outstripping supply.

As a consequence, the emphasis shifted immediately to boosting production to meet back-orders.

In the meantime, data flowed to Green Brain from 80 sites, and down in the MEA basement those of us in the product development team waited anxiously to learn our fate.

Had we out-witted Murphy’s Laws?

Nope!

Finally, after a lengthy silence from the modem manufacturer and increasingly strident demands in the engineering forums, the chip maker admitted to a firmware bug that’s been the chief cause of our lack of confidence; they acknowledged a combination of events that could lock out connection to the Internet through a failure to safely enter ultra-low power mode.

Weeks after our product launch, they released a new firmware version purporting to have fixed the bug.

Simultaneously, reports began to arrive back from the field showing product lock-up after weeks of perfect operation. Data just stopped flowing from a modest percentage of stations.

AAaaHHhRRrr!

More midnight oil, and we upgraded to the modem manufacturer’s latest firmware – incorporating the critical bug-fix – and we tightened up a few possible edge-cases while we were in there; anything to improve safety and fault recovery.

Here’s where over-air programming (OAP) would have allowed us to seamlessly upgrade all deployed systems without leaving the office.

That developmental luxury had been set aside under time-to-market pressures and slated for Version 2 release.

Without that remote upgrade facility, our techs had to load up the company truck and head out into the vineyards and orchards of our irrigated agricultural districts to begin the laborious task of individually and directly upgrading those first deployed units (photo, below)

The silver-lining in this debacle is that the engineering team now has a stronger case for developing an over-air programming facility in these new loggers, expensive though that development will be.

A week has passed, the truck and its weary tech are back in Adelaide and every deployed CAT-M1 logger is back on the air and bullet-proofed as best we know how.

New stock is due in next week and life will return to normal, whatever that is around here.

image.png.5722f3ada990c913adc9992c5a2fdc67.png

MEA's open-air Green Brain Logger upgrade station somewhere in rural Victoria.
Our reserve stock of 20 units – loaded with the latest firmware – is swapped into field sites and the older versions get to enjoy the sunshine on some local park bench while their heads are upgraded. Then another round begins, with the attraction of over-air upgrades rising by the kilometre.

Share this post


Link to post
Share on other sites

Frustrations in the IoT.

It’s a day marked by high winds and even higher temperatures, with the ‘Fire Danger Index’ ratcheted over to ‘Catastrophic’ and 14 bushfires burning throughout South Australia.

All this is perfect for testing worst-case conditions for an ugly problem that’s taken nearly two months to resolve; ‘noise’ in our CAT-M1 IoT loggers when making ac resistance measurements in soil moisture tension sensors called ‘gypsum blocks’, invented nearly eighty years ago.

Frustratingly, the CAT-M1 network is down again; we later find that Telstra are installing a 5G network in the area and hence the intermittent LTE-M service. I’m having to bite my nails and trust in the data logging function to record the performance of this upgrade versus the faulty ‘control loggers’ mounted alongside it in the MEA Test Yard.

‘Noise’ to an irrigator looking at his soil moisture data  means data bouncing about in some unseemly fashion that frustrates easy interpretation and that destroys confidence in the equipment.

‘Noise’ to an electronics engineer means electrical noise, and I’ve ploughed through endless measurements chasing elusive sources of spikes and other artefacts that can fool modern analog-to-digital converters. Fixes, patches, filters, firmware changes, more careful grounding – nothing makes any difference to the lousy data spewing forth.

I’ve worked on this type of ac measurement through four-generations of gypsum block loggers. The problem is to generate a pure sinewave at the lowest possible cost. In this latest evolution, I’ve managed the sinewave generation, gypsum block excitation, gain block and full-wave rectification in a single quad op-amp. But it’s not working under field conditions…

Finally, I push my test regime up beyond the 53° limit of our ancient environmental oven and the problem shows itself at 64°; it’s temperature-related, and being exacerbated in this new logger operating up to 70° thanks to its built-in solar panel. The digital circuitry doesn’t care, but the analog circuitry does. The sinewave collapses with temperature, and what looks like noise is actually a quantization error brought on by collapsing range, despite being held in check by proper ratiometric measurements against internal reference resistors.

My fix of four extra resistors works in this worst of hot weather, and as a bonus I find that I can now run these sensors over tens of meters of cable; handy for odd deployments of gypsum blocks on particular farms. Sensor output is rock solid.

Now MEA need only endure the product recall of the dozen units released to beta customers.

Sure, I could have generated that sinewave with a digital-to-analog converter for $20, and I’ve done that in the past. But my 20-cent solution will serve the company better as we strive for a mass-market, riding on the coat-tails of the IoT wave.

In the immortal words of Arthur M. Wellington “An engineer can do for a dollar what any fool can do for two.”

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
You are posting as a guest. If you have an account, please sign in.
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Sign in to follow this  

×
×
  • Create New...