Jump to content

Andrew at MEA

Registered
  • Content Count

    101
  • Joined

  • Last visited

  • Days Won

    15

Everything posted by Andrew at MEA

  1. A Virus in the IoT. Geeks have got it good in this post-normal world: vulnerable old folk like me are sent home and told to stay away from the office. Are you kidding me? Home is where my garden is, my own personal sacred site that is at the root of my interest in agricultural technology and irrigation. Home is also where my favourite lab is. Here, I have two benches. One is set up with all the fancy test equipment I need to create and test MEA stuff. The second one (pictured below) is where I fix my eBay-acquired fine old test instruments from back in the 1960’s and ‘70s when I was undertaking my first electronic engineering degree. Fixing stuff hones one’s problem-solving skills*. Gear that worked once is a closed problem with a single solution: it should work again. The quality of thinking that went into these old knob-driven instruments is exactly that of today's Masters. Only the technology has changed, not the engineering mind-set. And finally, old test equipment is admirably accessible for fixing. It’s also beautifully documented – an education inside shabby covers. I spend as much money on old manuals as I do on Tektronix plug-ins. I particularly love the fold-out schematics which I pore over endlessly until I can finally understand how a circuit really works. Yesterday I was in one of my usual funks. In the past seven years I’ve worked on long-haul ZigBee, Bluetooth, satellite, 3G and CAT-M1 IoT products for agriculture: they’re all out there across Australia, contributing to the 3 billion+ data records up there in our Green Brain web app. Now what? The remedy for such a bad dose of the blues is a 1960’s day. So yesterday I played hooky from Slack, Trello, Gantt charts, emails, texts and phone calls back to the office and my far-flung staff. Instead, I repaired a wonderfully old 1968 Nixie tube 8-digit counter. The 15V supply regulator came back into regulation with the help of a new PNP control transistor. I threw out the old beryllium-filled oven-controlled crystal oscillator and hand-built a modern one (no purists around here – it just has to work!) I pin-pointed a temperature sensitive old TTL decade counter that was shutting down two of the eight time base channels in the ranging function. Today I awoke – back in my modern IoT world – with a fully-formed concept in my head for how to measure soil-moisture content in irrigated pot-plants in commercial tree nurseries. There's always someone wanting me to work magic for beer-money, and I’d set this problem aside for a rainy day. It seems that a day among the Nixie tubes is also sub-conscious therapy for an old brain taxed with new IoT problems. My 1960’s ‘funk-fixing’ bench, courtesy of Jim Williams and eBay *‘The Importance of Fixing’: Jim Williams in ‘The Art and Science of Analog Circuit Design’. EDN Series for Engineers. Butterworth-Heinemann
  2. Edge Cases in the IoT. Time’s dragging in the MEA labs; we’re grinding out the finishing touches to Version 2 of our GBL (CAT-M1) data logger and it’s painful. Ah, but it’s always like this! With some hundreds of these new loggers in service for the past six months, three units have been returned to the product development team exhibiting weirdness. Yet these pain-points are exactly what we must eliminate here and now, so we’re holding up production of future stock until we resolve these edge-case issues to our satisfaction. Future ugliness would be our certain fate if we let these glitches go feral. There are three issues, and it’s testing the ‘cures’ that’s taking up all our time and ratcheting up stress levels as we deal with technical uncertainties under a marketing and management cloud. Such is an engineer’s daily fare. To resolve Issue 1, we need a hardware safety mechanism to shed the load under low-battery conditions: small improvements to the battery-monitoring circuit and start-up logic fix this. We are also unconvinced that electronic eFuses fit-the-bill at the minuscule current levels found in IoT devices. We take the opportunity to toss those out and replace them with software-controlled electronic load switches instead. We tick this first box. Issue 2 fixes the ESD protection circuits of the SD Memory Card; turns out those old ESD devices just weren’t handling the higher-speed data transfers occurring with this most recent modem technology. The fix is hacked onto the PCB and works perfectly in the MEA test yard. But by now we’re rightly worried about higher-temperature operation because there is some indication that that’s what triggers the fault condition. We take our ancient environmental oven apart and update it to a digital PID temperature controller (available on eBay for $23) that allows us to soak test at 70°C ± 0.8°C then ramp down temperature to ambient. The new design checks out perfectly over the full temperature range while the old ‘control’ unit locks up at 48°C. The new design is approved as ‘tested’. Issue 3 arose through human error – a production-line failure to securely plug the solar panel into the motherboard on two units during final assembly. We tweak the production line process to catch this condition. Yet what we really need is sharper diagnostic tools for monitoring battery charge current during five different field scenarios. We design this into the hardware and firmware while we have the chance. This charge-current trace will allow our service team to differentiate between solar charging failures and faulty load conditions if battery voltage is declining. Once all boxes have been ticked and I sign off on changes, we’ll modify the motherboard schematic and PCB, change the Bill of Materials and generate the updated Gerber files. We’ll be good-to-go. But by now the COVID-19 pandemic has hit and the Aussie dollar is plunging against the US dollar that is our standard manufacturing exchange currency. Fortunately, nearly all components for this latest production run were paid for back when our currency was stronger; it’s only the PCB and assembly costs that have to be paid for in our weaker coin. Despite global uncertainties, forward is the only way for MEA to go: we have back-orders to fill. Our old environmental oven gets fitted with a new PID Temperature Controller. Looks ugly, and yet it’s strangely hot!
  3. Powering the IoT. A positive energy budget is a ‘sine qua non’ (absolutely essential condition) in an IoT field station. Early MEA data loggers deployed in agriculture were battery-powered affairs, but unless the battery capacity is adequate for 3-5 years the logistics of battery swap-outs – with their inherent risk of breaking environmental seals put in place in the factory – can increase user frustration and shorten product lifetime. Energy harvesting via solar panels extends product field life indefinitely; we now have thousands of IoT solar-powered sites still in service after up to seven years. Despite this wealth of experience, each new product release raises our collective anxiety for at least a solid year after field deployment. Canopy growth, bird netting, dirt build-up, high and low temperatures, white plastic rain covers over table grapes, seasonal changes in solar zenith angle, shorter winter days, periods of extended cloudy weather, poor installation, weak solar panels, excessive loads – all these things call out weaknesses in the energy budget. And so the arrival of eight CAT-M1 IoT field units back to MEA’s service department triggered immediate self-doubts and a heightened forensic investigation. In two of the units, the solar panel had come loose from the motherboard; this is a product assembly failure corrected by rapping knuckles and dreaming up post assembly tests. Four more worked perfectly as designed and operated for week after week in the test yard without a glitch; the only possible explanation would be unknown field conditions, such as canopy over-growth. But the signature lesson flowed from the last two units, highlighting a design flaw that can only occur if battery voltage falls below about 3.3V for a single-cell Lithium-Ion battery operating in the 3.6V to 4.2V range. The logger refuses to wake up. Our original intent had been to have Green Brain monitor battery voltages for all deployed units and to send commands to shed load and reduce logging rates to automatically balance the energy budget. Somehow, we never implemented that in the first release. So we do what we should have done in the first place – implement a hardware belt-and-braces solution to shed load based on battery voltage. Firmware gets tweaked as well. With the immediate problem solved, we re-examine our diagnostic tools. We need something better than just watching battery voltage traces, so we incorporate additional solar charge monitoring circuitry. ‘Gas-gauging’ the battery load would also be good using coulomb counters, but this proves difficult to implement quickly and at low cost. We defer that to Release 3. But we have one more trick to help us extend the energy budget; we can ship field stations at full charge while also testing the charging circuits between solar panel and battery. The first month’s operation is on us! We build a bulk charger capable of handling 120 logger boards at once (see photo). All this is made possible by diligent hunting on eBay for a low-voltage high-current power supply rated at 8V and 50A, all for $500. A custom-built ‘bulk charger’ for MEA’s CAT-M1 data loggers handles 30 PCBs at once, with an additional three charger racks still within the capacity of the low-voltage high-current Hewlett-Packard HP 6551A power supply. Full overnight charge takes about eight hours.
  4. One Small Anniversary in the IoT. At 3pm exactly a year ago today (19th Feb 2020) MEA transmitted the first 20 bytes of measurement data on the Myriota satellite network from the MEA Test Garden. Data consisted of four measurements of soil moisture tension in the soil profile below dry-grown Shiraz grapevines, plus battery voltage. After the first anxious few months – where I peered at the incoming data on a daily basis – a certain trust developed that data would keep flowing, and other developments took priority. Along the way though, I checked in on the growing Green Brain data set and sought explanations from the ever-helpful folk at Myriota about spikes, missing data records and long latencies. We were managing four to five readings per day via a single Myriota satellite in polar orbit. Our understanding of this new IoT technology crept forward with the seasons. Other test installations followed during March and April 2019, from stations monitoring tank water level, rainfall and micro-climate. A year later the soil moisture record is sufficiently consistent and detailed to track rain-fed irrigation suitable for sub-surface moisture measurements in dry-land agriculture, should we choose to exploit such a market.
  5. Tripwires in the IoT. Once more to the barricades… MEA’s new CAT-M1 IoT data loggers – like the majority of Australians – have their own SIM cards and (a meagre 3Mb) data plan. But with the production line rolling and the number of deployed loggers rising inexorably some way of keeping these bulk data plans in check proved critical. Fortunately, specialist companies provide boxes of 100 SIMs and platforms for generating alerts when things go awry. Just weeks into the new decade we hit a tripwire: a significant number of loggers are exceeding their data allowance while draining batteries. At the same time, data flow inexplicably becomes bumpy from all these new loggers. Customer complaints hit our agents who hit our marketing folk who hit up product development for answers and action. What’s going on? Once again, we huddle in corners and pore over screens and flowcharts and circuit schematics. Within the week the source of all these woes comes to light; two hundred loggers locked to UTC time hit up Green Brain at exactly the same moment and jam the Green Brain server CPU to 100%. Late comers who can’t get through and deliver their data get shrugged off and must try again, at the expense of a finite energy budget or no luck at all before communication attempts time out four minutes later. This feels like a denial of service attack! But we’re on Google’s IoT platform and the Internet is supposed to be infinitely elastic, surely? There’s no scalability if we are saturating with deployments still down in the hundreds… It turns out that this problem is something that never occurred under our older Plexus ftp data transfer systems; those were solidly buffered. Green Brain Loggers use newer https secure data transfers. It transpires that Green Brain runs each logger’s database interaction to the bitter end before moving to the next caller. Buried in there is a ‘backup-to-server’ piece of code that’s taking forever. Things only get worse as the day rolls on and the files to be retrieved and appended just get bigger. The Green Brain boys excise that redundant piece of backup code and once again we’re back on track. CPU activity drops back into the normal range and deployed systems get through to the mothership with metronomic certainty. Phew! But something good has come from all this; we’ve discovered a source of universal ‘roaming’ international SIM cards for our CAT-M1 loggers. We get samples in from the USA and they work exactly as described. Green Brain loggers can now be deployed anywhere in the world after set-up and test in Australia. They simply log onto whatever telco network is providing the strongest CAT-M1 signal when they arrive on site. Prices are good, service is great and – best of all – we can set up our own trip wires to generate alerts and track deployments and activity on-line. es t Waiting patiently for the trip wires to trip...
  6. The IoT under the Microscope. Terror stalks the floor at MEA: Christmas is only a fortnight away and sales orders for our new CAT-M1 data loggers are raining down from above. But production has jammed: SD memory cards are failing to pass production testing and the engineer who designed this part of the circuitry is incommunicado somewhere in Europe. Our production engineer is going quietly mad with fear and frustration. I need to step in. It’s now 45 years since I graduated in electronic engineering from the South Australian Institute of Technology so these moments of terror are nothing new, though their impact never seems to diminish. At these times I follow a standard routine to bring the young engineers through the crisis, while knowing full-well that I can no longer solve many of these problems myself. Designing IoT technology is a team effort. No single one of us on the product development team has all the skills to function alone. So, I stay outwardly calm and clear some mental and physical space to sit down in a quiet place with our production engineer. I ask to be walked through the history of the problem. Then we take a look at all the circuit schematics and relevant data sheets. It’s not that I am here to fix the problem myself, but merely to act as a mirror and a sounding board, asking penetrating questions if I find a weakness in the fabric of the case and quietly letting these talented youngsters solve the problem themselves. The problem itself is simple enough. MicroSD memory cards that worked in previous batches and previous products don’t work in this new product though circuitry remains the same. Even part numbers remain unchanged. Worse, devices from the same manufacturer work from one source but not from another. We scour the computer stores around Adelaide, buying up small handfuls of different memory cards for testing, then ordering up many hundreds of the apparent successes from warehouses interstate. These then fail on arrival to work at all. Nothing makes any sense. Just for something to say, I ask to look at the schematic for the jellybean ESD protection part that protects the memory card from damage from static discharge during installation. Whoa! I’m no digital genius, but us old analog engineers recognise a low-pass filter when confronted by one; this thing is sitting on the memory card data bus, as it has done in previous products from this modem manufacturer, and has never before caused problems. I ask for it to be removed and tracks bridged over. Suddenly, all the lights come on and even our worst-case memory cards start passing test. Once again hardware re-work is needed. This is only possible thanks to a beautiful new microscope, beloved of all of us aging techos squinting at parts having twelve legs in the space of tiny resistors that have only two. The production line grinds back into life and filling back-orders before Christmas is once again a possibility. Off to the side, we rotate our techs through the microscope desk, laboriously upgrading valuable PCB assemblies with this new fix then feeding them into production. Should I be feeling professionally remiss that this happened at all? Nah! Software engineers roll out fixes seemingly forever. Scaling up production in the IoT will inevitably produce more of these moments of terror, allowing me to invoke yet one more old adage: “When the going gets tough, the tough get going!”
  7. Frustrations in the IoT. It’s a day marked by high winds and even higher temperatures, with the ‘Fire Danger Index’ ratcheted over to ‘Catastrophic’ and 14 bushfires burning throughout South Australia. All this is perfect for testing worst-case conditions for an ugly problem that’s taken nearly two months to resolve; ‘noise’ in our CAT-M1 IoT loggers when making ac resistance measurements in soil moisture tension sensors called ‘gypsum blocks’, invented nearly eighty years ago. Frustratingly, the CAT-M1 network is down again; we later find that Telstra are installing a 5G network in the area and hence the intermittent LTE-M service. I’m having to bite my nails and trust in the data logging function to record the performance of this upgrade versus the faulty ‘control loggers’ mounted alongside it in the MEA Test Yard. ‘Noise’ to an irrigator looking at his soil moisture data means data bouncing about in some unseemly fashion that frustrates easy interpretation and that destroys confidence in the equipment. ‘Noise’ to an electronics engineer means electrical noise, and I’ve ploughed through endless measurements chasing elusive sources of spikes and other artefacts that can fool modern analog-to-digital converters. Fixes, patches, filters, firmware changes, more careful grounding – nothing makes any difference to the lousy data spewing forth. I’ve worked on this type of ac measurement through four-generations of gypsum block loggers. The problem is to generate a pure sinewave at the lowest possible cost. In this latest evolution, I’ve managed the sinewave generation, gypsum block excitation, gain block and full-wave rectification in a single quad op-amp. But it’s not working under field conditions… Finally, I push my test regime up beyond the 53° limit of our ancient environmental oven and the problem shows itself at 64°; it’s temperature-related, and being exacerbated in this new logger operating up to 70° thanks to its built-in solar panel. The digital circuitry doesn’t care, but the analog circuitry does. The sinewave collapses with temperature, and what looks like noise is actually a quantization error brought on by collapsing range, despite being held in check by proper ratiometric measurements against internal reference resistors. My fix of four extra resistors works in this worst of hot weather, and as a bonus I find that I can now run these sensors over tens of meters of cable; handy for odd deployments of gypsum blocks on particular farms. Sensor output is rock solid. Now MEA need only endure the product recall of the dozen units released to beta customers. Sure, I could have generated that sinewave with a digital-to-analog converter for $20, and I’ve done that in the past. But my 20-cent solution will serve the company better as we strive for a mass-market, riding on the coat-tails of the IoT wave. In the immortal words of Arthur M. Wellington “An engineer can do for a dollar what any fool can do for two.”
  8. Primitive upgrades in the IoT. If the first casualty of war is truth, then the first casualty of product launch is confidence. The harsh reality is that following product launch burgeoning deployments, the passage of time, the machinations of Mother Nature and the compounding of human errors will eventually throw up a bug; it’s a numbers game. Such bugs end the ‘jubilation phase’ of a successful product launch and call in the ‘humiliation phase’. In an ideal world there would be no in-field bugs causing customers and agents distress. Lengthy and private field trials over hundreds of farms in multiple crops right across the country would have shaken out all such gremlins. In the real world of practical IoT engineering, such a leisurely approach to perfection is denied the product development team. Commercial, budgetary and market imperatives intrude. The first customers become the beta testers. MEA’s CAT-M1 IoT data logger launched just in time ­ – in early September 2019 ­­– and our prototype stock of 100 units was sold out within three weeks. Now, that’s a great feeling! A long and sustained period of intensive product development effort was rewarded by first-to-market status and demand outstripping supply. As a consequence, the emphasis shifted immediately to boosting production to meet back-orders. In the meantime, data flowed to Green Brain from 80 sites, and down in the MEA basement those of us in the product development team waited anxiously to learn our fate. Had we out-witted Murphy’s Laws? Nope! Finally, after a lengthy silence from the modem manufacturer and increasingly strident demands in the engineering forums, the chip maker admitted to a firmware bug that’s been the chief cause of our lack of confidence; they acknowledged a combination of events that could lock out connection to the Internet through a failure to safely enter ultra-low power mode. Weeks after our product launch, they released a new firmware version purporting to have fixed the bug. Simultaneously, reports began to arrive back from the field showing product lock-up after weeks of perfect operation. Data just stopped flowing from a modest percentage of stations. AAaaHHhRRrr! More midnight oil, and we upgraded to the modem manufacturer’s latest firmware – incorporating the critical bug-fix – and we tightened up a few possible edge-cases while we were in there; anything to improve safety and fault recovery. Here’s where over-air programming (OAP) would have allowed us to seamlessly upgrade all deployed systems without leaving the office. That developmental luxury had been set aside under time-to-market pressures and slated for Version 2 release. Without that remote upgrade facility, our techs had to load up the company truck and head out into the vineyards and orchards of our irrigated agricultural districts to begin the laborious task of individually and directly upgrading those first deployed units (photo, below) The silver-lining in this debacle is that the engineering team now has a stronger case for developing an over-air programming facility in these new loggers, expensive though that development will be. A week has passed, the truck and its weary tech are back in Adelaide and every deployed CAT-M1 logger is back on the air and bullet-proofed as best we know how. New stock is due in next week and life will return to normal, whatever that is around here. MEA's open-air Green Brain Logger upgrade station somewhere in rural Victoria. Our reserve stock of 20 units – loaded with the latest firmware – is swapped into field sites and the older versions get to enjoy the sunshine on some local park bench while their heads are upgraded. Then another round begins, with the attraction of over-air upgrades rising by the kilometre.
  9. The IoT takes centre-stage at the World Electronics Forum (WEF) in Adelaide in early December 2019 The WEF is an annual meeting of Electronics Industry leaders from around the world. This is the first time since 2003 that the WEF has taken place in Australia and will be a major event for our industry, bringing together investors and CEOs from the world’s largest electronics companies. It is expected to create business links, collaboration and investment opportunities. LPWAN – Low Power Wide Area IoT Networks – is a central stream of the conference. This is your chance to hear from some of the key players in the Australian IoT scene: 1. Alex Grant of Myriota (satellite-connected IoT) 2. Adrian Tchordjallian of Thinxstra (Sigfox LPWAN) 3. Andrew Suttle of U-Blox (international IoT experience across many fields) 4. Andrew Skinner of MEA (AgTech IoT – long-range ZigBee, Bluetooth, 433 MHz LPWAN, 3G, CAT-M1, Myriota satellite) Exhibition spaces are available for electronic companies looking for a local venue displaying to an international audience. There are awards to be won for best product and young professional, plus others. How about being a speaker, selling Australian innovation to the world? See the web-site for details: - https://www.wef-adelaide.com Only seven (7) weeks to go – register now, apply for an award, exhibition space or a speaking slot.
  10. Hi Geoff Hoping this finds you well.. There are seven items of spam on two of our IoT Forum pages - looks like a failure of the EA Cyber Security system? Can you pull some strings and clean us up? Cheers for now Andrew at MEA
  11. Cash-flow in the IoT. Even for a mature company like MEA, the expensive business of IoT product development can be a scary process. It’s always a race against the clock, as one burns cash reserves against the promise of returns from early sales of new-to-market products. More companies go broke from cash-flow crises than anything else. Just a few weeks’ delay in product launch can invoke such a cash-flow crisis (unless you have extraordinarily-deep pockets). This is especially true in a seasonal business such as irrigated agriculture – late to market can be disastrous. What might well have sold a month ago must now wait a further eleven months to be of interest to customers. If – that is – some competitor hasn’t gained the upper-hand by then… However, when all that risk pays off, it’s a true delight. Cash flow turns positive and management breathes a sigh of relief. These last few weeks before product launch of MEA’s CAT-M1 on-farm IoT data logger have had all those elements of tension, suspense, crisis and fear. Last-minute software bugs, production issues, field trial feedback, creation of extra test jigs and finally, ramming product through the new production line – all these things added to the pressure. As Engineering Director, it’s my job at these times is to stay outwardly calm and cheerful, buffering an engineering staff beset by technical problems from marketing staff beset by an agent network clamouring for product delivery. Today – just a few weeks after product launch – we’re well on the way to selling out of our entire first production run of 100 units. Funds expended in their manufacture will now flow back into the company coffers. Now we face new challenges to our cash reserves; we have to swing into full production, investing in greater numbers of units while seasonal demand lasts and before the rapidly dwindling existing stock of these new loggers runs dry. Such is business. A brand-new MEA solar-powered CAT-M1 data logger for soil moisture monitoring, deployed in table grapes in the Victorian ‘Riverina’ region. Neatly installed at the foot of a wooden trellis post (in the foreground), the GBL-C logger is safe from farm machinery and foxes and hares that chew exposed cables.
  12. End Game in the IoT It’s been a long autumn and winter effort completing development of a CAT-M1-based on-farm data logger for release in the southern Spring, now only two days away. These product development sprints are a long litany of small crises that are surmounted and left behind in the rear-view mirror. When you’re stuck in the middle of it, it feels like crawling over broken glass. But we have early orders, some stock on the shelf with more next week, and rudimentary field trails conducted that have already thrown up a weird software bug in the ultra-low power state that this logger depends upon to keep energy consumption within budget. A software work-around nipped that in the bud before it got released into the wild. Our marketing department have been on the road, talking up the benefits of a completely new Internet-connected data logger that’s attractively priced and robustly packaged. This new product offering has been enthusiastically embraced by our key agents. Great technology is indistinguishable from magic. In this case, much of the excitement would appear to be generated by the simplest part of this technological wizardry: the coloured light behind our Green Brain symbol. This indicates CAT-M1 network connectivity, data transfers and GPS fixes to the guys with the muddy boots working under primitive conditions out in the field. This is MEA’s third IoT development in eighteen months, with two more key projects beginning as soon as this new technology is properly bedded down. I’ll lead the product development team through the coming Spring and Summer, until my 67th birthday next March. So, the real end-game in this IoT race is actually to invest the next generation of MEA engineers with the spirit of the sprint, a ‘can-do’ ethos and the sense that with hard work, anything is possible.
  13. Photo: MEA CAT-M1 Data Logger MEA’s CAT-M1 enabled data logger in functional prototype form (exploded view): Left: Clear lid. Centre: PCB, including Li-Ion battery, CAT-M1 modem (modem chip not installed), SIM-card and SD memory card holder (cards not installed) and PCB dual antenna. Most of the electronic circuity is on the rear of the PCB. The blue PCB is the plug-in sensor daughter-board. Right: Wedge-shaped solar panel custom-built to fit inside the tapered enclosure and attached to the black polycarbonate inlay. Cable assemblies are not shown.
  14. In our longest blog story yet, MEA describes the market forces that pushed us towards the latest CAT-M1 logging technology for use in the Australian bush, and how we engineered this new product to shoe-horn it into a small low-cost enclosure. All this, while adding a custom-made on-board solar panel, a dual PCB aerial for CAT-M1 and GPS, high efficiency battery charging, smarter connectivity, pluggable sensor interfaces and a Linux-based open-source operating system. Phew! This latest yarn can be found on this forum under ‘IoT Engineering/ What does it take to be an IoT engineer?/ CAT-M1 replaces 3G in MEA’s IoT’ at the bottom of Page 3.
  15. CAT-M1 replaces 3G in MEA’s IoT. The death knell of 3G logging systems – the backbone technology of many thousands of MEA monitoring systems ­in agriculture, climate and renewable energy systems across regional Australia for the past decade or more – will soon be ringing. As Telcos try to meet consumer demand for streaming videos and cloud-based services, getting customers off 3G allows carriers to free up wireless frequencies for 4G signals (the ‘long-term evolution’ or ‘LTE’ standard) over broader swaths of the radio spectrum. That trend towards more cellular bandwidth is of no interest to MEA; ours is a slow and patient drip-feeding of data from ground-based sensors up to our Green Brain platform, now holding somewhere north of 1.3 billion data records and rising at the rate of over a million records a day from more than 4000 sites. All is not lost, however – the telcos are climbing onto the IoT bandwagon by providing CAT-M1 and NB-IoT options on their LTE networks for those of us in the slow lane. MEA chooses CAT-M1, because we can buy that technology now (whereas NB-IoT is still not commercially available in Australia). We get a 2.6 times boost in transmission distance in comparison to 3G loggers, a lowering of our power consumption and – best of all – free IoT gateways all over the country! No more on-farm hubs to build and deploy – we jump from the sensor site direct to those tall cell towers. Down in the MEA basement the product development team has shrunk from four to three with the departure of our mechanical engineer for the warmth of Kununurra in Western Australia. Once again, product development timelines for a CAT-M1 product to replace MEA’s 3G loggers is predicated upon the opening of the irrigation season in July, leaving us 16 weeks to go from a clean sheet of paper to product on the shelf. Well, perhaps not quite a blank slate… To meet a four-month development cycle for an IoT product requires deep pockets full of all sorts of technological wizardry and understandings. These have been developed laboriously over the 35 years since MEA itself was a start-up, and I was its sole employee. The new logger operating system – for example – is Linux-based and uses the Legato operating system. We spent over a year figuring out this new platform to upgrade the 3G Hub of our Plexus on-farm sensor networks. Both hardware and firmware were debugged back then, and now we are simply recycling that IP in a new form. Software libraries were also developed back then to handle secure data uploads to Green Brain, data storage into lifetime memory on an 8 GB SD card and all the peripheral logic and switching to achieve ultra-low power mode. Likewise our solar-battery systems; the art of maximum power-point tracking (MPPT) and constant-current/constant voltage (CC/CV) charging of Lithium-Ion batteries from small solar panels was perfected in other projects over the past two years. Despite that head-start we’ve had some serious rabbits to pull out of the hat for this product to be commercially competitive. Fundamentally, the price of the enclosure and connectivity was predicted to have the greatest impact on the cost of the final product. To get around this, we needed to be able to use our smallest and least expensive enclosure. We already have the tooling for this robust and fully field-tested housing, through our GDot product. We’d learnt to ‘build it out’ during our Bluetooth IoT product developments in 2018. However, this small fully-sealed enclosure brought its own problems; somehow we had to shoe-horn a CAT-M1 aerial and a GPS antenna into there. We solved this with a tiny dual PCB antenna system that required special rf modelling to get the matching networks and micro-strip lines on the PCB to operate at our specified frequencies. Similarly, the solar panel had now got to be inside that small low-cost enclosure with everything else, behind the clear polycarbonate enclosure top. We solved that one by tooling up in China for a custom 960 mW 5.5 V solar panel with a cute rhomboidal shape designed for a perfect fit, but also over-sized to allow for efficiency losses imposed by the use of a vertical solar panel. There was also no room for an ‘on-off switch’; plugging and unplugging sensors has to start and stop logging and prevent power consumption during shelving and shipment. That design spec alone stalled us for some two weeks of serious head-scratching. Similarly, grumbling from our marketing department had us searching for a smaller circular connector to fit inside conduit used to protect cabling against rabbits, parrots and farm machinery. We found such a connector out of China, but at a prohibitive assembly cost if we soldered it on to multi-cored cables here in Australia. So we moved cable assembly to China as well, driving costs down by 66%. Sensor interfacing had to be plug-and-play. Fortunately, MEA developed pluggable daughter-boards during last year’s Bluetooth project. These were designed to work across multiple IoT platforms so they clip easily into this CAT-M1 logger. This gives us SDI-12 connectivity and the ability to make AC resistance measurements of thermistor temperature sensors and gypsum blocks. Two more sensor daughter-boards pop out of the works; one for all sorts of dumb analog and digital sensors and another for ultrasonic level measurement to be used in tank-level monitoring, cattle movements or whatever. These have been set aside for later in the year – we’ve no resources left to complete the firmware for these last two plug-ins. By week 12 we have working prototypes, CAT-M1 connectivity, GPS reception and functional logging in the MEA Test Yard. Its deepest winter here in Adelaide and the shadows of the tall buildings surrounding us in this industrial estate limit the hours of available solar charging to five hours each day before shadows encroach. Battery voltage climbs under load, and we all breathe a sigh of relief. On top of the CAT-M1 logger design is the parallel development of the test jigs and software needed on the production line. This stutters forward slowly as we divert resources to gain EMC compliance on our Bluetooth IoT products (so we can fill an order from the largest dairy in Europe – in Turkey) and sort out issues with the Green Brain mobile ‘Retrieve’ app to cater for the latest Android and iPhone models. But functional doesn’t mean finished. Serious software development effort is still required to cater for ‘edge cases’ under field conditions, to link us more closely with Green Brain to automate the installation process and to implement over-air upgrades. But we are in production, and there’s a very good chance we’ll go to market with a highly competitive CAT-M1 logger that will rejuvenate MEA systems in the market once our trusty work-horse 3G systems are put out to pasture in the next year or two. Photos will follow – just as soon as I find that spare minute I’ve been looking for all year…
  16. A Two-Legged Race in the IoT Twelve weeks after pivoting back from the Myriota transmission trials to start the development of MEA’s CAT-M1 logger, I sign the purchase orders and we’re in production of the first 100 units. That story must wait for another day. In the meantime, our three Myriota loggers rack up their first three months of continuous unattended operation. Rainfall has boosted the moisture in the top 60 cm of the soil profile and lifted the level in the rainwater tanks, while the first frosts have occurred and been captured. The panel of three photos shows my Green Brain view of these three satellite-delivered systems on my smart phone, which I open at odd moments to ease my nerves. Perhaps the real test has been that I no longer hang about anxiously twice-daily, waiting for the latest satellite pass. I get on with other things. In the race to develop these two new IoT technologies to meet the demands of the looming Australian irrigation season, it is the satellite-based Myriota field stations that are clocking up the data-hours. But it will be MEA’s terrestrial CAT-M1 loggers – with a mere two weeks of logged data to their credit – that will do the heavy lifting.
  17. Pivoting in the IoT Harsh competitive realities find MEA’s 4G-connected Plexus networks competing with older 3G technology in IoT on-farm applications. This, even though the 3G network has entered its Sunset Phase and disruptions to this network are expected to increase over the coming year as Telstra and other carriers reclaim these frequencies for 4G/LTE services. Telstra have announced 3G closures from 2020, now only a year away. Although MEA also has these older 3G offerings, it’s been clear that they will need to be replaced as soon as possible by newer LTE/CAT-M1 offerings, if only we could actually buy such technology to design into products. What these 3G offerings have shown us is that ‘sensor to tower’ terrestrial IoT technologies offer the real cost advantage of ‘no gateway’. Yet the hype swirling around our heads is promoting diametrically opposite solutions; a return to one-way sensor-to-gateway solutions of the sort that we built and deployed twenty years ago. Sure, Sigfox and LoRaWAN offer much improved radio technologies compared to our older 433 MHz simplex products. But the main bugbear from back-in-the-day remains: if you can’t make the jump from the sensor site to some local gateway, you can’t make the measurement, so you can’t make the sale. The problem then was always barriers - topography, tank farms, wood lots, distance, weak allowable radio power, wet canopies, ground-effects and the sheer clutter of other radio traffic on these free-to-air bands. MEA’s Plexus networks solved that problem in 2013 by forming co-operative sensor mesh networks to work around such barriers. Data back-haul from the single Plexus hub operated on both 3G and 4G bands. 3G/4G technologies could always beat that ‘barriers’ rap by shouting louder on quieter channels; the telcos – after all – own and control those portions of the radio-frequency spectrum. CAT-M1 will do even better, with a gain in transmission distance of over 2.5 times compared to 3G/4G modems. Myriota’s satellite solution offers the only other viable alternative; transmit upwards instead of sideways. The ‘gateway’ passes overhead everywhere a number of times a day. A year ago – with that background – MEA made the choice to work on CAT-M1 and Myriota IoT technologies for our newer offerings. MEA's CAT-M1 product development began late in 2018 while we waited for news on the availability of Myriota technology, which turned up in January of 2019. A quick pivot into Myriota territory saw MEA build three operational Myriota links in February; these have been transmitting soil moisture, micro-climate and tank level data solidly to MEA's Green Brain since then. That done, by March those harsh competitive realities saw us pivot quickly back to the CAT-M1 developments to help us face rising threats in our traditional irrigated agriculture markets. These new CAT-M1 Green Brain Loggers are to be released in the third-quarter of 2019 to replace our ageing 3G technology. In the meantime, we keep a close eye on our Myriota traffic, learning much of value about how we need to build that product after our CAT-M1 launch in a few months’ time.
  18. Tales from the Trenches Part 2 More learnings from the three Myriota sensor-to-satellite systems installed in the MEA Test Garden are told in the latest yarn at the bottom of Page 3 in “What Does it take to be an IoT Engineer?’ in this forum. ‘A Hat Trick in the IoT’ covers issues with over-zealous data feed rate, the vagaries of UARTs and glimpses into topographical issues caused by the Mt Lofty Ranges. Oh, and one more thing: check the sealing around the Myriota SDK antenna connection where it enters the enclosure. Sometimes the O-ring seal is under insufficient pressure to prevent water ingress if the unit is placed outside under rainy conditions. Simply tighten the connector nut inside the enclosure to fix this.
  19. A Hat-Trick in the IoT. Myriota have repaired our third recalcitrant sensor-to-satellite modem and sent around one of their engineers to hand-deliver it. Good companies are tested in bad times. This has been a superb experience in dealing with Myriota’s support team, and we have gained confidence in them rather than losing it. Facing down technical issues is a learning ground; one is forced to set aside specious hand-waving and theorizing and to let the problem do the talking. Thus, what started with some manufacturing faults by one of Myriota’s suppliers turned into deeper insights at MEA into how this new technology really works. First up, we’d been trying to account for missing records. We’d assumed these were simply poor radio transmissions that hadn’t gotten through to the satellite, or could not be decoded. More careful analysis showed that they weren’t actually missing, but up to thirty-hours late. How could that be? Turns out we’d over-stretched the pipe, by trying to send four records twice a day for each pass of the single satellite. Dropping back to three records per pass (six log records per day at four-hour intervals) found us moving data reliably every 12 hours. Next up were issues in our own data – inexplicable glitches that suggested we were dropping digits, causing ugly spikes and unseemly drop-outs in our lines graphs. This turned out to be coding issues in using the Myriota low-power UART to obtain serial data from our sensors. Implementing a system of error-checking and re-tries fixed this. Finally, a first glimpse into the mysteries of topographical interference in transmissions to passing satellites… The single Myriota satellite that we see here in Adelaide is in polar orbit, with the Earth rotating under it twice per day. Even though the satellite completes an orbit every ninety minutes, we are only likely to see it in our field of view for between five and eight minutes during each fly-by. And for MEA - sitting up close to the foothills of the Mt. Lofty Ranges - this means that satellite transits to our east may be blocked on some passes. This is mere supposition at this stage, but it's something we picked up listening to Myriota’s engineers discussing our connectivity issues. Older (MEA is now officially 35 years old), wiser and somewhat less brash, we re-install our repaired tank level Myriota transmitter, complete with data re-tries, slower four-hour log interval and a wary eye to the East. The first rains of autumn appear from the west shortly thereafter, and now all three MEA-Myriota test stations work together to show the measured rainfall, the consequent lift in soil moisture in the surface layers, falling temperature and rising relative humidity associated with this cold front and – best of all – a rise in the level of the rainwater tanks as they begin to recharge.
  20. Lessons from Myriota I’ve written three blog posts on how MEA started up our adventures in moving IoT data over Myriota’s nano-satellite link to our Green Brain web application. Those stories can be found at the end of my forum posts entitled “What does it take to be an IoT Engineer?” on this Engineers Australia IoT site. I’d like to hear how other engineers are travelling with their new Myriota SDKs (Software Development Kits), so jump onto this thread and contribute. Let me kick-off by mentioning that I’ve got three sites set up moving soil moisture, climate and tank level data – about 350 messages so far in the past five weeks. Proof-positive of five weeks of soil moisture tension data linked from MEA's test garden to Myriota's tiny satellites The folks at Myriota have been wonderfully helpful as we walked into all the usual bear-traps, sharing “Ah, blast!” moments on either side of the fence. Here’s two: MEA has been jamming too many messages into the pipe, at the rate of 8 per day. This constipation results in lost messages. We’re dropping back to six messages per day. Our tank level monitoring site has not transmitted a single message (see that last story “The IoT via Satellite”). Turns out this was due to multiple instances of dry solder joints on that particular SDK, particularly under battery pads and around the UHF aerial-connections corner of the module (pads 19 to 29). Lesson: Keep an eye out for hardware faults if you are not getting connectivity. The Myriota folk are taking this up with the SDK-kit manufacturer. OK, open to Myriota IoT stories from engineers working with this gear...
  21. The IoT via Satellite In less than a week after receiving two more Myriota satellite modems (in SDK form) we’ve built two more operational IoT devices, tripling the rate of data flow and improving our chances of understanding this brand-new technology as we race to create working products for new markets. Modem #2 is a mini-weather station, capable of accurate measurements of rainfall, air temperature, relative humidity and their derivatives – dew point and frost point temperatures and vapour pressure deficit. Data compression allows meaningful data to be pushed up via the very slow and narrow pipe that the Myriota technology offers, essentially 14 bytes per three hours. Modem #3 is doing something a bit more exciting; we’re measuring tank water level using a downward-looking ultrasonic depth sensor mounted in the tank roof, which is the perfect mounting spot for upward-looking satellite transmissions. This new technology makes use of our latest ADC technology and one of a series of generic sensor interfaces we’re developing to plug-and-play with sensor front-ends and different comms back-ends in MEA’s small IoT world. Calibration checks use far less sophisticated technology; a tape measure with me up a ladder hoping my glasses won’t take the deep dive to the tank bottom. This is the end of playtime; now we have to get serious about building real IoT products. For that we need a ‘reference design’ from Myriota to help us with specifics like antenna types, special radio-frequency PCB layout issues and so forth. It is commonplace now for manufacturers to use their own engineers to help the applications teams in the customers’ world to get to market as quickly and painlessly as possible. All this focus on the hardware and data transmission technologies hides one of the fundamental truths of the IoT; without a web-based data platform to connect data to customers all this ground-based techno-effort is for nought. MEA’s Green Brain is the final repository for the data that’s left my test garden, gone straight up 800 kilometres to the low-earth orbit satellite whizzing by at many kilometres per second twice per day, to be carried in polar orbit for another forty minutes until the satellite passes over Norway where it downlinks the raw radio data for web-based processing using Myriota’s special algorithms. Finally, the sorted and scaled data is hopped across to Green Brain from whence I can track things – just metres away from where it all began. We corral all these three demo sites into a shared link to Green Brain for Myriota so that their engineers can see what we see from their own offices, and share that link in turn with others so that they too may play and be convinced. In the meantime, we’re running a careful watching brief on the demonstration systems. Almost immediately we’re reminded of how easily one can be lulled into complacency; only one of the two new units is getting data through. We recheck battery sensor and aerial connections, press the RESET button on the Myriota modem board, check that we haven’t made a typo in the Device ID, try to get radio transmissions to the PC dongle in the dev kit, reflash the code, and wait through various twice-daily transmission windows. Everything checks out. No fault found at our end. Finally – by standing outside in the MEA car park – we realize that we’re not getting a GPS fix from this module. Without that, nothing starts up. We are brutally reminded that our understanding of this new data transmission technology has all the depth of a car park puddle.
  22. The IoT on a Shoestring. With MEA’s 35th birthday now only weeks away, we find ourselves cobbling together demonstration systems at a furious rate, calling up snippets of technology from our IP grab-bag to make ‘Minimal Viable Products’, or MVPs. The problem facing MEA is that we don’t know these new markets that the Myriota ultra-low-cost satellite IoT technology has opened up before us – field measurements beyond the bounds of the cellular IoT network. Rather than spend our limited marketing resources wandering expensively into the Outback to survey the views of potential agricultural customers, we are assembling working prototypes around the modems we have been able to obtain from Myriota and hooking various applications across to our Green Brain web platform. We’re after rapid feedback from a select few. These MVPs are only 80% operational, but our target audience – Early Adopters – prefer this so that they can be the first to play with this new technology. They are even prepared to pay a premium for that privilege, and to put up with MEA’s engineers hovering anxiously nearby as we tweak out obvious bugs. By the time Myriota can get into solid volume production of their technology, MEA will also be ready with the application technologies to take full advantage of it. This is only possible because of the ‘home-town advantage’; both Myriota and MEA are Adelaide-based companies, and I’ve been with them along their journey since they first came out of the University system some years ago with a wad of patents, seed funding and the drive to disrupt the current IoT paradigm, where satellite links have traditionally been prohibitively expensive. Marketing folk call this ‘networking’; I just call it ‘staying alert’. And so – in a matter of weeks – we’ve hacked together satellite-connected demonstration models for five typical applications in both irrigated and broad-acre agricultural applications: 1. ultrasonic and hydrostatic tank water level monitoring 2. sub-soil moisture tension measurements in grain-growing areas 3. standard soil moisture content profiling in irrigated agriculture 4. frost and micro-climate recording systems 5. rainfall monitoring MEA’s two IoT teams proceed at a swift pace into this new arena, working on a shoestring budget but drawing upon a deep well of experience and proven technologies that took us years to develop. So here they are – young and old: - The younger team are our Green Brain developers: Matthias (Green Brain back-end), Brent (Green Brain design front-end) and Dominic (business development and customer insights) The product development team builds all the ground-based stuff not in the Cloud: Andrew (electronic engineering), Terry (production engineering and phone apps), Jack (mechanical engineering and document control) and Raf (firmware engineering)
  23. ‘Straight-Up’ with the IoT. A new year has fired up at MEA and the troops have rested. I wish I could have rested, but with MEA closed over the Christmas-New Year period, I inevitably spent that down-time reading up on all those technologies I’d sidelined in my head amidst the daily blur. This year I got up to speed on the latest ultrasonic technologies for level sensing, just in case we set about marrying Myriota’s remote-area satellite IoT technology with tank level monitoring and sensing cattle movements. And it’s been hot ‘Down Under’, so the Christmas break also found me conducting a large-scale study in the MEA Test Garden on the impact of current-collapse in our IoT solar panels at temperatures above 40°C. To complicate things, the summer sun climbs high into the sky during the summer solstice, with shorter charging days for north-facing solar panels exacerbating solar-battery charging difficulties caused by radiant heat load. January and the return to work saw us hosing down last-minute alarms in our new Bluetooth-gypsum block IoT technology, destined for shipment to some remote tropical island off Australia’s north-east coast where the Green Turtles are breeding. Things that shouldn’t have gone wrong did go wrong. The ADCs and direct memory access functions in our MSP430 embedded controllers didn’t work as either the data sheet or Errata notes suggested they ought. Occasional data gaps were appearing in the logged record because measurements didn’t complete. We fixed this with a re-write using basic code and much cursing. Then the outdoor air temperature hit 46.9°C (116.4F) in the MEA Test Garden in Adelaide – with over 52°C inside the enclosures – and suddenly we had data spikes. That last-minute glitch was also hosed down and we entered production. With all that done, we got back to debugging our Myriota satellite IoT test system, which had stopped transmitting following a rain event. That too got sorted, teaching us yet again that one can’t be careless with enclosures even when hacking stuff together to test first principles. February rolled around and we dug deep into our back pockets for intellectual property – hardware and firmware – to lash the Myriota modem to our newly-developed smart gypsum block interface. Sometimes it’s good not to be a start-up, as there is no need to develop every darn thing from scratch before you can get down to business. Test beds evolve quickly. Six weeks into the New Year in the MEA Test Garden and we’re monitoring the soil moisture tension profile in the orchard and beaming data straight-up to those whizzing satellites that pass overhead twice-a-day. We’ve already learnt that we can send eight 20-byte packets by queuing them and letting the Myriota modem tackle the business of attempting multiple packet transmissions at each pass. This will stop-gap us against otherwise poor sampling rates until more satellites are launched. Once again we find ourselves under time pressure. Does this satellite IoT technology really work as expected? Can we reliably design product around it? Where are the pitfalls? Can we compress the data sufficiently to match the very small payload? How do we connect data across from the Myriota server to MEA's Green Brain in the cloud? Only squads of data – and time to collect it – can help us here, and we need all sorts of weather events to shake loose nasty gremlins that will surely bight us if they get through to production and our customer base. In the meantime, my Christmas sabbatical has fired my imagination and I’m spending the wee morning hours in my home lab testing new sensor ideas. There just never seems to be enough time in this IoT race!
  24. Hi Geoff Yep - exciting times, indeed. We've had ours since last year and have been transmitting data from the MEA roof via satellite about four times per day since then. I can hardly believe it's possible! And so many interesting applications opening up for IoT (truly-) anywhere... The Myriota applications team have been quick and helpful sorting out bugs and queries. All the best with your developments. Cheers for now Andrew
  25. Another Christmas in IoT Land Weariness pervades all as another working year winds down here at MEA. With the pressure to release new IoT products and the need to shore up the ravages of Mother Nature among the old ones, it’s been a tough year in Product Development. We learnt this year to prototype and manufacture in China, speeding up PCB manufacture while driving costs down in a competitive arena. In 2018 we designed and built a Bluetooth logger for SDI-12 digital sensors, then immediately set to work on a specialized version capable for making AC resistance measurements in gypsum blocks (a simple 78-year-old soil moisture tension sensor technology that we’ve sold by the tens of thousands into Australian agriculture). We finished that gypsum block logger only this morning (just hours short of our close-of-business); we already have orders for a whole raft of them in the New Year. Test jigs and the business of building production lines was a mixed bag in 2018; we’re going to have to start that process earlier in the product development cycle and allocate a greater portion of resources to it. Next year we’ll do better. Our Green Brain web app for farmers passed a billion records in 2018 and boasted over 99.8% up-time and a 90% customer retention rate. Google Analytics tells us that most irrigators with our gear are hitting up their Green Brain account on a daily basis. Adelaide-based Myriota have progressed their satellite IoT technology to the point where they released their software development kit (SDK) in November 2018. Within weeks of purchasing ours we’d established a link via satellite from the rooftop at MEA via the Internet back to MEA. Next year we’ll turn that ground-breaking IoT technology into practical agricultural applications. In my own small corner of MEA’s engineering world I’ve completed my three-decade effort to build a crop-water stress (CWS) sensor for use in horticultural crops. These self-contained solar-powered instruments make use of all that new 2018-technology for data logging with Bluetooth data pickup. In the wee morning hours, I’ve also solved the sensor insertion issues needed to bury the tiny probe into the xylem tissue of vines and trees. The prototype CWS sensor has run flawlessly for two solid months in a grapevine in the MEA Test Garden. The production run of the first 25 operational units ran smoothly. As part of that CWS instrument design we created a sensor shelter to house the vapour-pressure deficit sensor. This specially-shaped nylon housing costs under $60, was designed by MEA’s mechanical engineer and manufactured in Brisbane in less than a week on a HP Jet Fusion 3D-printer. Commercially, 2018 was a hard year for us at MEA as we transitioned to second and even third generation products – this effort absorbed huge portions of our profitability but kept us in the marketplace. There’s no standing still in IoT land. However, help was at hand from the Australian Federal Government – in November 2018 MEA learnt that we had been awarded an ‘Accelerating Commercialisation Grant’ to build a new generation of on-farm IoT products. That will be the job in 2019-2020. With the official MEA year at an end, I’ll be heading home shortly to drink a beer on the back porch and look out over the MEA Test Garden where I spend my engineering down-time growing things. This garden is the secret sauce behind my working life in irrigated agriculture, just as my remote hillsides at Pine Hut Knob were the secret sauce behind the field measurements in wind and solar energy that fueled MEA’s early decades. It’s inside this huge garden that I will begin the shakedown testing of the crop water stress sensors over the Christmas-New Year period and into 2019. Just like the CWS sensor, my tree crops and vines represent decades of patient cultivation - avocados, figs, almonds, oranges, lemons, grapefruit and mandarins, bananas, mulberries, plums, peaches, nectarines, wine and table grapes, apples and pears, kiwi fruits and passionfruit – all the trial crops surround me, and my home lab stands in the middle. There’s plenty of scope in my own back yard for a quick and broad test of the CWS technology across a whole gamut of horticultural crops. So where will I be over the holiday season? Hovering anxiously over all these new sensors in my garden, nurturing and mothering and brooding away, willing all this magic to sing in unison for the greater good. And keeping a close eye on my irrigation through the heat of an Australian summer…
×
×
  • Create New...