Wifi /Zigbee based temperature sensor to be connected to web-based application

[Vizier87]

Hi guys, I'm currently working on a temperature/humidity sensor and just learning the ropes on the IoT architecture and I'd appreciate some pointers here.

The **broken link removed** has the following specs:

• ZigBee 1.2, Wifi 2.4GHz compliant.
• Zigbee coordinator of the home area network.
• Supports up to 50 devices
• Secured Wifi: WEP, WPA/WPA2

I do have some Raspi experience but here I'm mentally stuck on a few concepts on the IoT. I'll be getting some help from a friend to work on the web-based application but I need some advice to get things together first.

It seems the device's readings can only be read with a smartphone app. I wonder if we can also access the raw readings if we direct the data to a script via VB.NET? The user guide is attached in this thread.

The reason for my inquiry is simple: We wanted to monitor a few of similar devices and use a custom-made web application (with its own rules and monitoring algorithm) which shows the readings in real time rather than depending on the smartphone app.

Any pointers are appreciated.

Thanks!
Vizier87

 

[rjenkinsgb]

You need a central server to coordinate and collect data from the individual devices, and make it available.

The simplest approach would be use something like Homeseer running on a Raspberry Pi as the central system, with a USB Zigbee device.

HomeSeer HS4-Pi Smart Home Software

HS4 is an advanced smart home automation software that's designed to work a wide range of smart home products and technologies. Use to remotely and automatically control smart home devices such as smart switches, thermostats, door locks, sensors & more. This edition is designed to run on...

shop.homeseer.com

Add one of the custom web page plugins, or write you own - all the device data can be read by accessing the appropriate homeseer URL
There are hundreds of other interface plugins available for that system.

There are other home automation type systems that could be used, but I've no personal experience with those.

 

[Vizier87]

Thanks for the suggestions.

The simplest approach would be use something like Homeseer running on a Raspberry Pi as the central system, with a USB Zigbee device.

HomeSeer HS4-Pi Smart Home Software

HS4 is an advanced smart home automation software that's designed to work a wide range of smart home products and technologies. Use to remotely and automatically control smart home devices such as smart switches, thermostats, door locks, sensors & more. This edition is designed to run on...

shop.homeseer.com

The thing is, it seems like I have to stick to the device I was dealt with. And it seems like now I have to deal with the manufacturers maybe and see if that kind of support is given.

Does this mean that I could technically siphon the data from the device I shared earlier and then take it from there with the Homeseer?

Oh well I saw Homeseer's list of supported devices and the Gateway is not listed. Aww shucks.

You need a central server to coordinate and collect data from the individual devices, and make it available.

Add one of the custom web page plugins, or write you own - all the device data can be read by accessing the appropriate homeseer URL
There are hundreds of other interface plugins available for that system.

I do have some experience with Microsoft Azure which enables the cloud computing for IoT for the central server part but before I even go there, I need to access the devices' data. Seems like it's going to be quite a task.

Thanks. I'll see if there is a workaround there.

 

[rjenkinsgb]

No, for Homeseer you would need a USB Zigbee interface.

That Maevi device seems very restrictive; you could access it indirectly via another web server on a separate device to collect and store data; nothing particularly fancy or powerful, any basic system that can run a LAMP stack for web interface should be OK to make requests, store data and access it - so again, a Pi or similar would work.

The hard part would be monitoring all the interactions between the Maevi unit and its own app, to reverse engineer the communications so you could read data from you own device.

It is a terrible choice for an interface, unfortunately, as nothing about its internet protocol is documented.
Even the makers web site seems dodgy, Chrome is trying to block access as it's unsecured.

 

[Vizier87]

That Maevi device seems very restrictive; you could access it indirectly via another web server on a separate device to collect and store data; nothing particularly fancy or powerful, any basic system that can run a LAMP stack for web interface should be OK to make requests, store data and access it - so again, a Pi or similar would work.

The hard part would be monitoring all the interactions between the Maevi unit and its own app, to reverse engineer the communications so you could read data from you own device.

Sorry now I'm in unfamiliar waters.

I think I'm quite good at parsing streams of data, that I can do with VB.NET and maybe on the Pi's Python3.

How do I grab this "raw" stream of data though? If that's how it works.. because the data is (I think) decrypted and displayed on a smartphone app, and at this point I'm unsure if there are "intermediary" databases out there which I can access, if allowed. I'm under the impression that these streams of data are encrypted and maybe restricted?

Thanks a lot Mr Jenkins.

 

[rjenkinsgb]

You would need something like Wireshark running on the same LAN as the device and phone, to capture ethernet packets to and from each, then try and make sense of them.

Wireshark · Go Deep

Wireshark: The world's most popular network protocol analyzer

www.wireshark.org

I'd guess the device tries to connect to the manufacturers servers and the phone app also connects to that, to read back data from "your" device.

If you could make sense of the data, then accessing the device or its storage, storing data yourself and making it available via a web page would be easy enough using PHP and MySQL
[The P and M in LAMP - Linux, Apache (web server), MySQL (database) and PHP programming language.]

 

[Vizier87]

Ok thanks a lot for the feedback. I'll get back to this thread if I've made progress

 

[Nigel Goodwin]

Look up RandomNerd for lot's of different methods to do what you're wanting, using Arduinos, ESP's etc. a great resource.

I've LONG since always used PHP and MYSQL on web servers for such purposes, but always remote ones - as I've got them, why not use them? (and I've never had much luck with PI's).

 

[Vizier87]

You would need something like Wireshark running on the same LAN as the device and phone, to capture ethernet packets to and from each, then try and make sense of them.

Hi.. I've been going through some rabbit holes but I need to confirm something first before I get deeper.

Does this mean that the Wifi router I'm using to connect to the device should also be the one I'm connected to the PC running Wireshark?

I was getting into securing a few things for now like setting up a proper VM for malware testing since I'm planning to run scripts if necessary for streamlining the whole process via VB.

Thanks.

 

[rjenkinsgb]

Wireshark may need to run on the router to be able to monitor "third party" traffic, rather than in & out of the PC or device it's running on.

If your router can run DD-WRT or similar, you should be able to do it directly on that. More info here:

DD-WRT :: View topic - Capturing traffic with WireShark and dd-wrt on Apple TV

Otherwise, a linux machine with two ethernet ports could be used, with things set up so all data to be monitored passes through it.
A "man in the middle" setup, in other words.

 

[Vizier87]

Wireshark may need to run on the router to be able to monitor "third party" traffic, rather than in & out of the PC or device it's running on.

If your router can run DD-WRT or similar, you should be able to do it directly on that. More info here:

DD-WRT :: View topic - Capturing traffic with WireShark and dd-wrt on Apple TV

Otherwise, a linux machine with two ethernet ports could be used, with things set up so all data to be monitored passes through it.
A "man in the middle" setup, in other words.

Alright, on it now, thanks!

Edit: So is it achievable with a completely wireless router? I'm planning to buy this one: TP-Link 4G LTE M7350 Portable Wireless WiFi Direct SIM Router – ALL IT Hypermarket

Or is only achievable (or better achieved) with this type? (I got it from the forum) I have to say.. we don't have LAN ports in our home *sigh*

 

[rjenkinsgb]

I doubt any generic router will have facilities for monitoring raw data.

Two cheap APs and a Linux machine? Have one AP in client mode, connected to your WiFi and set the other AP set up with a different WiFi SSID and connect the device you are testing to that.

That's the only way I can think of to guarantee the data flow is accessible for monitoring..

 

[Vizier87]

So hypothetically does this look like the topology which I should be using?

Thanks a million bro. You da man.

 

[rjenkinsgb]

So hypothetically does this look like the topology which I should be using?

no - that does not put the Pi in the signal path to the IoT device. The Pi will never see the data between the IoT device and the internet.

The Client mode AP would be used to connect a computer to the internet. With a Pi, that could be built-in Wifi or a USB WiFi device etc., whatever connect the Pi to the internet.

Then the second AP connected to the PC / Pi via Ethernet, with that providing the internet connection for the IoT device.

Set the Ethernet port up with a different LAN IP subnet and enable DHCP so things can connect to it and be allocated a suitable IP address for that subnet.

Then enable IP forwarding and routing so things on the second AP get internet access.

That allows programs on the Pi to directly tap in to all the data going through the connection from the second AP, so you can view or log it to work out what is happening.

Just hope it's not using HTTPS protocol, as that makes things more complex again. Same physical setup, but you would also have to emulate the IoT server on the Pi, to see decrypted data.

 

[Vizier87]

no - that does not put the Pi in the signal path to the IoT device. The Pi will never see the data between the IoT device and the internet.

The Client mode AP would be used to connect a computer to the internet. With a Pi, that could be built-in Wifi or a USB WiFi device etc., whatever connect the Pi to the internet.

Then the second AP connected to the PC / Pi via Ethernet, with that providing the internet connection for the IoT device.

Set the Ethernet port up with a different LAN IP subnet and enable DHCP so things can connect to it and be allocated a suitable IP address for that subnet.

Then enable IP forwarding and routing so things on the second AP get internet access.

That allows programs on the Pi to directly tap in to all the data going through the connection from the second AP, so you can view or log it to work out what is happening.

Just hope it's not using HTTPS protocol, as that makes things more complex again. Same physical setup, but you would also have to emulate the IoT server on the Pi, to see decrypted data.

I think I'm getting there. Here's my understanding now:

This makes a whole lot of sense I have to say. This means I'll need two Pis (it is the one which I have experience with.. and setting the DHCP etc etc shouldn't be too difficult I think).

Am I getting it?

Thanks... you have been such a huge help.

 

[rjenkinsgb]

That's the signal flow I was thing of.

You only really need one Pi / PC though, with an ethernet connected AP where you have "Raspi 2".
It depends which is easier and cheaper to set up.

 

[Vizier87]

Ah I get it now. An AP instead of a "Raspi 2".

So basically what I'll need is basically a Pi (I guess Pi 4) and a wireless AP with an ethernet port. (Is the product linked sufficient?)

Just one last question before I go shopping, why a Linux machine though?

Pi's Raspbian which is pretty much a mini Debian version, from what I understood.

 

[KeepItSimpleStupid]

hey, just questioning.. That's all.

Can you set the monitoring machine as the "gateway", inspect the packet, and port-forward to the real gateway?
You would be guaranteed to see each packet.
Do, remember that packets can arrive in any order. I don;t know where that assembly takes place.

 

[rjenkinsgb]

Ah I get it now. An AP instead of a "Raspi 2".

So basically what I'll need is basically a Pi (I guess Pi 4) and a wireless AP with an ethernet port. (Is the product linked sufficient?)

Yep, exactly.

I'm not sure of the details of that device - it says "powerline" so it may only function if fed via another powerline adapter; it only mentions client mode / ethernet output..

Something like this looks reasonable - it says explicitly it works as an AP if connected via ethernet:

https://shopee.com.my/TENDA-A15-Dual-Band-5Ghz-2.4GHz-AC750-Wireless-WiFi-Extender-Wi-Fi-Repeater-with-AP-Mode-Access-Point--i.148072485.4872393109?position=162

Linux - because far more networking tools are available and it provides better access to data streams between network connections; windows is pretty much crippled for things like that.

Can you set the monitoring machine as the "gateway", inspect the packet, and port-forward to the real gateway?

That's my exact idea, and why data must go through that machine, in & out two separate network connections.

 

[KeepItSimpleStupid]

See:https://www.opensourceforu.com/2011/02/capturing-packets-c-program-libpcap/