Help with RX UART Serial Data Conflicting With TX on an ESP32

[_John_]

Hello,

I've been commissioned to build a prototype of a measurement device used in construction. I'm using an ESP32 with a couple of LIDAR distance sensors and a Nextion touch display. I'm using an accelerometer as a digital level that displays all of the time on the display.

I'm having trouble receiving serial data from the Nextion display (button presses followed by values) because the accelerometer data is constantly being sent to the display. Unfortunately, I can not post the code because it's considered proprietary at this point. But the problem can be expressed like this:

void loop()
{
if(Serial.available()>0)
{
char received = Serial.read();
if(received == 'k'){
val = Serial.parseInt();
}
}
Serial.print("Something");
}

That code will not receive the data because it's constantly printing "Something". If I take the Serial.print("Something"); out, it receives fine.

My question is: Is there a way to prioritize RX over TX? I tried using ESP32 Software Serial to receive the data separately but couldn't get it to work at all. I'm hoping I can just pause the TX briefly while it's receiving the data, I just don't know how to do it.

If there isn't a software solution, can I add a fourth UART the the ESP32 and just receive through it? If so, what do I need to do it?

I would really appreciate any help with this because I'm almost done but totally stuck now. I'm very ignorant when it comes to serial communication so please dumb down any suggestions.

Thanks in advance!

 

[Beau Schwabe]

Check if the Serial.print is working in the background. (<-Arduino default operation) In my past experiences with the Arduino any serial activity is pushed into background mode. Which in some cases is great, it allows for you to do other tasks, but in other cases it can be a problem. I forget right off hand, but there is a way to programmatically force it to wait until the transmission is complete.

 

[_John_]

Beau Schwabe Thats interesting. I'm looking into it now. It would be great if I could put the TX in background mode but still have RX at the forefront. I don't need every transmission to happen but I do need all of the received data. Thanks for the info!

 

[Tony Stewart]

I'm not sure either, but if limited by no RTS/CTS interface signals. These must be declared.

With Interrupts enabled and using software flow control, when the ESP32 receives an XOFF character, it should stop transmitting data, and when it receives an XON character, it can resume transmission.

Ring buffers are needed for the LIDAR ports.

 

[Nigel Goodwin]

Assuming you are using hardware serial ports (and I seem to recall the ESP32 has three of them?), then both transmit and receive should be interrupt driven using circular buffers. Possibly you can increase the TX buffer size?

 

[_John_]

Tony Stewart Thank you for the help! Honestly, I don't know what most of what you said means. But, I'll start Googling and try to learn something.

Nigel Goodwin Hey Nigel! I am using hardware serial ports and there are three. I don't know if they are interrupt driven or circular buffers. I haven't figured out how to increase the TX buffer yet but was able to increase the RX with: Serial.setRxBufferSize(1024); and there is a significant improvement. It's working for the most part but freaks out every once in a while. Thank you so much!

I also threw in a Serial.flush(); at the end of the loop which seems to help a little. Hard to say though. I was hoping I could increase the TX buffer with Serial.setTxBufferSize(1024); but I got an error in the Arduino IDE.

Anyway, I think I'm on the right path here though. So, thank you guys for the help!

 

[Nigel Goodwin]

Tony Stewart Thank you for the help! Honestly, I don't know what most of what you said means. But, I'll start Googling and try to learn something.

Nigel Goodwin Hey Nigel! I am using hardware serial ports and there are three. I don't know if they are interrupt driven or circular buffers. I haven't figured out how to increase the TX buffer yet but was able to increase the RX with: Serial.setRxBufferSize(1024); and there is a significant improvement. It's working for the most part but freaks out every once in a while. Thank you so much!

I also threw in a Serial.flush(); at the end of the loop which seems to help a little. Hard to say though. I was hoping I could increase the TX buffer with Serial.setTxBufferSize(1024); but I got an error in the Arduino IDE.

Anyway, I think I'm on the right path here though. So, thank you guys for the help!

Generally circular serial buffers use interrupts, so your program just sends data to the 'serial port' and it's stored in the circular buffer, then emptied out by the interrupt routine.

On a PIC you've got full and obvious control, because it's not hidden away in a library - so if the ESP library doesn't allow you to increase the TX buffer, you might need to edit the library itself.

Using MCC generated serial routines:

#define EUSART1_TX_BUFFER_SIZE 128
#define EUSART1_RX_BUFFER_SIZE 128
#define EUSART2_TX_BUFFER_SIZE 64
#define EUSART2_RX_BUFFER_SIZE 64

 

[_John_]

Generally circular serial buffers use interrupts, so your program just sends data to the 'serial port' and it's stored in the circular buffer, then emptied out by the interrupt routine.

On a PIC you've got full and obvious control, because it's not hidden away in a library - so if the ESP library doesn't allow you to increase the TX buffer, you might need to edit the library itself.

Using MCC generated serial routines:

#define EUSART1_TX_BUFFER_SIZE 128
#define EUSART1_RX_BUFFER_SIZE 128
#define EUSART2_TX_BUFFER_SIZE 64
#define EUSART2_RX_BUFFER_SIZE 64

Ah yes, I figured I'd have to go in and change the HardwareSerial library with something like that. But there has been a development, a good one! While I was figuring out the buffer size stuff I ran across a different SoftwareSerial library. It seemed a bit simpler so I gave it a shot. After converting a few ints to strings everything worked great. I'm sending the TX through the SS and the RX is coming into the regular Serial port.

So, the problem is 100% fixed now and I can start building the housing.

This is the second problem you helped me out of Nigel and I really appreciate it. Have a great day!

 

[danadak]

For future some considerations to tuck away in your mellon (in my case its all mellon) :

1. RTOS is a way of allocating MIPS for processors for various processes and tasks,
FREERTOS is one out there.

2. A useful method is to create a virtual display interface in RAM, and test if there
is display activity weather or not if its up to date. If not up to date then write buffer
and actual display. Thast useful for "dumb" dispolays, ones w/o their own controller
and buffer. If uptodate NOP is performed. I have used that to eliminate display
"artifacts" that cause flickers and weird transitory display behaviour and characters.

3) There are processors out there that can interface a display and using DMA and LUT
state engines do everything in background. Even including separate busses so main
core hums along doing its thing. So only write activity interrupting main processor is
the update of the display buffer. Of course there are displays with their own buffers
and controllers.....


Regards, Dana.

 

[_John_]

Hey danadak ! I read your post last night and entered the rabbit hole of FREERTOS. It's a bit advanced for my mellon (which is mostly mush at this point) but I'm trying to understand. If I'm correct, it's a more direct way to program, which allows you more control. I could be totally off-base, I was reading about it right before bed. But definitely something I want to learn more about. I'll be able to dive in more once I get this project done.

I've only recently gotten into touch displays. Being my first experience with them, I went with a Nextion bc the software seems to do a lot of the heavy lifting. At $60 for the smallest "intelligent" version, I definitely want to learn to do it myself. The "virtual display interface" sounds interesting. How does that work? Does it recreate the display in RAM every cycle and compare the actual display to it? That can't be right, seems like it would be too taxing. That's something else I'd like to learn about.

Thanks for all the info and future Google searches!

Oh, and my problem wasn't 100% fixed like I had thought. Turns out that when I change pages on the display it overflows the buffer again. At least that's my limited take on it. It takes almost 20 seconds for it to catch back up. One workaround I just found this morning is if I clear the buffer with:
while (Serial.available() > 0) {
Serial.read();}
at the end of the loop, it won't hang. But I think I lose data occasionally doing so.

I was trying to increase the TX buffer in the library like Nigel suggested but can't even find the doc to change it. urgh...

Anyway, thanks again for the info! I'm going to get back to the nightmare I've created.

 

[Nigel Goodwin]

Hey danadak ! I read your post last night and entered the rabbit hole of FREERTOS. It's a bit advanced for my mellon (which is mostly mush at this point) but I'm trying to understand. If I'm correct, it's a more direct way to program, which allows you more control. I could be totally off-base, I was reading about it right before bed. But definitely something I want to learn more about. I'll be able to dive in more once I get this project done.

I've only recently gotten into touch displays. Being my first experience with them, I went with a Nextion bc the software seems to do a lot of the heavy lifting. At $60 for the smallest "intelligent" version, I definitely want to learn to do it myself. The "virtual display interface" sounds interesting. How does that work? Does it recreate the display in RAM every cycle and compare the actual display to it? That can't be right, seems like it would be too taxing. That's something else I'd like to learn about.

Thanks for all the info and future Google searches!

Oh, and my problem wasn't 100% fixed like I had thought. Turns out that when I change pages on the display it overflows the buffer again. At least that's my limited take on it. It takes almost 20 seconds for it to catch back up. One workaround I just found this morning is if I clear the buffer with:
while (Serial.available() > 0) {
Serial.read();}
at the end of the loop, it won't hang. But I think I lose data occasionally doing so.

I was trying to increase the TX buffer in the library like Nigel suggested but can't even find the doc to change it. urgh...

Anyway, thanks again for the info! I'm going to get back to the nightmare I've created.

Have a look at:

arduino-esp32/cores/esp32/HardwareSerial.cpp at master · espressif/arduino-esp32

Arduino core for the ESP32. Contribute to espressif/arduino-esp32 development by creating an account on GitHub.

github.com

Line 545 for TX and line 529 for RX

So it looks like you can easily increase the TX buffer, but you must do it before you start the serial port.

 

[danadak]

I've only recently gotten into touch displays. Being my first experience with them, I went with a Nextion bc the software seems to do a lot of the heavy lifting. At $60 for the smallest "intelligent" version, I definitely want to learn to do it myself. The "virtual display interface" sounds interesting. How does that work? Does it recreate the display in RAM every cycle and compare the actual display to it? That can't be right, seems like it would be too taxing. That's something else I'd like to learn about.

The virtual display in RAM I used primarily with low end character displays, as most graphic displays these
days have on their controller interface their own ram. The approach does create an equivalent memory
size, an array, in RAM. And that is where comparisons are made, decisions to write the real display, and
the virtual also kept updated. Every time you want to do a write/update of display is when you check
the virtual one in RAM, and only when the RAM is not current with requested write do you update
both HW display and virtual display. So you can tell the display in code to write data as often as you want,
but only when the display does not reflect the data being requested to send to display does actual display and
virtual get updated. The virtual display is just a memory of past activity in essence.,


Regards, Dana.